Skip to content

fix(dgw): retry transient KDC connection drops#1863

Merged
Benoît Cortier (CBenoit) merged 5 commits into
masterfrom
fix/kdc-proxy-retrial
Jul 24, 2026
Merged

fix(dgw): retry transient KDC connection drops#1863
Benoît Cortier (CBenoit) merged 5 commits into
masterfrom
fix/kdc-proxy-retrial

Conversation

@CBenoit

@CBenoit Benoît Cortier (CBenoit) commented Jul 24, 2026

Copy link
Copy Markdown
Member

The KDC proxy path sometimes sees a connection accepted and then dropped before a reply is returned (suspected DC-side load: connection throttling or port exhaustion), surfacing to clients as a 502 Bad Gateway. Direct KDC exchanges are now retried up to 3 times on such transient failures, spaced with equal-jitter exponential backoff (max 1.2s / 2.4s / 4.8s) so a loaded KDC gets breathing room and concurrent retries do not synchronize into a storm. Permanent failures are still surfaced immediately, and the KDC target host is now shown at info level for easier diagnosis.

Issue: DGW-423

Copilot AI review requested due to automatic review settings July 24, 2026 07:44
@CBenoit
Benoît Cortier (CBenoit) enabled auto-merge (squash) July 24, 2026 07:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds bounded retries for transient direct KDC failures with exponential backoff and improved diagnostics.

Changes:

  • Classifies transient and permanent KDC I/O failures.
  • Retries direct exchanges with equal-jitter backoff.
  • Adds supporting dependencies and helper tests.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File Description
devolutions-gateway/src/kdc_connector.rs Implements retry behavior, logging, classification, and tests.
devolutions-gateway/Cargo.toml Adds retry and randomness dependencies.
Cargo.lock Locks the new dependency graph.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread devolutions-gateway/src/kdc_connector.rs
Comment thread devolutions-gateway/src/kdc_connector.rs Outdated
Comment thread devolutions-gateway/src/kdc_connector.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

devolutions-gateway/src/kdc_connector.rs:147

  • The retry tests verify attempt counts but not the configured 1.2/2.4/4.8-second delay maxima, so an accidental change to this base or factor would leave all tests passing while changing the main operational behavior. Please assert the elapsed paused Tokio time stays within the equal-jitter bounds, or expose the strategy and test each generated delay range.
        let backoff = ExponentialBackoff::from_millis(2)
            .factor(600)
            .max_delay(Duration::from_secs(10))
            .map(equal_jitter)
            .take(KDC_MAX_RETRIES);

Comment thread devolutions-gateway/src/kdc_connector.rs Outdated
Comment thread devolutions-gateway/src/kdc_connector.rs Outdated
The KDC proxy path sometimes sees a connection accepted and then dropped
before a reply is returned (suspected DC-side load: connection throttling
or port exhaustion), surfacing to clients as a 502 Bad Gateway. Direct KDC
exchanges are now retried up to 3 times on such transient failures, spaced
with equal-jitter exponential backoff (~1.5s / 3s / 6s) so a loaded KDC
gets breathing room and concurrent retries do not synchronize into a storm.
Permanent failures are still surfaced immediately, and the KDC target host
is now shown at info level for easier diagnosis.

Issue: DGW-423
@CBenoit
Benoît Cortier (CBenoit) merged commit de6c29e into master Jul 24, 2026
78 of 84 checks passed
@CBenoit
Benoît Cortier (CBenoit) deleted the fix/kdc-proxy-retrial branch July 24, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants