Skip to content

eth: retry sync after peer-set replacements - #2391

Merged
vbhattaccmu merged 5 commits into
developfrom
fix/witness-peer-recovery
Sep 9, 2026
Merged

eth: retry sync after peer-set replacements#2391
vbhattaccmu merged 5 commits into
developfrom
fix/witness-peer-recovery

Conversation

@vbhattaccmu

@vbhattaccmu vbhattaccmu commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Addresses the sync-recovery blind spot observed in #2292. The peers-unavailable cooldown previously compared only the peer count, so replacing one peer with another at the same count did not trigger an early retry. This change tracks a peer-set revision, increments it on registration and removal, and clears the cooldown whenever the topology changes.

Executed tests

  • go test ./eth -run TestChainSyncerCooldownSurvivesBlockAnnounce -count=20
  • go test ./eth
  • go test -race ./eth
  • go vet ./eth
  • make lint
  • Diffguard mutation testing: 100% kill rate (1/1 mutation)
  • No Kurtosis, devnet, Amoy, or mainnet tests were run.

Rollout notes

This change is not consensus-affecting and does not modify any protocol or wire format. It is backward-compatible, requires no coordinated upgrade, and has no operator-facing configuration changes.

Copilot AI lite review requested due to automatic review settings September 7, 2026 17:12

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is localized to witness fetching, preserves existing retry exhaustion behavior, and is covered by focused unit tests for the new empty-response path.

Pull request overview

This PR updates the eth/fetcher witness fetching flow to treat empty witness responses as “not ready yet” and to retry them with a dedicated backoff strategy, instead of penalizing (dropping) the responding peer. This fits into the block fetcher’s witness acquisition path used during syncing / block import.

Changes:

  • Add per-request emptyRetries tracking plus an exponential backoff helper for empty witness responses.
  • Route empty witness responses through a retry/reschedule path that does not drop peers.
  • Add targeted tests for the new empty-response retry behavior and backoff function.
File summaries
File Description
eth/fetcher/witness_manager.go Adds empty-response retry scheduling/backoff and avoids dropping peers on empty witness payloads.
eth/fetcher/witness_manager_test.go Adds tests validating rescheduling, peer-not-dropped behavior, and backoff timing.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread eth/fetcher/witness_manager.go
@vbhattaccmu
vbhattaccmu force-pushed the fix/witness-peer-recovery branch from e54a683 to d852ddc Compare September 7, 2026 17:38
Copilot AI review requested due to automatic review settings September 7, 2026 17:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is small, thread-safe (lock-protected revision), and includes a targeted regression test for the new behavior.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@vbhattaccmu
vbhattaccmu requested a lite review from Copilot September 7, 2026 17:49
@vbhattaccmu

Copy link
Copy Markdown
Contributor Author

@claude review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is small, well-scoped, and adds targeted test coverage for the new cooldown behavior.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good — this is a small, correct concurrency bookkeeping fix. What was reviewed: the revision counter is incremented under peerSet's existing write lock in both registerPeer and unregisterPeer, and read through a new RLock-guarded currentRevision(), so there's no new race; onPeerEvent/onSyncDone correctly swap the count-equality check for a revision-inequality check, which fixes the described peer-replacement-at-same-count blind spot; and the updated test explicitly reproduces that scenario (register a second peer, unregister the original, count stays at 1) and asserts the cooldown clears.

Extended reasoning...

Overview

The diff (single commit "eth: retry sync after peer-set replacements") touches only eth/peerset.go, eth/sync.go, and eth/sync_test.go. It adds a revision uint64 field to peerSet, bumped inside registerPeer/unregisterPeer under the existing ps.lock, plus a new currentRevision() accessor guarded by RLock. chainSyncer swaps its old peersUnavailableAtCount/len() count comparison for a peersUnavailableAtRevision/currentRevision() comparison, fixing a real logic gap: a peer replacement (disconnect + connect) leaves the peer count unchanged but should still clear the "peers unavailable" retry cooldown. The test was updated to explicitly unregister the original peer after registering a second one (keeping count at 1) and assert the cooldown clears, directly covering the previously-mishandled case.

Note: the "Pull request" title/description/timeline supplied in this task's context (about eth/fetcher witness-retry logic touching witness_manager.go) does not match this diff at all — that file isn't part of the change, and git log between the given commits shows exactly one commit, matching the peerset/sync revision fix described in "What changed." Per review policy, code is ground truth over description, so this review evaluates the actual diff; the mismatched metadata and prior-review warning appear to belong to unrelated PR content and don't raise any objection against this specific change.

Security risks

None identified. This is internal synchronization bookkeeping in the eth handler's sync-retry cooldown logic — no untrusted input parsing, no crypto, no consensus-path changes. The shared mutable field is protected by the pre-existing mutex consistent with security-common.md's requirement.

Level of scrutiny

Low-to-moderate is appropriate: the change is small, self-contained, and mechanical (swap a count comparison for a monotonic revision comparison), with correct locking and a test that exercises exactly the bug scenario described.

Other factors

No CODEOWNERS conflict apparent for these paths; the fix is well-scoped per the "small, focused PR" guidance in AGENTS.md, and the accompanying test change gives confidence the fix behaves as intended without needing to execute the test suite myself (sandbox denied running go test).

@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.61%. Comparing base (39ec8c4) to head (cb0360e).
⚠️ Report is 2 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2391      +/-   ##
===========================================
+ Coverage    55.57%   55.61%   +0.04%     
===========================================
  Files          918      918              
  Lines       167133   167145      +12     
===========================================
+ Hits         92877    92962      +85     
+ Misses       68775    68714      -61     
+ Partials      5481     5469      -12     
Files with missing lines Coverage Δ
eth/peerset.go 92.11% <100.00%> (+25.33%) ⬆️
eth/sync.go 65.15% <100.00%> (+0.31%) ⬆️

... and 21 files with indirect coverage changes

Files with missing lines Coverage Δ
eth/peerset.go 92.11% <100.00%> (+25.33%) ⬆️
eth/sync.go 65.15% <100.00%> (+0.31%) ⬆️

... and 21 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lucca30
lucca30 previously approved these changes Sep 8, 2026
pratikspatil024
pratikspatil024 previously approved these changes Sep 8, 2026
@vbhattaccmu vbhattaccmu changed the title eth/fetcher: retry empty witness responses without dropping peers eth: retry sync after peer-set replacements Sep 8, 2026
Copilot AI review requested due to automatic review settings September 8, 2026 05:41
@vbhattaccmu
vbhattaccmu dismissed stale reviews from pratikspatil024 and lucca30 via d41f64b September 8, 2026 05:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new revision-based cooldown logic has a race where onSyncDone can snapshot a revision that includes an unprocessed topology change, preventing the corresponding peer event from clearing the cooldown.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

eth/sync.go:165

  • onPeerEvent compares the peer-set revision against peersUnavailableAtRevision, but it never updates peersUnavailableAtRevision after processing an event. If onSyncDone stops snapshotting the revision (to avoid missing pending topology changes), onPeerEvent should record the latest revision seen by the sync loop so future events can be classified as topology changes vs. head announcements.
func (cs *chainSyncer) onPeerEvent() {
	if !cs.peersUnavailableUntil.IsZero() && cs.handler.peers.currentRevision() != cs.peersUnavailableAtRevision {
		cs.peersUnavailableUntil = time.Time{}
	}
}
  • Files reviewed: 4/4 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread eth/sync.go
pratikspatil024
pratikspatil024 previously approved these changes Sep 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The cooldown logic can be cleared spuriously because onSyncDone doesn’t snapshot the current peer-set revision when arming the cooldown, allowing a later non-topology peer event (e.g. block announce) to clear it.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread eth/sync.go
Copilot AI review requested due to automatic review settings September 8, 2026 08:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The revision-based peer-topology tracking is correctly synchronized, integrates cleanly into the sync loop, and is backed by focused tests covering the previously missed peer-replacement case.

Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@vbhattaccmu

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread eth/sync.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is small, concurrency-safe (revision guarded by the existing peerset lock), and includes targeted tests covering the reported failure mode and ordering edge case.

Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@vbhattaccmu
vbhattaccmu requested a review from a team September 9, 2026 04:32
Copilot AI review requested due to automatic review settings September 9, 2026 09:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

peerSet.unregisterPeer does not decrement witPeers, leaving internal peer counters inconsistent after disconnects.

Review details

Suppressed comments (1)

eth/peerset.go:294

  • peerSet.registerPeer increments ps.witPeers when a peer has a WIT extension, but unregisterPeer never decrements it. This leaves witPeers permanently inflated after peers disconnect, making the internal counters inconsistent with the actual peer set (and potentially breaking any prioritization logic that relies on this count).
	if peer.snapExt != nil {
		ps.snapPeers--
	}

	return nil
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@vbhattaccmu

Copy link
Copy Markdown
Contributor Author

🔵 Needs a closer look

peerSet.unregisterPeer does not decrement witPeers, leaving internal peer counters inconsistent after disconnects.

Review details

Suppressed comments (1)

eth/peerset.go:294

  • peerSet.registerPeer increments ps.witPeers when a peer has a WIT extension, but unregisterPeer never decrements it. This leaves witPeers permanently inflated after peers disconnect, making the internal counters inconsistent with the actual peer set (and potentially breaking any prioritization logic that relies on this count).
	if peer.snapExt != nil {
		ps.snapPeers--
	}

	return nil
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Good catch. This predates the PR: witPeers is incremented but never decremented. However, it currently has no readers, so it does not affect peer prioritization or this retry fix. I’ll keep this PR scoped and address the counter separately.Also witPeers is only an internal counter and is not read anywhere. ErrPeersUnavailable is determined by filtering the actual connected peers using SupportsWitness(), not by this counter.

@vbhattaccmu
vbhattaccmu merged commit aae7e66 into develop Sep 9, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants