fix(peerstore): bound the identify stream close - #2940
Conversation
32cc557 to
fff2b5a
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2940 +/- ##
==========================================
+ Coverage 82.21% 82.23% +0.01%
==========================================
Files 177 177
Lines 32372 32375 +3
Branches 12 11 -1
==========================================
+ Hits 26616 26623 +7
+ Misses 5756 5752 -4
🚀 New features to boost your workflow:
|
|
I created the following PR to verify if nimbus aint affected: status-im/nimbus-eth2#8885 |
I am not sure of this PR because of the following reasons:
This is not true, because streams get closed under inactivity (actually this is something i gotta check in quic) Did you try removing the |
This is true for mplex and yamux, or, in claude's words: "the 5 minute channel idle timer resets the stream and ends that read". On QUIC the stream has no idle timer at all.
|
|
Yeah this is something to be fixed for Quic. Streams being infinite when Idle is a bug. |
Summary
peerStore.identifytears its stream down in afinallywithawait noCancel stream.closeWithEOF()and no bound.closeWithEOFhalf-closes and then reads until the remote's EOF (libp2p/stream/lpstream.nim:337), so a remote that answers identify and then holds the stream open parks that read.noCancelclears the cancel callback, so the caller's deadline cannot end the wait either.What ends the wait instead is the muxer idle timer, 5 minutes on mplex and yamux (
libp2p/builders.nim:185,libp2p/muxers/yamux/yamux.nim:531). On QUIC nothing does:QuicStreamsetstimeout: 0.millis(libp2p/transports/quictransport.nim:68), so it runs no idle monitor, and the transport idle timeout is refreshed by every other stream on the connection.Both callers run identify under a 30 second deadline that therefore does not hold:
Dialer.finishUpgrade(libp2p/dialer.nim:289) holds the peer's dial lock, so every later dial to that peer queues behind it.Switch.upgrader(libp2p/switch.nim:242) holds one of the 32ConcurrentUpgradesslots.DefaultDialerTimeoutandUpgradeTimeoutare both 30 seconds, so the hold outlasts its budget by 10x on a muxed stream, and without limit on QUIC.The fix is the
retireStreampattern fromlibp2p/protocols/kademlia/message_sender.nim:83: the close gets its own bound, and the stream is reset when the remote does not answer it.noCancelwraps the wholecloseWithEOF().withTimeout(...), so the inner close stays cancellable for the timer while the teardown still ignores the caller's cancellation.Sibling of #2936, on the dial path instead of the Kademlia RPC path.
Affected Areas
Compatibility & Downstream Validation
unstablewithvendor/nim-libp2pset to this branch.nimbus_beacon_nodebuilds, and bothmake local-testnet-minimalandmake local-testnet-mainnetreport "The simulation completed successfully" (exit 0). The build also needsnim-libplumv0.6.2 invendor/, which nimbus-eth2 does not vendor yet; that gap is independent of this PR.Impact on Library Users
One new public constant,
IdentifyCloseTimeout(5 seconds). No API, wire or config change.A remote that answers identify and never closes the stream now costs 5 seconds instead of up to 5 minutes on mplex or yamux, and instead of no bound at all on QUIC. The dial then completes normally, because identify already stored its info.
Risk Assessment
noCancelstays. It comes from fix(peerstore): reset identify stream on cancellation #2459, which guarantees the teardown runs. Without it thefinallydies on the first cancel and leaves the stream neither closed nor reset, which nothing cleans up on QUIC.await noCancel stream.reset(), is still unbounded, becauseLPChannel.resetwrites throughnoCancel s.conn.writeMsg(libp2p/muxers/mplex/lpchannel.nim:114). It wants its own PR.References
Additional Notes
One regression test,
A remote that never closes the identify stream frees the dialintests/libp2p/test_dialer.nim. The remote answers identify and then holds the stream, and the test dials twice, so the second dial passes only if the first freed the dial lock. It fails at 15.0 s on the old code and passes with the fix (dialer suite 6 OK). Each dial is bounded withjoin().withTimeout(...), notwait(), because await()would cancel into thenoCancelteardown and hang.libp2p/services/identify_pusher.nim:98holds the same unbounded close, but its caller detaches the send, so it stalls no dial.retireStreamnow exists twice and wants a shared home inLPStream. Both are follow-ups.