fix(kad-dht): reprovide CIDs in Kademlia key order to reduce new dials#3426
Open
paschal533 wants to merge 1 commit intolibp2p:mainfrom
Open
fix(kad-dht): reprovide CIDs in Kademlia key order to reduce new dials#3426paschal533 wants to merge 1 commit intolibp2p:mainfrom
paschal533 wants to merge 1 commit intolibp2p:mainfrom
Conversation
Collect all CIDs that need reproviding before queueing them, sort by their Kademlia key, then queue in that order. XOR-adjacent CIDs share the same K closest peers, so connections opened for one CID are reused for the next, reducing the number of new dials across a reprovide run. Ports the SweepingProvider optimisation from go-libp2p: libp2p/go-libp2p#2774
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
During a reprovide run, the reprovider iterates all stored CIDs and calls
provide()for each one. Eachprovide()opens connections to the K closest peers for that CID. Because CIDs are processed in datastore iteration order (essentially random), each CID's K closest peers are likely different, so every CID requires a fresh set of dials.This ports the SweepingProvider optimisation from go-libp2p: collect all CIDs that need reproviding, sort them by their Kademlia key, then queue them in that order.
XOR-adjacent CIDs share the same K closest peers. By processing them consecutively, the connections opened for one CID are still live when the next CID is queued, so they get reused instead of requiring new dials. Over a full reprovide run with many CIDs this significantly reduces the total number of new connections opened.
Changes
src/reprovider.ts: collect CIDs into an array before queueing, sort by Kademlia key, then queue in sorted ordertest/reprovider.spec.ts: inserts 5 CIDs in reverse Kademlia key order and verifiescontentRouting.provideis called in ascending Kademlia key orderTest plan
should reprovide in Kademlia key orderinserts CIDs in reverse order, verifies provide calls arrive in correct ascending Kademlia key order