Fix kvrocks: pipeline smembers to eliminate N+1 Redis round trips - #161
Open
t0kubetsu wants to merge 1 commit into
Open
Fix kvrocks: pipeline smembers to eliminate N+1 Redis round trips#161t0kubetsu wants to merge 1 commit into
t0kubetsu wants to merge 1 commit into
Conversation
In get_uids_by_criteria, get_uids_by_criteria_scoped, and _get_uids_for_http_headval, the scan_iter loops issued one smembers call per matching key — potentially thousands of serial round trips per search request. Replace with a two-phase approach: collect all matching keys first (scan_iter), then issue a single pipelined batch of smembers calls. Also add count=1000 hint to the two scan_iter calls that were missing it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to the closed PR #155. The
not/ntmodifier change has been dropped per maintainer feedback — that is intentional and will be addressed in a future parenthesis/syntax refactor.This PR contains only the N+1 fix, which is independent of the modifier work.
Problem
In
get_uids_by_criteria,get_uids_by_criteria_scoped, and_get_uids_for_http_headval, thescan_iterloops issued one synchronousself.r.smembers(key)call per matching key. On a large index, afield.like:valuequery could produce thousands of serial Redis round trips in a single search request.Change
Two-phase approach in all three methods:
scan_iter(key filtering only, no data reads)smemberscalls in a singlepipeline(transaction=False)batchAlso adds the missing
count=1000hint to the twoscan_itercalls that were omitted.Test plan
field.like:valuequery returns the same results as beforefield.begin:valuequery returns the same results as before