Skip to content

Fix kvrocks: pipeline smembers to eliminate N+1 Redis round trips - #161

Open
t0kubetsu wants to merge 1 commit into
D4-project:mainfrom
t0kubetsu:fix/kvrocks-smembers-pipeline
Open

Fix kvrocks: pipeline smembers to eliminate N+1 Redis round trips#161
t0kubetsu wants to merge 1 commit into
D4-project:mainfrom
t0kubetsu:fix/kvrocks-smembers-pipeline

Conversation

@t0kubetsu

Copy link
Copy Markdown
Contributor

Summary

Follow-up to the closed PR #155. The not/nt modifier change has been dropped per maintainer feedback — that is intentional and will be addressed in a future parenthesis/syntax refactor.

This PR contains only the N+1 fix, which is independent of the modifier work.

Problem

In get_uids_by_criteria, get_uids_by_criteria_scoped, and _get_uids_for_http_headval, the scan_iter loops issued one synchronous self.r.smembers(key) call per matching key. On a large index, a field.like:value query could produce thousands of serial Redis round trips in a single search request.

Change

Two-phase approach in all three methods:

  1. Collect all matching keys from scan_iter (key filtering only, no data reads)
  2. Issue all smembers calls in a single pipeline(transaction=False) batch
# Before — N+1 round trips
for key in self.r.scan_iter(f"{base_field}:*"):
    if matches(key):
        matching_uids.update(self.r.smembers(key).intersection(partial_result))

# After — 1 pipeline per query
matching_keys = [k for k in self.r.scan_iter(f"{base_field}:*", count=1000) if matches(k)]
if matching_keys:
    pipe = self.r.pipeline(transaction=False)
    for k in matching_keys: pipe.smembers(k)
    for members in pipe.execute():
        matching_uids.update(set(members).intersection(partial_result))

Also adds the missing count=1000 hint to the two scan_iter calls that were omitted.

Test plan

  • A field.like:value query returns the same results as before
  • A field.begin:value query returns the same results as before
  • Redis monitor shows a single PIPELINE batch instead of N individual SMEMBERS commands

In get_uids_by_criteria, get_uids_by_criteria_scoped, and
_get_uids_for_http_headval, the scan_iter loops issued one smembers
call per matching key — potentially thousands of serial round trips
per search request. Replace with a two-phase approach: collect all
matching keys first (scan_iter), then issue a single pipelined batch
of smembers calls. Also add count=1000 hint to the two scan_iter
calls that were missing it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant