Skip to content

branch-4.1: [fix](score) disable search topn with extra predicates #65821 - #67327

Merged
airborne12 merged 1 commit into
apache:branch-4.1from
airborne12:pick-65821-branch-4.1
Sep 1, 2026
Merged

branch-4.1: [fix](score) disable search topn with extra predicates #65821#67327
airborne12 merged 1 commit into
apache:branch-4.1from
airborne12:pick-65821-branch-4.1

Conversation

@airborne12

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: N/A

Related PR: #65821 (master), picked from commit 2b6a45e

Problem Summary:

Backport of #65821 to branch-4.1. Search score TopN pushdown may return incorrect results when the search predicate is combined with additional predicates, because the pushed TopN limit can be applied before the remaining predicates are evaluated. This change disables the pushed search TopN limit in those cases while preserving the virtual score column pushdown, and adds regression coverage (search + equality / range / match / score range / multiple search predicates, plus limit+offset overflow).

Hunk audit (source diff → this PR):

Source hunk Status
PushDownScoreTopNIntoOlapScan.java @@ -194,17 +194,22 @@ (overflow guard rework + pushedScoreLimit) Adapted ×2: ① branch-4.1 never had the #64633 overflow-guard block, so the hunk's removed lines have no counterpart here; ② Utils.addOverflows does not exist on 4.1 (#64633 not backported) — inlined the equivalent check topN.getLimit() > Long.MAX_VALUE - topN.getOffset() (identical to the master helper's implementation).
PushDownScoreTopNIntoOlapScan.java @@ -243,6 +248,19 @@ (shouldDisableSearchTopN helper) Ported
test_search_score_topn_predicates.out (new) Ported (verbatim)
test_search_score_topn_predicates.groovy (new) Adapted: dropped set enable_segment_limit_pushdown = true — the variable comes from #62222 which is not on 4.1; it defaults to true on master and only controls a BE-side segment limit optimization, unrelated to this FE-plan-level fix.

Local verification on this branch: full ASAN BE+FE build green; run-regression-test.sh -d inverted_index_p0 -s test_search_score_topn_predicates → 1 suite, 0 failed against a local 1FE+1BE cluster built from this PR. No FE UT exists for this rule on 4.1 and the source PR added none (its coverage is the regression suite above).

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…ache#65821

Search score TopN pushdown may return incorrect results when the search
predicate is combined with additional predicates, because the pushed TopN
limit can be applied before the remaining predicates are evaluated. This
change disables the pushed search TopN limit in those cases while
preserving the virtual score column pushdown, and adds regression coverage.

Conflicts:
  fe/.../PushDownScoreTopNIntoOlapScan.java
    - branch-4.1 has no apache#64633 overflow guard block to replace; applied the
      apache#65821 form directly.
    - Utils.addOverflows does not exist on branch-4.1 (apache#64633 not
      backported); inlined the equivalent check
      (limit > Long.MAX_VALUE - offset).
  regression-test/.../test_search_score_topn_predicates.groovy
    - dropped 'set enable_segment_limit_pushdown' (apache#62222 not on 4.1;
      master-default-true BE-side optimization, unrelated to this fix).

(cherry picked from commit 2b6a45e)
@airborne12
airborne12 requested a review from yiguolei as a code owner August 31, 2026 05:05
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉
Increment coverage report
Complete coverage report

@airborne12

Copy link
Copy Markdown
Member Author

run p0

1 similar comment
@airborne12

Copy link
Copy Markdown
Member Author

run p0

@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/33390381106

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 3.63% (13/358) 🎉
Increment coverage report
Complete coverage report

@airborne12
airborne12 merged commit 211f616 into apache:branch-4.1 Sep 1, 2026
30 of 32 checks passed
@airborne12
airborne12 deleted the pick-65821-branch-4.1 branch September 1, 2026 01:25
@airborne12

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review summary for exact head f16ae639b7f6095dd93e830cd5cf35841514fae5.

One inline P2 test-coverage issue remains: the new nested OR/NOT fixtures do not make their outcomes depend on recursive SearchExpression discovery, so a shallow classifier can pass the full changed suite.

Critical checkpoints:

  • Predicate correctness: planner-visible residual predicates conservatively send the zero sentinel; score-range extraction and filter rebuilding are consistent.
  • Runtime/compatibility: present zero survives logical/physical scan copies, Thrift, and BE field-presence checks, disables both early Search Top-K and segment score TopN, and preserves full score/range materialization on the branch-4.1 BE.
  • Arithmetic/backport: the inline overflow guard is equivalent for reachable nonnegative limit/offset values, and the branch adaptation is semantically faithful to the source fix.
  • Tests: the main equality, MATCH, range, offset, multiple-Search, and score-range witnesses are deterministic and exercise the original wrong-result path; the recursive-classification oracle is the remaining gap.
  • Scope: a late DUP_KEYS predicate-delete issue was verified but is unchanged from the base lone-Search behavior, so it is not raised as a defect of this faithful backport. The proposed AGG_KEYS merge issue was disproved by per-segment K and memtable key-consolidation invariants.
  • User focus: no additional focus was provided.
  • Completion: three review rounds converged; all final-round reviewers returned NO_NEW_VALUABLE_FINDINGS, all candidates were adjudicated, and no unresolved suspicious point remains.

No builds or tests were run, as required by the review task.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants