branch-4.1: [fix](score) disable search topn with extra predicates #65821 - #67327
Conversation
…ache#65821 Search score TopN pushdown may return incorrect results when the search predicate is combined with additional predicates, because the pushed TopN limit can be applied before the remaining predicates are evaluated. This change disables the pushed search TopN limit in those cases while preserving the virtual score column pushdown, and adds regression coverage. Conflicts: fe/.../PushDownScoreTopNIntoOlapScan.java - branch-4.1 has no apache#64633 overflow guard block to replace; applied the apache#65821 form directly. - Utils.addOverflows does not exist on branch-4.1 (apache#64633 not backported); inlined the equivalent check (limit > Long.MAX_VALUE - offset). regression-test/.../test_search_score_topn_predicates.groovy - dropped 'set enable_segment_limit_pushdown' (apache#62222 not on 4.1; master-default-true BE-side optimization, unrelated to this fix). (cherry picked from commit 2b6a45e)
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
|
run p0 |
1 similar comment
|
run p0 |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
FE Regression Coverage ReportIncrement line coverage |
|
/review |
There was a problem hiding this comment.
Automated review summary for exact head f16ae639b7f6095dd93e830cd5cf35841514fae5.
One inline P2 test-coverage issue remains: the new nested OR/NOT fixtures do not make their outcomes depend on recursive SearchExpression discovery, so a shallow classifier can pass the full changed suite.
Critical checkpoints:
- Predicate correctness: planner-visible residual predicates conservatively send the zero sentinel; score-range extraction and filter rebuilding are consistent.
- Runtime/compatibility: present zero survives logical/physical scan copies, Thrift, and BE field-presence checks, disables both early Search Top-K and segment score TopN, and preserves full score/range materialization on the branch-4.1 BE.
- Arithmetic/backport: the inline overflow guard is equivalent for reachable nonnegative limit/offset values, and the branch adaptation is semantically faithful to the source fix.
- Tests: the main equality, MATCH, range, offset, multiple-Search, and score-range witnesses are deterministic and exercise the original wrong-result path; the recursive-classification oracle is the remaining gap.
- Scope: a late DUP_KEYS predicate-delete issue was verified but is unchanged from the base lone-Search behavior, so it is not raised as a defect of this faithful backport. The proposed AGG_KEYS merge issue was disproved by per-segment K and memtable key-consolidation invariants.
- User focus: no additional focus was provided.
- Completion: three review rounds converged; all final-round reviewers returned
NO_NEW_VALUABLE_FINDINGS, all candidates were adjudicated, and no unresolved suspicious point remains.
No builds or tests were run, as required by the review task.
|
PR approved by at least one committer and no changes requested. |
What problem does this PR solve?
Issue Number: N/A
Related PR: #65821 (master), picked from commit 2b6a45e
Problem Summary:
Backport of #65821 to branch-4.1. Search score TopN pushdown may return incorrect results when the search predicate is combined with additional predicates, because the pushed TopN limit can be applied before the remaining predicates are evaluated. This change disables the pushed search TopN limit in those cases while preserving the virtual score column pushdown, and adds regression coverage (search + equality / range / match / score range / multiple search predicates, plus limit+offset overflow).
Hunk audit (source diff → this PR):
PushDownScoreTopNIntoOlapScan.java@@ -194,17 +194,22 @@(overflow guard rework + pushedScoreLimit)Utils.addOverflowsdoes not exist on 4.1 (#64633 not backported) — inlined the equivalent checktopN.getLimit() > Long.MAX_VALUE - topN.getOffset()(identical to the master helper's implementation).PushDownScoreTopNIntoOlapScan.java@@ -243,6 +248,19 @@(shouldDisableSearchTopNhelper)test_search_score_topn_predicates.out(new)test_search_score_topn_predicates.groovy(new)set enable_segment_limit_pushdown = true— the variable comes from #62222 which is not on 4.1; it defaults to true on master and only controls a BE-side segment limit optimization, unrelated to this FE-plan-level fix.Local verification on this branch: full ASAN BE+FE build green;
run-regression-test.sh -d inverted_index_p0 -s test_search_score_topn_predicates→ 1 suite, 0 failed against a local 1FE+1BE cluster built from this PR. No FE UT exists for this rule on 4.1 and the source PR added none (its coverage is the regression suite above).Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)