[flink] Gate single-row lookup pushdown on partition liveness - #3442
Conversation
|
@luoyuxia @loserwang1024 PTAL 🙏 |
924cd13 to
4f71fdc
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes Flink lookup pushdown correctness for lake-enabled, partitioned primary-key tables by preventing single-row KV lookups from being used when the target partition no longer exists in Fluss (but still exists in the lake), ensuring the planner falls back to the union scan path in that case.
Changes:
- Gate full-PK single-row lookup pushdown on Fluss partition liveness for lake-enabled partitioned tables.
- Add a utility to check partition existence via Fluss Admin metadata.
- Add an IT case that reproduces the “expired-in-Fluss but present-in-lake” point-query scenario and verifies correct results.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/flink/FlinkUnionReadPrimaryKeyTableITCase.java | Adds an integration test ensuring point lookups on expired partitions still read from the lake. |
| fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/utils/PushdownUtils.java | Adds partitionExists(...) helper to check partition liveness via Admin. |
| fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/source/FlinkTableSource.java | Uses partition liveness to decide whether the single-row lookup pushdown is safe. |
| fluss-client/src/main/java/org/apache/fluss/client/table/getter/PartitionGetter.java | Exposes getResolvedPartitionSpec(...) to reuse partition spec resolution logic. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
@fresh-borzoni Thanks for the pr. lgtm overall. Only one minor comment
Also, https://cwiki.apache.org/confluence/display/FLUSS/FIP-28%3A+Improve+historical+Partition+Support+for+Fluss+Datalake-Enabled+Tables
will make lookup expire partition from paimon possible. Could you please left a todo when FIP28 is finished, we can then fall into the normal path to lookup instead of full scan?
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
@fresh-borzoni Thanks LGTM! But seems ci fails
|
@luoyuxia Thank you, looking into CI update: fixed the issue, will wait until green |
84fecf8 to
f12b6cb
Compare
f12b6cb to
6b7781f
Compare
closes #3443
A full-PK point query on a lake-enabled partitioned table returns empty when the partition was expired from Fluss but still lives in the lake and the lookup pushdown skips the union scan.
Gate the lookup on partition liveness, fall back to the scan otherwise.