PartitionSelectRule should now strongly favor creating right deep plans

[alecgrieser]

This modifies the way ParititonSelectRule works so that we only should create right deep plans. It does this by always selecting n - 1 elements from the n quantifiers of each select.

Existential predicates are punted in this example because of how they interact with fan out indexes. For now, this just reverts those to their old logic.

The idea is that this vastly cuts down on the search space of join exploration, as we were currently producing many different groupings of the join constituents, which wasn't actually helpful given the join algorithms that we currently support.

[github-actions]

📊 Metrics Diff Analysis Report

Summary

• New queries: 0
• Dropped queries: 0
• Plan changed + metrics changed: 2
• Plan unchanged + metrics changed: 22

ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

• New queries: Queries added in this PR
• Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
• Plan changed + metrics changed: The query plan has changed along with planner metrics.
• Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

Plan and Metrics Changed

These queries experienced both plan and metrics changes. This generally indicates that there was some planner change
that means the planning for this query may be substantially different. Some amount of query plan metrics change is expected,
but the reviewer should still validate that these changes are not excessive.

Total: 2 queries

Statistical Summary (Plan and Metrics Changed)

task_count:

• Average change: -2182.5
• Median change: -1434
• Standard deviation: 748.5
• Range: -2931 to -1434
• Queries changed: 2
• No regressions! 🎉

transform_count:

• Average change: -925.5
• Median change: -804
• Standard deviation: 121.5
• Range: -1047 to -804
• Queries changed: 2
• No regressions! 🎉

transform_yield_count:

• Average change: -93.0
• Median change: -59
• Standard deviation: 34.0
• Range: -127 to -59
• Queries changed: 2
• No regressions! 🎉

insert_new_count:

• Average change: -470.5
• Median change: -235
• Standard deviation: 235.5
• Range: -706 to -235
• Queries changed: 2
• No regressions! 🎉

insert_reused_count:

• Average change: -48.0
• Median change: -48
• Standard deviation: 0.0
• Range: -48 to -48
• Queries changed: 2
• No regressions! 🎉

Significant Regressions (Plan and Metrics Changed)

There were 2 outliers detected. Outlier queries have a significant regression in at least one field. Statistically, this represents either an increase of more than two standard deviations above the mean or a large absolute increase (e.g., 100).

• yaml-tests/src/test/resources/join-with-order-by-tests.metrics.yaml:465: EXPLAIN select (t1.*) as a, (t2.*) as b, (t3.*) as c from t1, t2, t3 where t1.a1 = 1 and t2.b1 = 1 and t3.c1 = 1 and t2.b3 = t1.a3 and t3.c2 = t1.a2 and t3.c3 = t2.b3
  • old explain: ISCAN(t1$a1 [EQUALS promote(@c36 AS LONG)]) | FLATMAP q0 -> { ISCAN(t2$b1_b3 [EQUALS promote(@c36 AS LONG), EQUALS q0.a3]) | FLATMAP q1 -> { ISCAN(t3$c1_c2 [EQUALS promote(@c36 AS LONG), EQUALS q0.a2]) | FILTER _.c3 EQUALS q1.b3 AS q2 RETURN (q2 AS _0, q1 AS _1) } AS q3 RETURN ((q0.id AS id, q0.a1 AS a1, q0.a2 AS a2, q0.a3 AS a3) AS a, (q3._1.id AS id, q3._1.b1 AS b1, q3._1.b2 AS b2, q3._1.b3 AS b3) AS b, (q3._0.id AS id, q3._0.c1 AS c1, q3._0.c2 AS c2, q3._0.c3 AS c3, q3._0.aRef AS aRef) AS c) }
  • new explain: ISCAN(t1$a1 [EQUALS promote(@c36 AS LONG)]) | FLATMAP q0 -> { ISCAN(t2$b1_b3 [EQUALS promote(@c36 AS LONG), EQUALS q0.a3]) | FLATMAP q1 -> { ISCAN(t3$c1_c2 [EQUALS promote(@c36 AS LONG), EQUALS q0.a2]) | FILTER _.c3 EQUALS q1.b3 AS q2 RETURN (q1 AS _0, q2 AS _1) } AS q3 RETURN ((q0.id AS id, q0.a1 AS a1, q0.a2 AS a2, q0.a3 AS a3) AS a, (q3._0.id AS id, q3._0.b1 AS b1, q3._0.b2 AS b2, q3._0.b3 AS b3) AS b, (q3._1.id AS id, q3._1.c1 AS c1, q3._1.c2 AS c2, q3._1.c3 AS c3, q3._1.aRef AS aRef) AS c) }
  • task_count: 9219 -> 6288 (-2931)
  • transform_count: 2559 -> 1512 (-1047)
  • transform_yield_count: 554 -> 427 (-127)
  • insert_new_count: 1699 -> 993 (-706)
  • insert_reused_count: 82 -> 34 (-48)
• yaml-tests/src/test/resources/join-with-order-by-tests.metrics.yaml:484: EXPLAIN select (t1.*) as a, (t2.*) as b, (t3.*) as c from t1, t2, t3 where t1.a1 = 1 and t2.b1 = 1 and t3.c1 = 1 and t2.b3 = t1.a3 and t3.c2 = t1.a2 and t3.c3 = t2.b3 order by t1.a2
  • old explain: ISCAN(t1$a1_a2 [EQUALS promote(@c36 AS LONG)]) | FLATMAP q0 -> { ISCAN(t2$b1_b3 [EQUALS promote(@c36 AS LONG), EQUALS q0.a3]) | FLATMAP q1 -> { ISCAN(t3$c1_c2 [EQUALS promote(@c36 AS LONG), EQUALS q0.a2]) | FILTER _.c3 EQUALS q1.b3 AS q2 RETURN (q2 AS _0, q1 AS _1) } AS q3 RETURN ((q0.id AS id, q0.a1 AS a1, q0.a2 AS a2, q0.a3 AS a3) AS a, (q3._1.id AS id, q3._1.b1 AS b1, q3._1.b2 AS b2, q3._1.b3 AS b3) AS b, (q3._0.id AS id, q3._0.c1 AS c1, q3._0.c2 AS c2, q3._0.c3 AS c3, q3._0.aRef AS aRef) AS c) }
  • new explain: ISCAN(t1$a1_a2 [EQUALS promote(@c36 AS LONG)]) | FLATMAP q0 -> { ISCAN(t2$b1_b3 [EQUALS promote(@c36 AS LONG), EQUALS q0.a3]) | FLATMAP q1 -> { ISCAN(t3$c1_c2 [EQUALS promote(@c36 AS LONG), EQUALS q0.a2]) | FILTER _.c3 EQUALS q1.b3 AS q2 RETURN (q1 AS _0, q2 AS _1) } AS q3 RETURN ((q0.id AS id, q0.a1 AS a1, q0.a2 AS a2, q0.a3 AS a3) AS a, (q3._0.id AS id, q3._0.b1 AS b1, q3._0.b2 AS b2, q3._0.b3 AS b3) AS b, (q3._1.id AS id, q3._1.c1 AS c1, q3._1.c2 AS c2, q3._1.c3 AS c3, q3._1.aRef AS aRef) AS c) }
  • task_count: 5758 -> 4324 (-1434)
  • transform_count: 1975 -> 1171 (-804)
  • transform_yield_count: 405 -> 346 (-59)
  • insert_new_count: 830 -> 595 (-235)
  • insert_reused_count: 72 -> 24 (-48)

Only Metrics Changed

These queries experienced only metrics changes without any plan changes. If these metrics have substantially changed,
then a planner change has been made which affects planner performance but does not correlate with any new outcomes,
which could indicate a regression.

Total: 22 queries

Statistical Summary (Only Metrics Changed)

task_count:

• Average change: -741.9
• Median change: -603
• Standard deviation: 331.0
• Range: -1434 to -336
• Queries changed: 19
• No regressions! 🎉

transform_count:

• Average change: -408.1
• Median change: -368
• Standard deviation: 183.0
• Range: -806 to -174
• Queries changed: 19
• No regressions! 🎉

transform_yield_count:

• Average change: -30.4
• Median change: -26
• Standard deviation: 13.8
• Range: -57 to -16
• Queries changed: 19
• No regressions! 🎉

insert_new_count:

• Average change: -96.0
• Median change: -66
• Standard deviation: 63.1
• Range: -214 to -28
• Queries changed: 19
• No regressions! 🎉

insert_reused_count:

• Average change: -20.3
• Median change: -12
• Standard deviation: 15.2
• Range: -48 to -2
• Queries changed: 22
• No regressions! 🎉

There were no queries with significant regressions detected.

Minor Changes (Only Metrics Changed)

In addition, there were 22 queries with minor changes.