perf(solana): flatten stake_account_delegations split branch (~7x faster)#9685
perf(solana): flatten stake_account_delegations split branch (~7x faster)#9685jeff-dude wants to merge 4 commits into
Conversation
…e scans The model is incremental but the split branch was nested in a way that re-iterated stake_actions × DelegateStake history, then cross-joined the result to outer split actions. On a 1-day incremental run this produced a ~67M-row cross-product on top of two unbounded DelegateStake scans (14.4M rows each). Three changes, all semantically equivalent — verified by a side-by-side regression that returned identical 122,498-row result sets on a 12-hour window: 1. Pre-filter the destination-side DelegateStake LEFT JOIN to the incremental window. Safe because the destination is created by the split, so any DelegateStake on it has call_block_time >= a.block_time >= incremental cutoff. 2. Flatten the nested old_vote subquery into a single-level LEFT JOIN from stake_actions to stake_call_DelegateStake. The nesting was redundant — its inner constraint (del.time <= a.block_time) is the same as the outer join's constraint. NULL handling preserved exactly via CASE WHEN src_del.account_stakeAccount IS NOT NULL THEN COALESCE(src_del.time, '2000-01-01') END. 3. Drop the now-unused all_delegates wrapper CTE. EXPLAIN ANALYZE on the optimized incremental SELECT against prod data: 9.53s wall, 0.751 credits, 336K-row cross-product (down from 67M). Co-Authored-By: Claude Opus 4.7 <[email protected]>
PR SummaryMedium Risk Overview The new split logic uses direct Reviewed by Cursor Bugbot for commit 9307dd5. Configure here. |
TEMPORARY — to be reverted before merge. Builds an identical copy of the pre-PR staking_solana_stake_account_delegations model under the alias stake_account_delegations_baseline so CI builds both the original and optimized versions side-by-side from the same source-data snapshot. This rules out source drift as an explanation for the prod-vs-CI diff observed earlier (33,852 only_in_ci, 208,041 only_in_prod over 4+ years). Co-Authored-By: Claude Opus 4.7 <[email protected]>
Same-source CI build comparison (PR 9685 run id 26113973693): - Optimized model: 17.07s, 17,060,199 rows, success - Original-SQL baseline: errored out at 3,746.92s (62.4 min) — hit DuneSQL query timeout before producing a comparable table The optimization is so dramatic that the baseline can't full-build inside the DuneSQL timeout, so a same-source row-level diff isn't possible via CI. Correctness evidence relies on the 12-hour side-by-side regression (122,498 = 122,498, zero diff) and the logical NULL-handling equivalence proof in the PR description. Co-Authored-By: Claude Opus 4.7 <[email protected]>
CI side-by-side full-build comparisonRan a temporary baseline-copy model alongside the optimized one in CI so both build from the same source-data snapshot:
The baseline can't full-build inside DuneSQL's query timeout, so a same-source row-level diff isn't possible via CI. The optimization is approximately 220x+ faster on full rebuild (and the baseline never finished). Correctness evidence:
Baseline copy reverted in the latest commit. |
Summary
stake_actions×stake_call_DelegateStakeself-join, producing a ~67M-row cross-product on top of two full DelegateStake history scans (14.4M rows each).old_votesubquery into a single-levelLEFT JOIN. NULL semantics preserved exactly viaCASE WHEN src_del.account_stakeAccount IS NOT NULL THEN COALESCE(src_del.call_block_time, '2000-01-01') END.call_block_time >= a.block_time >= cutoff.Measured impact (EXPLAIN ANALYZE on prod data, 1-day incremental window)
Baseline (no optimization) EXPLAIN ANALYZE was killed after running >30 min on
largetier — itself confirms the original scan profile is the bottleneck.Correctness — regression test
Side-by-side comparison of the original nested split branch vs the flattened single-level join on a 12-hour window (run via Dune large tier):
Identical result sets, zero diff in either direction.
The flattening's NULL handling is provably equivalent — when
src_delmatches, time fills with'2000-01-01'sentinel; when it doesn't match, the column falls through todst_delor NULL (same as the original nested behavior).What is intentionally NOT changed
block_timeis still backdated to the source's historical delegation time, matching prod behavior. Downstreamstaking_solana_validator_stake_account_epochs_rawdepends on this viavote.block_slot < epoch.block_slot.incremental_predicatesadded on the merge target — split-inherited output rows carry historicalblock_timevalues, which would defeat a destination-side window predicate and re-insert duplicates.Test plan
dbt slim cifor the solana subproject)dbt_utils.unique_combination_of_columnsdata test still passes onstaking_solana.stake_account_delegationsstaking_solana_validator_stake_account_epochs_rawoutput against prod after this lands (downstream consumer)🤖 Generated with Claude Code