fix(kalshi): merge live volume_fp from market_updates_raw into market_details

[los-xyz]

Summary

kalshi.markets_raw.volume_fp can freeze at creation-time for actively traded markets because the market-feed markets-sync cron queries Kalshi with min_created_ts and never re-fetches existing markets. The kalshi_market_details spell then filters on volume_fp >= 100 against that stale value, dropping markets that look idle on the raw side but trade heavily in reality. The downstream kalshi.market_trades spell INNER JOINs market_details on ticker, so any dropped market also drops every trade for it.

This regression went live on 2026-05-08, the day market-feed PR #110 routed the ticker WebSocket stream into a new kalshi.market_updates_raw table instead of refreshing markets_raw. The dbt spell still read from markets_raw only.

Measured impact (prod, before this fix)

date range	% of kalshi.market_trades dropped vs market_trades_raw
2026-05-04 to 2026-05-07	2-9% (baseline tail)
2026-05-08	52% (regression begins)
2026-05-11 onwards	78-91%

Example: KXIPLGAME-26MAY15CSKLSG-CSK had 21,627 trades / 10M+ contracts on 5/15 but markets_raw.volume_fp = 0 and updated_time frozen at 2026-05-08. market_updates_raw.volume_fp = 20,546,389 for the same ticker.

Fix

• Add a latest_market_updates CTE that pulls live per-ticker volume from kalshi.market_updates_raw (scoped by incremental_predicate('from_unixtime(ts)') in incremental runs).
• LEFT JOIN it onto markets_raw in the markets CTE.
• Apply the >= 100 filter against coalesce(updates.volume_fp, markets_raw.volume_fp) so freshly active markets are no longer dropped.
• Project the live value as volume_fp downstream; consumers (incl. kalshi.market_trades via the INNER JOIN) get the current value.
• Existing fallback semantics preserved for tickers with no updates in the incremental window.
• Declare kalshi.market_updates_raw in sources/kalshi/_sources.yml (the table existed in raw catalog but wasn't wired into spellbook) and add a freshness caveat to the markets_raw description.

Validation

Run on prod via Dune (see query 7550793 for the per-ticker spot-check and 7550797 for the aggregate rescue count):

• ✅ Test ticker KXIPLGAME-26MAY15CSKLSG-CSK now passes the filter with effective_volume_fp = 20,546,389 (live, from updates_raw) instead of 0 (stale, from markets_raw).
• ✅ Rescued universe across the catalog: 469,719 additional tickers pass the >= 100 filter, carrying ~6.75B contracts of trade volume currently being silently filtered out.
• ✅ uv run dbt --warn-error compile -s kalshi_market_details is clean.

⚠️ Backfill required after merge

kalshi.market_details is incremental on m.updated_time; tickers whose updated_time is older than the incremental window won't refresh on the next scheduled run even after this merge lands. Please run:

dbt run --select kalshi_market_details --full-refresh
dbt run --select kalshi_market_trades --full-refresh

to recover the ~2 weeks of dropped trades, otherwise the chart at /collection/prediction-markets/overview and any other consumer of prediction_markets.trades stays partially broken until natural ticker churn catches up.

Follow-up (separate PR)

The ingest-side root cause is market-feed's markets-sync using min_created_ts as the cursor. ListMarketsParams.MinUpdatedTS already exists in client/kalshi.go and would refresh existing markets on state change. Worth filing.

Test plan

• Compile passes locally (uv run dbt --warn-error compile -s kalshi_market_details).
• Sample validation query returns the rescued ticker.
• Run --full-refresh on kalshi.market_details and kalshi.market_trades in the deploy.
• Confirm kalshi.market_trades row count returns to ~3M trades/day post-refresh.
• Confirm Weekly Trade Volume by Venue chart at /collection/prediction-markets/overview recovers.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Changes kalshi.market_details incremental logic and its volume_fp >= 100 eligibility filter, which can materially alter which tickers (and thus downstream trades) are included. Risk is mainly data correctness/backfill needs and incremental edge cases, not security-sensitive code.

Overview
Fixes kalshi.market_details to use live per-ticker volume from kalshi.market_updates_raw when deciding whether a market passes the volume_fp >= 100 filter, and to project that live volume downstream.

Adds a latest_market_updates CTE (incrementally scoped) and left-joins it to markets_raw, updating the incremental selection to also process tickers with new update-stream activity. Declares the new market_updates_raw source in sources/kalshi/_sources.yml and documents that markets_raw volume can be stale.

^{Reviewed by Cursor Bugbot for commit 1364658. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Source timestamp used instead of ingest timestamp
  • Changed incremental predicate from ts (source timestamp) to received_hour (ingest timestamp) to prevent silently dropping updates with backdated timestamps.

Or push these changes by commenting:

@cursor push 3b86ca1de5

Preview (3b86ca1de5)

diff --git a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql
--- a/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql
+++ b/dbt_subprojects/daily_spellbook/models/_projects/kalshi/kalshi_market_details.sql
@@ -29,7 +29,7 @@
 	from
 		{{ source('kalshi', 'market_updates_raw') }}
 	{% if is_incremental() -%}
-		where {{ incremental_predicate('from_unixtime(ts)') }}
+		where {{ incremental_predicate('received_hour') }}
 	{% endif -%}
 	group by
 		market_ticker

_{You can send follow-ups to the cloud agent here.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1364658. Configure here.}

[jeff-dude]

will wait to deploy.