WIP#2571

Draft

scottsand-db wants to merge 6 commits into

delta-io:mainfrom

scottsand-db:stack/incremental_crc_5

Collaborator

scottsand-db commented May 15, 2026 •

edited

Loading

🥞 Stacked PR

Use this link to review incremental changes.

stack/incremental_crc_1 [Files changed] [MERGED]
- stack/incremental_crc_2 [Files changed] [MERGED]
  - stack/incremental_crc_3 [Files changed] [MERGED]
    - stack/incremental_crc_4 [Files changed]
      - stack/incremental_crc_5 [Files changed]

What changes are proposed in this pull request?

How was this change tested?

scottsand-db force-pushed the stack/incremental_crc_5 branch from 751aa73 to 547cbb2 Compare

May 15, 2026 20:32

This was referenced May 15, 2026

refactor: introduce SetTransactionState typed enum #2570

Merged

refactor!: introduce DomainMetadataState typed enum #2567

Merged

github-actions Bot assigned scottsand-db

github-actions Bot commented May 15, 2026 •

edited

Loading

PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: WIP
PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: WIP
PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: WIP
PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: WIP
PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: WIP
PR title does not match the required pattern. Please ensure you follow the conventional commits spec.

Your title should start with feat:, fix:, chore:, docs:, perf:, refactor:, test:, or ci:, and if it's a breaking change that should be suffixed with a ! (like feat!:), and then a 1-72 character brief description of your change.

Title: WIP

codecov Bot commented May 15, 2026 •

edited

Loading

Codecov Report

❌ Patch coverage is 96.48000% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.77%. Comparing base (8844a0c) to head (7b46957).

Files with missing lines	Patch %	Lines
kernel/src/log_segment/crc_replay.rs	95.87%	2 Missing and 16 partials ⚠️
kernel/src/crc/state.rs	88.88%	2 Missing ⚠️
kernel/src/crc/writer.rs	96.87%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2571      +/-   ##
==========================================
+ Coverage   88.72%   88.77%   +0.05%     
==========================================
  Files         182      183       +1     
  Lines       62014    62493     +479     
  Branches    62014    62493     +479     
==========================================
+ Hits        55021    55481     +460     
- Misses       4854     4857       +3     
- Partials     2139     2155      +16

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions Bot added the breaking-change label

scottsand-db force-pushed the stack/incremental_crc_5 branch from 547cbb2 to c30a677 Compare

May 15, 2026 21:54

scottsand-db added 6 commits

May 19, 2026 19:57


          refactor(crc): introduce SetTransactionState typed enum

6f83335

Mirrors DomainMetadataState: Complete(map) when the CRC file had the
setTransactions field (authoritative for misses), Partial(map) when it
did not (hits are authoritative, misses fall through to log replay).

- Replaces Option<HashMap<String, SetTransaction>> on Crc with
  set_transaction_state: SetTransactionState
- TryFrom<CrcRaw>: Some(vec) -> Complete(map), None -> Partial(empty)
- TryFrom<&Crc>: Complete -> Some(vec), Partial -> None (dropped)
- Crc::apply upserts into whichever variant is present, preserving it
- into_crc_for_version_zero produces Complete (full state known)
- get_app_id_version: Complete miss returns None (authoritative),
  Partial miss falls through to log replay


          impl self review feedback

36f4b68


          simplify tests with rstest

6aad45f


          respond to pr comments; add extra case to e2e test

6bfe9c7


          feat(log_segment): add CrcReplayAccumulator for stale-CRC reverse replay

11a27cf

Adds LogSegment::build_crc_delta_from_stale, which reverse-replays plain
commit files in (base_version, end_version] and produces a CrcDelta that
Crc::apply consumes to advance the base CRC to the snapshot version.
Foundation for PR5's Snapshot wiring.

Includes prefactor-marked changes to CrcDelta (HashMap-keyed DM/txn,
single is_incremental_safe flag, flat Option<i64> ICT) that will be split
into a separate pre-factor PR before merge; see PREFACTOR-PR: markers.


          review: write-path ICT check + design notes + kernel-review fixes

7b46957

Adds the spec-mandated invariant check in try_write_crc_file:
in_commit_timestamp_opt must be present iff the protocol's
inCommitTimestamp writer feature is enabled. Catch-net for malformed
deltas the reverse-replay producer doesn't validate inline.

Adds kernel/src/crc/DESIGN_NOTES.md documenting the design pivots made
on this branch (per-commit boundaries via _file, ICT first-commit-only
capture, HashMap-shaped CrcDelta) so the split into pre-factor +
production sub-PRs preserves the rationale.

Misc kernel-review fixes: column-index constants in the visitor;
clearer naming (is_first_commit, process_batch_start,
process_commit_file_end); base_version >= end_version precondition
error; additional tests (protocol replacement, base_version filter
boundary, parameterized N-files x M-batches accumulator).

scottsand-db force-pushed the stack/incremental_crc_5 branch from 0d36eb5 to 7b46957 Compare

May 19, 2026 19:59

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change