Keep transaction-log retention on a durable sequence floor

[kriszyp]

Ordinary and startup transaction-log retention now delete only an eligible contiguous oldest prefix below the persisted flush floor. The sequence file named by txn.state and every newer file remain, so txn.state, its cached file handle, the writer, and the existing contiguous reader stay in the same sequence space.

This deliberately does not change transaction-log reader behavior or add JavaScript/native cache-generation APIs. destroy: true remains the existing separate destructive path; Harper does not call it in production.

An idle store may retain one segment past the age cutoff until a later write rotates and flushes it. Disk retained past the cutoff is bounded by transactionLogMaxSize, except when a single transaction exceeds that target.

Windows CI on this branch failed the new should detach an already-missing prefix entry without wedging retention test with EBUSY on all five runtimes, because the segment the test unlinks was still open. The cause is startup discovery, not retention: registerLogFile() promoted whatever segment currently held the highest sequence — opening it and enabling its append-boundary marker — and nothing demoted it when a higher one turned up. Directory iteration order is unspecified, so on a sorted-enumeration filesystem every superseded segment kept an unused marker and an open handle for the life of the store; on Windows that handle is taken without FILE_SHARE_DELETE, so those segments were also undeletable from outside the process. Discovery now registers without opening, and load() marker-enables and opens the one file that is still current after the whole directory has been scanned. An unopenable current segment degrades as before — appends resume at a fresh sequence — while an append-boundary violation stays fatal.

Refs HarperFast/harper#846 — Transaction-log retention has no config-driven time/size control (and the storageReclamation hardening isn't on v5.1).

For the human reviewer

• The discovery regression test asserts that only the highest segment is open (procfs, Linux) and marked. It cannot be made fully order-independent from the test side: if directory_iterator happens to yield the highest sequence first, the old code produced the same state. On a sorted-enumeration filesystem — including the NTFS runners where this failed — it is deterministic. Both reviewers flagged this; the limitation is documented at the test.
• Independent review raised several pre-existing points about the retention commits that predate this fix and that I left alone as out of scope for a CI repair: purgeableFiles still counts files that doPurge's new contiguous-prefix rule will not delete; a non-ENOENT stat failure breaks retention permanently with only a DEBUG_LOG; the flushed-position extent guard is now unreachable on the !all path; and startup retention still deletes files during a read-only open. Worth a decision before this merges.

Verification

• The startup-retention regression fails on origin/main: an aged current segment is deleted.
• 90 transaction-log and stats tests passed; 3 platform-specific tests skipped.
• The full JavaScript suite passed: 828 tests, with 2 skipped.
• The native suite passed: 148 tests, with 3 macOS-only tests skipped.
• Native build, type checking, lint, formatting, and git diff --check passed.
• The discovery regression test fails against the pre-fix binding (1.txnlog.boundary present) and passes after.
• Re-verified after the discovery fix: full JavaScript suite 829 passed / 3 skipped, native suite 151 passed, pnpm check clean.

Complexity: moderate

_{Review-Coverage: authored=claude; ran=codex,gemini; declined=cursor-grok,cursor-composer,domain; rounds=2 @ c955d31}

_{Human-Review-Need: 3 @ c955d31}

[gemini-code-assist]

Code Review

This pull request refactors the transaction log store to ensure that only closed stores can lose their directory or flush watermark, preventing issues where active stores are deleted during ordinary retention purges. Key changes include updating close logic, rewriting timestamp-based position lookups, writing the flush state via a local stream, and handling watermark resumption on startup. Additionally, the transaction log reader is updated to skip missing segments. Feedback on the reader implementation highlights two potential bugs where empty log segments (containing only the header) could cause the iterator or reader to prematurely terminate; using while loops instead of if conditions is recommended to continuously advance past empty segments.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.48K ops/sec	40.85	39.47	604.045	0.125	122,409
🥈 rocksdb	2	10.47K ops/sec	95.48	91.63	31,166.598	1.23	52,370

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.58K ops/sec	34.99	33.74	652.216	0.108	142,901
🥈 rocksdb	2	11.47K ops/sec	87.19	85.11	704.933	0.052	57,347

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	25.38K ops/sec	39.41	36.39	1,833.551	0.300	126,883
🥈 rocksdb	2	15.99K ops/sec	62.53	54.48	1,076.207	0.123	79,962

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	369.09 ops/sec	2,709.379	62.97	72,420.764	18.76	744
🥈 lmdb	2	26.82 ops/sec	37,279.495	427.64	1,177,702.013	136.027	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	39.23K ops/sec	25.49	11.92	20,593.337	0.845	196,155
🥈 lmdb	2	439.39 ops/sec	2,275.902	266.015	31,280.555	1.72	2,197

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	743.28K ops/sec	1.35	1.17	4,540.601	0.197	3,716,397
🥈 lmdb	2	445.72K ops/sec	2.24	1.16	5,466.218	0.781	2,228,596

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	843.42 ops/sec	1,185.655	1,037.392	2,190.416	0.331	1,687
🥈 lmdb	2	1.14 ops/sec	877,703.948	814,838.836	938,602.318	2.67	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	23.70K ops/sec	42.19	28.63	20,417.073	2.08	47,403
🥈 lmdb	2	829.23 ops/sec	1,205.932	52.21	11,674.253	5.53	1,660

---

Results from commit 6deb387