perf(encoding): compact mini-block chunk index for cached page state

[Ali2Arslan]

Summary

The mini-block structural scheduler caches per-page state that is consulted on
every read. Today that state is a Vec<ChunkMeta> (num_values, size, offset per
chunk) plus a repetition index of per-chunk blocks (first_row,
starts_including_trailer, has_preamble, has_trailer) — ~48 bytes/chunk, almost
all of it derivable from a handful of cumulative quantities.

This PR replaces both with a compact MiniBlockChunkIndex (new
primitive/chunk_index.rs submodule) that stores only the non-redundant data
and derives the rest:

• byte_starts: PrefixSums — cumulative chunk byte sizes, stored as u32
  when the page's data buffer fits in 4 GiB and u64 otherwise.
• rows: RowMapping selected by page shape:
  • UniformFlat — fixed-width / bitpacked pages where every non-last chunk
    holds the same number of values; row↔chunk is pure arithmetic with no
    heap allocation.
  • Flat — non-uniform flat pages (RLE / FSST); a single cumulative value
    array.
  • Nested — repetition present; cumulative row starts + a has_trailer
    bitmap (preamble derived from the previous chunk's trailer) + leaf item
    counts.

byte_range/items_in_chunk/find_chunk/first_row/rows_in_chunk/
has_preamble/has_trailer preserve the exact semantics the scheduler relied
on (find_chunk still returns the first of duplicated start rows). Row→chunk
lookups remain O(1) (uniform flat) or O(log n) (binary search), and the
multi-range instruction merge in schedule_instructions now runs in place
instead of allocating a second Vec.

MiniBlockChunkIndex implements DeepSizeOf (composing the inner
PrefixSums/ItemCounts/RowMapping), so cache weighing reflects the smaller
footprint.

This is an idiomatic port of an internal change to current upstream: it uses
lance's DeepSizeOf trait (rather than a bespoke size method), threads the
existing initialize IO structure, and drops fork-only test scaffolding that
has no counterpart here.

Test plan

• cargo test -p lance-encoding --lib — 422 passed, 0 failed, 5 ignored
  (covers flat, non-uniform-flat, and nested/list miniblock round trips,
  e.g. test_sparse_large_string_list::...MINIBLOCK,
  test_sparse_boolean_list_uses_miniblock, test_nested_strings).
• New unit tests: page-shape detection (test_flat_detection cases),
  nested row/item axes (test_nested_detection_and_axes), uniform-vs-Flat
  scheduler/lookup parity (test_uniform_flat_matches_prefix_sum_flat),
  per-variant heap-size bounds vs the legacy layout
  (test_deep_size_per_variant_below_legacy), and a parse_nested_rep
  oracle test against the previous per-block decode.
• cargo fmt -p lance-encoding
• cargo clippy -p lance-encoding --tests --benches -- -D warnings

Made with Cursor

[codecov]

Codecov Report

❌ Patch coverage is 97.13740% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ing/src/encodings/logical/primitive/chunk_index.rs	95.83%	10 Missing and 2 partials ⚠️
.../lance-encoding/src/encodings/logical/primitive.rs	98.72%	2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Ali2Arslan]

this also fixes the under-reporting (wasn't accounting for chunk_meta before)

[Ali2Arslan]

bit of a drive-by, but addressed this as well