feat(index): share IVF partition scans across batch vector queries

[sezruby]

Which issue does this PR close?

Closes #6822.

Rationale for this change

Batch vector search (#6821, PR #6828) made indexed multi-query search work by looping the full single-query plan once per query vector — re-opening the index and rebuilding the prefilter for every query — and unioning the results. That throws away the main opportunity of a batch: query vectors mostly probe overlapping IVF partitions, so the same partition storage is read and decoded many times. This PR shares that index-level state across the batch.

What changes are included in this PR?

The indexed/ANN path now reads each probed IVF partition's storage once and scores every query that probes it, with the prefilter built once and shared.

• VectorIndex trait (lance-index): defaulted supports_batch_partition_search() + search_partitions_batch(...) (default returns not_supported). Additive — no breaking change.
• IVFIndex (ivf/v2.rs): batch search for flat-style sub-indices (IVF_FLAT/PQ/SQ/RQ). Invert per-query partition lists, load each distinct partition once, accumulate one top-k heap per query, reusing accumulate_prepared_partition_search / global_heap_to_batch.
• ANNIvfBatchExec (io/exec/knn.rs): ranks each query against the centroids, runs the shared-scan batch search per delta, merges per-query top-k across deltas, emits {query_index, _distance, _rowid}. Prefilter wiring shared with the single-query node via build_dataset_prefilter.
• Each query vector is normalized independently for cosine (normalize_batch_query_for_index).

Fallback matrix (no regression):

Case	Behavior
IVF_FLAT/PQ/SQ/RQ, fixed nprobes, fully indexed	shared-scan fast path
adaptive nprobes / refine_factor / IVF_HNSW_* / mixed indexed+unindexed	per-query indexed loop (exact)

Design notes (anticipating review):

• Why a new exec node, not a mode on the existing ANN nodes? The two-node single-query pipeline streams one partition-list per delta through a per-query top-k; sharing the scan requires inverting queries onto partitions and keeping one heap per query in a single pass — a different dataflow. The new node reuses the underlying primitives (partition load, build_dataset_prefilter, the index's per-partition accumulate) and leaves the single-query nodes untouched.
• Why gate on index-type metadata, not supports_batch_partition_search()? The gate is a planning-time decision and the single-query path likewise doesn't open the index there; derive_vector_index_type reads metadata with no I/O. The opened index re-checks the trait as a defensive invariant.
• nprobes gate (correctness). The shared path searches exactly minimum_nprobes partitions/query, but the single-query path is adaptive (early_pruning floor + late-search expansion), so the two only match when nprobes is fixed. The fast path is gated to minimum_nprobes == maximum_nprobes; adaptive nprobes falls back to the per-query loop (verified: an unpinned batch diverged on every query before the gate, 0 divergence after). Open question: fixed-nprobes-first with batched early/late as a follow-up, or the full adaptive path in one PR?
• Memory. Peak = the union of probed partitions held during scoring — the same buffering the existing single-query global-heap path (search_partitions) uses, widened to the batch's partition union; per-delta output is k-bounded, so cross-delta accumulation is O(deltas × k).

Are these changes tested?

Yes.

• cargo test -p lance --lib test_batch_knn — 15 tests: plan shape, exact batch-vs-repeated-single equivalence (nprobes pinned), cosine regression, shared prefilter, multi-delta cross-delta merge, and explicit fallbacks for refine, adaptive nprobes, and IVF_HNSW (acceptance criterion: "unsupported index types have explicit behavior and tests").
• cargo test -p lance --lib dataset::scanner::test::test_knn (29) — no single-query regression (exercises the shared build_dataset_prefilter).
• cargo fmt --all && cargo clippy -p lance -p lance-index --tests --benches -- -D warnings.
• Python: pytest -k batch (L2 + cosine × three/single queries); index-type sweep (IVF_FLAT/PQ/SQ/HNSW_PQ/HNSW_SQ) matches repeated single-query; ruff clean; pyright clean on changed lines.
• Benchmark (benchmarks/test_search.py): batch vs repeated-single ANN; local run (50k rows, dim 128, IVF_PQ 64 partitions, m=32, k=10, nprobes=10) → ~2.5× faster (ratio; absolute ms machine-dependent).

Are there any user-facing changes?

No API changes. The batch-query surface (2-D / list query with a query_index output column) already shipped in #6828; this PR only changes how indexed batch queries execute, so Python and Java bindings are unaffected. It is a performance improvement for fixed-nprobes indexed batch search; results are identical to issuing the queries one at a time. No format or compatibility change.

[codecov]

Codecov Report

❌ Patch coverage is 90.58824% with 56 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/io/exec/knn.rs	84.35%	32 Missing and 9 partials ⚠️
rust/lance/src/dataset/scanner.rs	97.48%	2 Missing and 5 partials ⚠️
rust/lance-index/src/vector.rs	0.00%	5 Missing ⚠️
rust/lance/src/index/vector/ivf/v2.rs	94.00%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!