perf: speed up PQ distance lookup

[ddupg]

Summary

• Speed up PQDistCalculator::distance by indexing transposed PQ codes directly instead of rebuilding an iterator with skip(id).step_by(num_vectors) for each candidate.
• Keep the existing PQ code layout, distance semantics, and public APIs unchanged.
• Add a focused Criterion benchmark for the per-candidate PQ distance lookup hot path.

Benchmark

Benchmark command:

cargo bench -p lance-index --bench pq_distance_lookup -- --warm-up-time 3 --measurement-time 10

Focused before/after Criterion comparison with the same benchmark code:

• Base 6c291ab14555d327dda61860433034bce7586bd7: 1.1122 ms median estimate per 16,384 distance calls, interval [1.0965 ms, 1.1290 ms]
• Optimized: 1.0085 ms median estimate per 16,384 distance calls, interval [1.0005 ms, 1.0168 ms]
• Criterion change: [−10.390%, −9.3597%, −8.3427%], p = 0.00 < 0.05

The benchmark uses 8-bit PQ with dim=512, num_sub_vectors=64, total=65,536, and deterministic seeds.

Validation

cargo fmt --all
cargo test -p lance-index vector::pq::storage -- --nocapture
cargo clippy -p lance-index --bench pq_distance_lookup -- -D warnings
cargo bench -p lance-index --bench pq_distance_lookup -- --warm-up-time 3 --measurement-time 10

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!