spawn_cpu deadlock risk in IVF streaming partition search (deferred from #7637)

[wjones127]

Summary

The audit in #7637 found one production spawn_cpu call site that violates the
"never wait inside spawn_cpu" rule: the streaming/sequential partition search
in IvfSubIndex::search_partitions.

rust/lance/src/index/vector/ivf/v2.rs (streaming branch, ~L1671): a single
spawn_cpu closure pumps a blocking_recv on prepared_rx and blocking_send
on a capacity-1 batch_tx. Both park the CPU-pool thread. The producer feeding
prepared_rx does object-store I/O (and its partition decode can itself need the
CPU pool), so on hosts whose CPU pool collapses to a single blocking thread
(<= 3 CPUs) — or under enough concurrent queries — the parked search thread and
the work that would unblock it can starve each other. This is the same deadlock
class as #7423.

Why it wasn't fixed in the audit PR

This is a deliberate optimization from #6475, not an accident. The sequential
path intentionally runs a query's entire partition search on one CPU worker to
avoid per-partition CPU-task fan-out (measured 14–30% latency improvement under
concurrency), and test_sequential_initial_search_prepares_all_then_searches_on_one_cpu_thread
locks in the "reuse one cpu thread" invariant.

The obvious fix (one spawn_cpu per partition) removes the deadlock risk but
reintroduces the fan-out #6475 avoided and breaks that test — so it trades a
benchmarked perf win against correctness on small hosts. That decision needs input
from the #6475 author rather than a drive-by change.

Options considered

1. Per-partition spawn_cpu: async recv/send loop, one spawn_cpu per
   partition. Safe, simple; reintroduces fan-out, needs the "one cpu thread" test
   updated. Mild regression on the sequential early-stop path.
2. spawn_blocking off the CPU pool: keep the single-thread pumping model but
   on tokio's large blocking pool instead of the tiny CPU pool. Preserves the
   one-thread shape and avoids CPU-pool deadlock, but CPU-bound search then runs
   outside the CPU reservation (core oversubscription) and the lance-cpu
   thread-name assertion changes.

The test mock PreparedThreadCapturingIndex in rust/lance/src/io/exec/knn.rs
(~L2646) mirrors the same blocking-in-spawn_cpu structure and would need to move
in lockstep with whichever option is chosen.

Acceptance

• search_partitions never parks a CPU-pool thread on a channel.
• The chosen approach's perf impact on the sequential path is characterized.
• Regression test covering the streaming path on a single-thread CPU pool (mirror
  the #7423 approach).

Follow-up to #7637.

[wjones127]

@BubbleCal @westonpace I'm curious what your opinions are here. How should we fix this?

If there's no backpressure required, I bet we can just make the input queue a Vec and the output a queue with a guaranteed capacity. Then we never have to block.

[a-agmon]

Thanks for adding this @wjones127

I'd also suggest a third option that could keep most of the #6475 speedup without the deadlock.

Instead of one spawn_cpu worker that blocks on the channels, let the async task keep the loop and the channels, and use spawn_cpu only for the actual search — but over a small batch of K partitions at a time instead of one. The recv/send stay async so no pool thread ever parks, and batching means the fan-out cost is paid N/K times instead of N, so it should stay close to the single-worker baseline. K is just a knob we can tune.

This keeps CPU work on the CPU pool (unlike spawn_blocking) and never blocks a pool thread on a channel.

Happy to put up a PR along these lines and benchmark K against the current version.

[BubbleCal]

I like @a-agmon's solution.

the streaming partition search tends to use single thread to avoid the context switching overhead, so Per-partition spawn_cpu would bring that overhead back

spawn_blocking off the CPU pool may spawn too many threads, or we need to add an additional limiter

[a-agmon]

Hi @wjones127 @BubbleCal , please see PR #7680
Built on @wjones127 and added some perf there