feat: Add prefilter support to SearchV for filtered vector search #834

Summary

Add support for prefiltering in SearchV to filter vectors during HNSW traversal rather than post-filtering results.

Problem

Currently, filtering vector search results requires post-filtering:

searched <- SearchV<MyType>(Embed(query), limit)
results <- searched::WHERE(_::{category}::EQ("electronics"))

This is inefficient because:

1. HNSW retrieves limit vectors first
2. Filter is applied after, potentially returning fewer than limit results
3. For highly selective filters, most retrieved vectors are discarded

Proposed Solution

Add filter support directly to SearchV:

// Option A: WHERE clause syntax
results <- SearchV<MyType>(Embed(query), limit, WHERE category::EQ("electronics"))

// Option B: Filter function syntax  
results <- SearchV<MyType>(Embed(query), limit, filter: _::{category}::EQ("electronics"))

Implementation Notes

The HNSW Rust code already supports filters:

fn search<F>(
    ...
    filter: Option<&'arena [F]>,
) where F: Fn(&HVector<'arena>, &RoTxn<'db>) -> bool

Changes needed:

1. Extend query DSL parser to accept filter expressions in SearchV
2. Modify query compiler to translate filter expressions to Rust closures
3. Pass filters to HNSW search function

Prefilter Considerations

Pure prefiltering can break HNSW's greedy navigation if filter nodes are skipped entirely. Two approaches:

1. Filtered search with bridge nodes: Allow non-matching nodes during traversal but only return matching ones
2. Over-fetch and filter: Retrieve limit * multiplier candidates, filter, return top limit

Use Cases

• Search products by category
• Search documents by date range
• Search codes by chapter/level (ICD-10, etc.)
• Any schema with filterable properties

🤖 Generated with Claude Code