feat(knowledge): retrieval quality improvements (verbatim search, rerank topN, per-query topK, qwen3 prefix)

[eeee0717]

What this PR does

Bundles four focused knowledge-base retrieval-quality improvements that were developed and validated together on a retrieval benchmark.

Before this PR:

• kb_search rewrote the query into keywords before ever trying the user's verbatim text, so exact-match passages were often missed on the first search.
• Rerank topN was decoupled from the search topK, so reranking could operate over a different candidate count than intended.
• Callers could not set a per-query result count for kb_search; every query used the base default.
• Query embeddings for instruction-aware asymmetric models (Qwen3-Embedding) were embedded as plain text — off the model's training distribution — weakening query/document alignment.

After this PR:

• kb_search tries the verbatim query first, then falls back to keyword rewriting.
• Rerank topN is aligned with the search topK.
• Callers can pass a per-query topK (result count) to kb_search; default behavior is unchanged when omitted.
• Qwen3-Embedding queries carry the model's official instruct prefix; documents stay plain, so existing indexes remain valid and no re-embedding is needed.

Fixes # N/A

Why we need it and why it was done in this way

All four target the same goal — higher first-pass retrieval recall for the knowledge base — and were measured on a 500-question retrieval benchmark. The query-side Qwen3 instruct prefix alone lifted overall Recall@10 by ~4pt with no regression.

The following tradeoffs were made:

• Qwen3 prefix is query-only. Instruction-aware models (Qwen3) train the instruct on the query while documents stay plain. Applying it only in embedKnowledgeQuery keeps indexed chunks plain, so existing indexes stay valid and the corpus does not need re-embedding.
• Per-model prefix registry. The prefix lives in a small provider::model registry so more query-prefix models can be added with one entry, without touching the embedding path.
• Per-query topK is optional. Omitting it preserves the previous default, so existing IPC/tool callers are unaffected.

The following alternatives were considered:

• Both-side prefix models (E5, nomic). Deliberately excluded from the registry: they also need a document-side prefix, which requires prefixing documents at index time and re-embedding the corpus — a separate, heavier change.

Links to places where the discussion took place: N/A

Breaking changes

None. The Qwen3 prefix affects query embeddings only (indexes unchanged); the per-query topK is an optional parameter that defaults to prior behavior.

Special notes for your reviewer

Kept as a draft while the associated benchmark round finishes. The four commits are individually scoped (verbatim-first search, rerank topN alignment, per-query topK, Qwen3 query prefix) and can be split into separate PRs if preferred for review.

Checklist

• Branch: This PR targets the correct branch — main for active development, v1 for v1 maintenance fixes
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Documentation: A user-guide update was considered and is present (link) or not required.
• Self-review: I have reviewed my own code before requesting review from others

Release note

Improve knowledge base retrieval quality: verbatim-first search in the kb_search tool, an optional per-query result count, rerank topN aligned with search topK, and an instruction prefix for Qwen3-Embedding query embeddings (query-only, no re-embedding required).