model-router-corpus

Task-classification prompts for the models-router pretrained classifier, farmed from public benchmarks.

Ten tasks: chat, code, creative, extraction, math, reasoning, sql, summarization, tool-use, translation.

Kept out of the models repository on purpose. The prompts are third-party text under a mix of source licences, and separating them means a licence question about one benchmark never blocks a release of the library. What models ships is the derived index — embeddings, not prompts.

Layout

File	What it is
benchmark-prompts.tsv	The corpus: split<TAB>task<TAB>source<TAB>prompt
sources.json	Per-source provenance, row counts, and licence
farm_benchmarks.py	Rebuilds the corpus from HuggingFace parquet shards
bakeoff.py	Compares embedding models as 1-NN task classifiers
calibrate_threshold.py	Measures in-domain against out-of-domain similarity
bakeoff-results.json	Last bake-off run
threshold-calibration.json	Last calibration run

Prompts only. Reference answers are never included, which keeps this a classification corpus rather than a redistribution of the benchmarks themselves.

Splits

1881 training prompts, 469 held out. Deduplicated within each task first, then any prompt appearing under more than one task is dropped — doing it the other way round removes every copy of an intra-task duplicate and silently guts the smaller tasks.

Building the index from it

The index ships inside models-router; this rebuilds it.

./gradlew :models-bench:run --args="task-index build \
  --model ~/.jvllm/models/embeddinggemma-300M-Q8_0.gguf \
  --model-id google_embeddinggemma_300m_gguf_q8_0 \
  --corpus /path/to/model-router-corpus/benchmark-prompts.tsv \
  --out models-router/src/main/resources/com/integrallis/models/router/index \
  --quantizer SQ4"

--quantizer SQ4 is not an optimisation to be dropped: without it the artifact is eight times larger, because a quantizer otherwise writes compressed codes beside the full-precision vectors rather than instead of them.

Measured on the held-out split, EmbeddingGemma-300M Q8_0, threshold 0.0:

Storage	Accuracy	Zipped
float32	0.9083	5.13 MB
SQ4 quantized-only	0.9019	0.64 MB

models-router pins the corpus SHA-256 in its manifest, so an index built from an edited corpus fails that project's tests rather than shipping quietly.

Current digest: aeb0fb6ec86d8a6d4dfed8b407bc6c36c0d5147aaaaf88789430a33afa62bfae

Licensing

sources.json records the licence of every source. The corpus is a derived collection of prompts; anyone redistributing it should read that file rather than assume a single licence covers it.