Measure routing across all advertised workflows

[04kash]

What to build

Deliver a prompt-mode routing eval that measures whether rhdh-templates is consulted for its advertised workflows and avoided for neighboring or unrelated work, then reports a confusion matrix, activation precision, recall, and named misses.

Use AEH’s dataset-generation facilities to propose candidate routing prompts and coverage gaps. Generated cases are candidates only: a maintainer must review each prompt, annotation, and judge contract before it is committed to the permanent matrix. Generate into a scratch/candidate location first so reviewed cases are never overwritten.

Acceptance criteria

• The reviewed matrix includes at least one positive case for each of the 13 advertised workflows plus explicit, implicit, terse, and ambiguous prompt styles.
• AEH-generated candidate prompts cover gaps across explicit, implicit, terse, ambiguous, and near-miss requests; only maintainer-reviewed candidates are promoted to the permanent case set.
• Negative cases cover rhdh-upgrade-helper, skill-maker, Backstage plugin work, Kubernetes work, generic YAML editing, and unrelated RHDH configuration.
• Activation is derived from normalized trace events showing the skill instructions were read; response text claiming activation is ignored.
• Three runs use the same cases and pinned model, and the report identifies precision, recall, variance, false-positive cases, and false-negative cases without setting a threshold.

Blocked by

• Ship a reproducible local AEH validation eval #6

[04kash]

AEH-generated candidate prompts cover gaps across explicit, implicit, terse, ambiguous, and near-miss requests; only maintainer-reviewed candidates are promoted to the permanent case set.

Currently this generation can be done with Anthropic models only