[CI/Build] Gate maintained recipe probes in conformance CI

[Xunzhuo]

Summary

• deliver one DCO-signed commit rebased on current origin/main, authored and committed only by Xunzhuo <xunzhuo.liu@amd.com>
• version and strictly validate every maintained recipe probe manifest while auto-discovering the four-file recipe catalog
• add a Go-native decision contract matrix and T0-T3 blocking coverage ratchets for decisions, entrypoints, fallbacks, signals, projections, algorithms, plugins, request shapes, and robustness classes
• define one declarative test-domain registry consumed by hosted classification and the local harness while preserving isolated CLI, memory, operator, E2E, performance, learning, packaging, and recipe executors
• rename the generic kubernetes baseline profile to the implementation-accurate envoy-ai-gateway name, with a compatibility alias for local callers
• retain separate blocking dynamic-config, multimodal-routing, routing-strategies, and agentgateway profiles because they own distinct CRD, image-modality, live-gateway, and external-controller contracts
• remove the obsolete Istio data-plane test that incorrectly targeted the ExtProc control API, and keep mesh-specific sidecar/mTLS/tracing coverage
• publish selected domain receipts, configured/asserted/uncovered recipe surfaces, live tag pass rates, per-recipe results, and failure logs in a consolidated 30-day artifact

Main CI failures repaired

The latest main failures exposed stale test assets and infrastructure assumptions. This change repairs them without lowering acceptance thresholds:

• dynamic and multimodal profiles now override the invalid inherited placeholder decision with a valid bootstrap route
• agentgateway uses a Kubernetes-compatible Gateway API release, canonical provider/keyword fields, bounded readiness diagnostics, and maintained-config validation
• anthropic-shim builds as a validated image; its incomplete backend route is retained as a manual diagnostic while 33 pure translation tests remain blocking
• production-stack creates the Prometheus rules ConfigMap required by its deployment
• routing-strategies maps provider aliases, declares its metadata recipe, and retries bounded transient OCI/Helm registry failures
• recipe live shards use an isolated model-cache namespace and exact live signal assertions
• the obsolete LegacyCacheBackend hybrid pending-flow test was removed; stable backend-specific pending contracts and primary-key tests remain

ml-model-selection remains a separate main defect: its current 20-case E2E mixes domain-classifier accuracy into model-selection acceptance and later reaches a native/runtime instability. This PR does not hide it by lowering thresholds.

Test plan

• full branch pre-commit hooks
• make workflow-ci-validate
• make agent-validate
• make recipe-conformance-static
• make build-e2e
• make test-semantic-router
• registry/classifier parity and E2E profile inventory tests
• remote PR contains exactly one Xunzhuo-authored/committed DCO commit with no Cursor trailer
• hosted recipe-conformance static gate, all three live CPU shards, and consolidated report
• hosted Core quality/security/tests, Images, all selected E2E profiles, and aggregate PR Gate

Final passing run: https://github.com/vllm-project/semantic-router/actions/runs/31408748763

Closes #2832

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	963e629
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/6a79fb28cf136400080f8dad
😎 Deploy Preview	https://deploy-preview-2835--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

Adds a first-class “recipe conformance” CI domain that automatically discovers all maintained recipes, strictly validates their probe manifests, generates deterministic coverage/sharding plans, and executes every base probe against a live CPU router with EvalTrace enabled—integrated into the existing PR/Main domain dispatchers.

Changes:

• Introduces versioned, strict probes.yaml parsing + JSON schema (v1) and removes hardcoded maintained-recipe enumeration in favor of directory discovery.
• Adds a hermetic decision-engine conformance corpus (v0.3) with a Go test harness.
• Wires a reusable Recipe Conformance workflow into PR/Main gating, plus local Make targets and a live-CPU runner script.

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/make/recipe-conformance.mk	Adds Make targets for static conformance, shard planning, and live CPU execution.
tools/ci/tests/test_pr_change_classifier.py	Adds classifier tests ensuring recipe conformance domain is selected.
tools/ci/classify_pr_changes.py	Adds recipe_conformance signal detection and job selection.
tools/agent/task-matrix.yaml	Registers a maintained-recipes task with fast/feature gates.
tools/agent/scripts/router_calibration_support.py	Tightens probe evaluation semantics (trace, alias, exact/contains matching, forbidden sets).
tools/agent/scripts/router_calibration_support_test.py	Extends tests for trace scoping, alias enforcement, and non-zero exit on failure.
tools/agent/scripts/router_calibration_report.py	Enhances markdown failure reporting with failed-check details and trace diagnostics.
tools/agent/scripts/router_calibration_manifest.py	Introduces v1 manifest validation (unknown-field rejection, unique IDs, match modes, forbidden sets).
tools/agent/scripts/router_calibration_loop.py	Returns non-zero when evaluation acceptance fails.
tools/agent/scripts/recipe_conformance.py	New: recipe discovery, strict four-file contract enforcement, reference validation, inventory/sharding, eval runner.
tools/agent/scripts/recipe_conformance_test.py	New: tests for schema/field parity, discovery validity, deterministic sharding, and directory contract enforcement.
tools/agent/schemas/recipe-probes-v1.schema.json	New: published JSON schema for maintained probe manifests (v1).
tools/agent/repo-manifest.yaml	Registers tools/agent/schemas/** and adds the PL-0040 plan to governance tracking.
tools/agent/e2e-profile-map.yaml	Adds a workflow-driven suite entry for recipe-conformance-live.
tools/agent/docs/testing-strategy.md	Documents the new blocking static + live-CPU maintained recipe conformance layers.
tools/agent/docs/plans/README.md	Adds PL-0040 to the indexed plan list.
tools/agent/docs/plans/pl-0040-recipe-conformance-ci.md	New: execution plan for implementing recipe conformance CI.
src/semantic-router/pkg/decision/testdata/conformance/v0.3/README.md	New: documents the hermetic decision-engine corpus contract and coverage matrix.
src/semantic-router/pkg/decision/testdata/conformance/v0.3/cases.jsonl	New: 23-case deterministic JSONL corpus for decision semantics.
src/semantic-router/pkg/decision/conformance_test.go	New: Go test harness that loads corpus, validates, and asserts trace invariants.
src/semantic-router/pkg/config/maintained_recipe_probe_contract_test.go	Tightens Go-side maintained probe manifest contract checks (schema/name + variant constraints).
src/semantic-router/pkg/config/maintained_asset_contract_test.go	Removes hardcoded maintained recipe list; iterates discovered recipe directories.
Makefile	Includes the new tools/make/recipe-conformance.mk.
e2e/testing/run_recipe_conformance.sh	New: orchestrates per-recipe live-CPU conformance runs and artifact/log collection.
config/recipes/README.md	Documents v1 probe schema + new conformance commands and CI behavior.
config/recipes/privacy/probes.yaml	Adds schema_version: v1.
config/recipes/multi-objective/probes.yaml	Adds schema_version: v1.
config/recipes/knowledge/probes.yaml	Adds schema_version: v1.
config/recipes/feedback/probes.yaml	Adds schema_version: v1.
config/recipes/balance/probes.yaml	Adds schema_version: v1.
config/recipes/agent/probes.yaml	Adds schema_version: v1.
config/recipes/accuracy/probes.yaml	Adds schema_version: v1.
.github/workflows/recipe-conformance.yml	New reusable workflow: static inventory + sharded live CPU execution with artifacts.
.github/workflows/pr.yml	Dispatches recipe conformance domain and includes it in PR Gate aggregation.
.github/workflows/main.yml	Dispatches recipe conformance domain on main when signaled.
.github/workflows/ci-changes.yml	Exports recipe_conformance change signal from the classifier workflow.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-08-11 02:06 UTC · Rule: default · triggered by rule queue approved pull requests
• ✅ Checks skipped · PR is already up-to-date
• 🚫 Left the queue — 2026-08-11 02:12 UTC · at 963e629b536fbbf91a8b73080bcb1abdba179568

This pull request spent 5 minutes 39 seconds in the queue, including 2 seconds running CI.

Reason

Pull request #2835 has been merged manually at 56f9375

Hint

You were too fast!

Tick the box to put this pull request back in the merge queue (same as @mergifyio queue).

• Requeue this pull request