feat(evals): Mason eval scaffold — ADK 11-criteria + dryfit→Langfuse datasets

[nadyyym]

Why

Scaffolding to evaluate the Mason (upsell_ranker) pipeline systematically and to compare model profiles (Gemini-3-Flash vs the Opus baseline) on the same eval set. Closes the gap between the dryfit E2E rig (which produces ground truth) and a repeatable, model-parameterized eval surface.

This is scaffolding — runnable skeletons + automation, not live-executed. Based on feat/mason-flash3-batched-grounded because the dryfit rig (scripts/dryfit_e2e/*) and the LiteLLM per-role model wiring live there, not on master.

What

• scripts/evals/eval_run.py — ADK google.adk.evaluation (11-criteria) harness. Model is parameterized via per-role LiteLLM env vars (SIGNAL_AGENT_MODEL, SIGNAL_REVIEWER_MODEL, DWH_ANALYST_MODEL, UPSELL_AGENT_MODEL) applied before agent import. No native compare mode — model-vs-model = run once per profile.
• scripts/evals/eval_dataset.py — runs the dryfit rig (dryfit_to_posthog.py) → captures ground_truth.json → uploads one Langfuse dataset item per planted signal template (stable ids → idempotent upsert). input = warehouse shape, expected_output = the signal Mason should surface.
• scripts/evals/eval_compare.py — runs the golden set across 2+ profiles in isolated subprocesses → markdown + JSON comparison table.
• scripts/evals/model_profiles.json — gemini-3-flash vs opus-baseline env maps.
• scripts/evals/golden/ — one checked-in golden eval set (seat_based_smoke.evalset.json) + ADK criteria config, so make eval-run has something to run.
• Makefile (v0.0.2 root) — eval-dataset / eval-dataset-offline / eval-run / eval-compare.
• scripts/evals/README.md — full runbook.

dryfit → Langfuse dataset flow

posthog_seat_based_mvp.yaml ─▶ dryfit_to_posthog.py (in-memory) ─▶ ground_truth.json
  ─▶ eval_dataset.py: group by template_id, dedup event_names ─▶ Langfuse dataset "mason-dryfit-<scenario>"

Validated offline

• All Python compiles; all JSON valid; --help works for all three scripts.
• The dryfit→Langfuse item transform verified against a synthetic ground_truth.json (correct template grouping, event dedup, instance counts).

Needs creds / live-run to fully validate

• pip install -r requirements.txt (ADK/langfuse/litellm not in this checkout) → then make eval-run.
• Model API keys (ANTHROPIC_API_KEY / GEMINI_API_KEY).
• Langfuse keys (~/.claude/secrets/mason-flash3.env) for make eval-dataset upload; DRYFIT_DIR + PostHog Project B keys for fresh generation (or make eval-dataset-offline).

🤖 Generated with Claude Code