feat(e2e): Beton/Mason E2E harness — PostHog + Supabase + Beton sinks

[nadyyym]

What

Ports the Beton / Mason end-to-end test harness into dryfit under e2e/. It was previously vendored inside getbeton/inspector-ml-backend (scripts/dryfit_e2e/ + scripts/test_env/); moving it here lets the synthetic data travel with the sinks that consume it.

The harness generates dryfit synthetic data and fans it out to three native sinks, then triggers Mason and scores its promoted signals against dryfit's hidden ground truth:

dryfit generate ──┬──▶ PostHog Project B        (synthetic events, HogQL-queryable)
                  ├──▶ Supabase branch          (Inspector workspace + data_sources)
                  └──▶ Beton / Mason on Railway (trigger → signal run)
                                                      │
                                            score.py vs ground_truth.json

Sinks

• PostHog — dryfit_to_posthog.py runs dryfit's engine in-memory (no Postgres) and POSTs events to Project B /batch with historical_migration: true. posthog_ingest.py is the Postgres-table variant.
• Supabase — bootstrap_workspace.mjs idempotently creates the Inspector test user / workspace / membership and the PostHog integration_configs + posthog_workspace_config rows (API key AES-256-GCM-encrypted to match Inspector's scheme). Node stdlib only.
• Beton / Mason — run_matrix.py switches the Mason model via railway, hits /api/agent/trigger-test, pulls experiment_report.json off the container via railway ssh, and runs a model×batch-size matrix.

Plus score.py (coverage-based recall/precision), posthog_dashboard.py, run.sh (local-Mason single-scenario), generate.sh, and run_e2e.sh — an orchestrator chaining the three sinks (STAGE=supabase|posthog|beton|all). README.md + .env.example document the rig.

Path rewiring from the inspector-ml-backend layout

• Artifacts now write under e2e/artifacts/ (gitignored), not the old $V0_DIR/artifacts.
• run_matrix.py --repo-dir no longer defaults to a relative parent (which used to be the Mason checkout); it now requires MASON_REPO_DIR — the inspector-ml-backend checkout that's railway link-ed to the e2e service, since run_matrix.py drives railway redeploys/ssh from there.

Validation

• py_compile / bash -n / node --check all clean.
• dryfit_to_posthog.py --dry-run generates 76,185 events + 230 signal instances against the current dryfit engine and writes ground truth + manifest (0 skipped). Confirmed EventRecord / GenerationResult field mapping matches this dryfit version.
• Not exercised: live PostHog ingest, Supabase bootstrap, and the Railway trigger — those need the test-project credentials + a railway login.

Context

This unblocks validating inspector-ml-backend#12 (LiteLLM adapter + Langfuse-alongside-AgentOps) on a separate Railway env with its own PostHog Project B + Supabase branch.

🤖 Generated with Claude Code

[nadyyym]

✅ First end-to-end run with this harness (gemini-3-flash)

Exercised the ported harness against the live rig: separate PostHog Project B (406509), the Inspector Supabase branch, and Mason on its Railway env (kind-sparkle / mason-flash3-e2e). Used it to validate inspector-ml-backend#12 (LiteLLM + Langfuse) on gemini-3-flash.

Sinks / stages that worked

• PostHog (Project B) — dryfit_to_posthog.py --dry-run generated 76,185 events / 230 signal instances against the current dryfit engine; Project B already held 73,600 seat-based events (6 event types incl. seat_activated), HogQL-queryable. Field mapping (EventRecord → PostHog) confirmed against this dryfit version.
• Supabase — reused the existing bootstrapped workspace 0c6c640e-… (validated PostHog data source). bootstrap_workspace.mjs is idempotent; note the branch's GoTrue auth-admin endpoint was returning 500, but the PostgREST rows (workspace / posthog_workspace_config / members) were intact, so the workspace was reusable.
• Beton / Mason — single signal_agent run triggered directly (POST /apps/.../sessions/... + /run); HTTP 200 in 315s on gemini-3-flash-preview.
• Scoring — score.py ran against ground truth: 4 templates, recall 0.0 / precision 0.0 (0 promoted — see below).

Telemetry captured (via the run)

Langfuse trace 1b6a3df9efac812afd671e1dfe92ff3c: 38 LLM generations on gemini-3-flash-preview, 445,836 tokens, $0.2374.

Caveat

The run promoted 0 signals: warehouse profiling returned an empty table allowlist (available_tables: []), so all 9 SQLs were policy-blocked (schema_forbidden) and the discovery loop never started. That's a Mason-side profiling ↔ Inspector-callback gap (the deployed code is master-based), not a harness problem — the harness drove all stages correctly and produced the diagnostic artifacts that pinpointed it.

Harness notes for reviewers

• run_matrix.py --repo-dir now requires MASON_REPO_DIR (the inspector-ml-backend checkout railway link-ed to the e2e service) — it drives railway redeploys/ssh from there.
• railway ssh needs explicit --project/--environment/--service IDs (not names) to pull experiment_report.json off the container.
• Artifacts land under e2e/artifacts/<scenario>/ (gitignored).

[nadyyym]

🔁 Corrected end-to-end run — harness drove all stages; found two Mason-side issues

Follow-up to the run above (which scored 0/0). Root cause was session registration, not the harness data path:

• signal_agent.list_tables → Inspector GET /api/agent/list-tables?session_id=<sid>, and Inspector resolves the workspace (→ PostHog data source) by looking that session up in workspace_agent_sessions.
• Minting the session directly on Mason ⇒ {"error":"Session not found"} ⇒ 0 tables ⇒ empty allowlist ⇒ all SQL policy-blocked.
• Fix: POST /api/agent/trigger-test {workspace_id} first to register the session, then run Mason with it. Verified list-tables then returns 110 tables incl. events.

Corrected run (gemini-3-flash, registered session)

The harness drove every stage correctly: PostHog Project B data (73.6k seat events) → registered session → Mason run → profiling succeeded (primary_events_table: events) → 14/14 SQL policy-allowed → 8 executions against Project B returning cohort rows → artifacts pulled via railway ssh.

Two findings for the harness / Mason

1. Session registration is load-bearing — run_matrix.py's trigger-test-then-run sequence is required; a direct Mason /run with a self-minted session silently yields an empty table allowlist. Worth encoding this explicitly in run_e2e.sh (mint via Inspector, read the session_id back from workspace_agent_sessions, then run) so others don't repeat it. I can push that to this branch.
2. Mason double-run-dir/finalize bug — each run creates two artifact dirs: the loop work (profiled events, executed SQL, candidates) lands in dir A, but finalize_experiment_report writes to a second re-initialized dir with an empty profile (loop_not_started). So experiment_report.json shows promoted: 0 regardless, which blocks score.py from producing a real recall/precision. This is a Mason-side bug (master signal_agent), to be filed against inspector-ml-backend.

Harness notes confirmed

• run_matrix.py --repo-dir requires MASON_REPO_DIR (the inspector-ml-backend checkout railway link-ed to the e2e service).
• railway ssh needs explicit --project/--environment/--service IDs (not names) to pull artifacts.
• Telemetry from the runs: Langfuse traces on gemini-3-flash-preview with token+cost (445,836 tokens / $0.2374 on the first run).