feat(v0.0.2): LiteLLM model adapter + Langfuse telemetry alongside AgentOps

[nadyyym]

What

Two focused changes to the v0.0.2 agent pipeline:

1. LLM choice → LiteLLM adapter. Replaces the native-Gemini-only model factory with a LiteLLM-fronted one, so each agent's model is provider-agnostic and selectable via the existing *_MODEL env vars (gemini/..., anthropic/..., vertex_ai/..., etc.).
2. Langfuse telemetry alongside AgentOps. Adds Langfuse as a second observability backend that runs concurrently with AgentOps. Each is independently gated by its own credentials — set neither, either, or both.

This is the focused litellm + langfuse subset extracted from feat/mason-flash3-batched-grounded; the batching / dryfit_e2e / memory-bank grounding work from that branch is intentionally left out.

LLM choice

• shared/model_factory.py: build_gemini() → build_model() returning LiteLlm.
  • Bare model names (the legacy gemini-3-flash-preview still in .env.example) auto-prefix to gemini/, so existing env values keep working unchanged.
  • Mirrors GOOGLE_API_KEY → GEMINI_API_KEY (litellm's gemini provider reads the latter) so the switch doesn't silently break auth.
  • Retries via litellm num_retries (LLM_NUM_RETRIES, default 6), replacing the native Gemini HttpRetryOptions. The old GEMINI_RETRY_* backoff/jitter knobs are no longer read.
• upsell_agent / dwh_analyst / signal_agent updated to import build_model.
• signal_agent/agent.py quota handling now catches both the native ADK _ResourceExhaustedError and litellm.exceptions.RateLimitError (post-LiteLLM, 429s surface as the latter), preserving the graceful-exit / quota_exhausted_error behavior.

Telemetry (side by side, env-gated, concurrent)

• shared/observability.py: init_observability() inits AgentOps (AGENTOPS_API_KEY) and Langfuse (LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY, LANGFUSE_HOST) independently and idempotently. Langfuse is wired via OpenInference's GoogleADKInstrumentor.
• shared/litellm_otel.py: bridges LiteLLM token/cost usage onto the active OTel span (OpenInference doesn't capture usage from LiteLLM-fronted models, so without this Langfuse shows zero tokens/cost).
• Agents call init_observability() at import time (ADK loads agent modules directly); the inline agentops.init() blocks are removed and centralized. v0.0.2/__init__.py also inits as a belt-and-suspenders entry point.

Deps / config

• requirements.txt: litellm, langfuse, openinference-instrumentation-google-adk, anthropic — pinned against google-adk==1.33.0 via a successful docker build: litellm==1.88.0, langfuse==4.7.1, openinference-instrumentation-google-adk==0.1.15, anthropic==0.107.1.
• .env.example: adds GEMINI_API_KEY, LANGFUSE_PUBLIC_KEY/SECRET_KEY/HOST, LLM_NUM_RETRIES, a note on the <provider>/<model> format, and a note that the old GEMINI_RETRY_* knobs are no longer read.

Validation

• docker build on the repo Dockerfile (python:3.11-slim) succeeds with the pinned deps against google-adk==1.33.0.
• Verified in the built image: google.adk.models.lite_llm.LiteLlm exists, accepts num_retries=6 (forwarded to litellm.completion, stored in _additional_args), and signal_agent.agent imports cleanly with the dual-exception quota guard.
• A live smoke /run was not exercised — it needs Langfuse + PostHog secrets. Please run a smoke /run and confirm traces appear in Langfuse + AgentOps before merge.

🤖 Generated with Claude Code

[nadyyym]

@Sashmark97 this pr lets me:

1. see telemetry of agent in langfuse/posthog
2. test different model providers – e.g. Claude/ChatGPT

Tested and it all works. Could you merge pls?

[nadyyym]

✅ E2E validation on Railway (gemini-3-flash)

Deployed this branch to the dedicated Railway env kind-sparkle / mason-flash3-e2e (separate PostHog Project B + Supabase branch) via railway up, and drove one signal_agent run with the dryfit posthog_seat_based_mvp scenario. Harness: getbeton/dryfit#3.

What this PR changes — validated ✅

Build / boot — build SUCCESS; uvicorn came up clean with no ImportError/traceback, so the new deps (litellm, langfuse, openinference-instrumentation-google-adk, anthropic) all resolve in the image.

LiteLLM adapter → gemini-3-flash — run returned HTTP 200 (315s) with "modelVersion": "gemini-3-flash-preview"; the agent generated valid HogQL against the events table. The Langfuse trace recorded 38 LLM generations, all on gemini-3-flash-preview — confirming build_model() routes through LiteLLM with the bare→gemini/ normalization.

Langfuse alongside AgentOps — one trace 1b6a3df9efac812afd671e1dfe92ff3c (invocation [signal_agent]), 87 observations / 38 generations:

• tokens: 445,836 (407,091 in / 38,745 out)
• cost: $0.2374

Crucially, token+cost are populated — the prior 2026-05-03 traces on this env show cost=0. That delta is exactly what litellm_otel.py adds (bridging LiteLLM usage onto the OTel span). So the litellm→Langfuse usage bridge works.

AgentOps was not live-exercised: AGENTOPS_API_KEY isn't set in this e2e env, so per the PR's independent gating only Langfuse activated — which is the intended graceful behavior. The AgentOps path is unchanged from master (still agentops.init() gated on the key) and verified by code/gating review, not a live trace.

Caveat — signal output was 0 (NOT caused by this PR)

The run promoted 0 signals (recall/precision 0/0). Root cause from experiment_report.json + policy_validations.jsonl:

• stop_reason: "loop_not_started", warehouse_profile.available_tables: []
• all 9 candidate SQLs rejection_class: "schema_forbidden" → ["no_allowlisted_tables_discovered", "unknown_tables_referenced"]

i.e. the warehouse-profiling step returned an empty table allowlist, so the SQL policy guard blocked everything before the discovery loop started. gemini-3-flash itself produced correct queries (FROM events WHERE event = 'seat_activated' …). This PR touches only the model factory + observability wiring — it does not touch profiling, the Inspector HogQL callback, or the policy guard. This branch is built on master, which lacks the mason-flash3 profiling/grounding work; the empty-allowlist behavior is a pre-existing master ↔ Inspector-callback gap and should be tracked separately.

Verdict

The two things this PR ships — LiteLLM model routing and Langfuse telemetry alongside AgentOps — are validated end-to-end on gemini-3-flash. The 0-signal result is an unrelated warehouse-profiling issue in the master-based pipeline.

[nadyyym]

🔁 Re-run with the harness bug fixed — litellm + Langfuse confirmed; signal score blocked by a separate Mason bug

Follow-up to the comment above. The 0-signal / empty-allowlist result there was my harness mistake, not this PR: I minted the Mason session directly (POST /apps/.../sessions/...), but signal_agent.list_tables calls Inspector GET /api/agent/list-tables?session_id=<sid> and Inspector resolves the workspace by looking that session up in workspace_agent_sessions. An unregistered session ⇒ {"error":"Session not found"} ⇒ 0 tables ⇒ empty allowlist ⇒ every SQL policy-blocked.

Fix: register the session first via POST /api/agent/trigger-test {workspace_id}, then run Mason with that session_id. With a registered session, list-tables returns 110 tables incl. events.

Re-run on gemini-3-flash (registered session) — pipeline executes end-to-end

• Warehouse profiling succeeds: primary_events_table: "events".
• SQL policy guard: 14/14 allowed (was 0/9).
• 8 SQL executions against PostHog Project B via Inspector sql-proxy, returning cohort rows (real HogQL with group joins). An earlier registered-session run proposed 5 candidate hypotheses / 10 SQL attempts.

What this PR ships — re-confirmed ✅

• LiteLLM → gemini-3-flash: all generations on gemini-3-flash-preview; valid HogQL produced.
• Langfuse alongside AgentOps: trace captured with token + cost populated (445,836 tokens / $0.2374 on the first run) — the litellm_otel.py bridge works; pre-PR traces show cost=0.

⚠️ Final promoted-signal score is blocked by a Mason-side bug (NOT this PR)

Every run creates two run dirs: dir A gets the real work (profiled events, executed SQL, candidates), but finalize_experiment_report writes experiment_report.json to a second, re-initialized dir with an empty warehouse_profile and stop_reason: "loop_not_started". So the report always shows promoted: 0 regardless of how the loop actually went — making recall/precision unscoreable. This is a state/run-dir handling bug in the master signal_agent; this PR touches only the model factory + observability. Recommend filing it separately as the blocker for signal-quality scoring.

Net: PR #12's two changes — LiteLLM routing and Langfuse-alongside-AgentOps — are validated end-to-end on gemini-3-flash. The unscoreable signal output is the unrelated finalize/run-dir bug. (AgentOps still not live-exercised — no AGENTOPS_API_KEY in the e2e env; gated off correctly.)