skill-trigger-lab

A research harness measuring whether a skill's conformance score predicts how reliably coding agents trigger the skill. Fork it, run the cells for your agent, PR your results back. Design details: ARCHITECTURE.md.

Prerequisites

• Python 3.9+ (stdlib only for the driver, scoring, and core analysis)
• Node / npx — to run the pinned validator skilldex-cli@1.1.2 (fetched on demand)
• An agent CLI for real runs: Claude Code (claude) and/or Codex (codex). Not needed for the mock dry-run below.

Quick start (no inference, ~30s)

Verifies the whole pipeline end-to-end with a mock agent — do this first.

# 1. (Re)generate the 9 skill variants + their conformance vectors
python variants/generate.py

# 2. Dry-run the full matrix with the mock harness (no API calls)
python runner/driver.py --config runner/config.example.json --run-id demo

# 3. Score the run → tidy results in data/results/
python runner/score.py --run-id demo

# 4. Print the metrics (survival curves, trigger lift, RQ1 correlations)
python analysis/metrics.py --run-id demo

Running a real study

1. Copy runner/config.example.json, set "harness" to claude-code or codex and "model" to the backend you're testing. See runner/README.md for all config keys.
2. Before the first real run, pass the 4-point harness verification in ARCHITECTURE.md — start with a tiny cell subset (a couple variants × L1/L4) to confirm detection works on live inference. Real runs cost inference budget and edit the workspace.
3. Then run the same three commands as above (driver.py → score.py → metrics.py) with your --config and --run-id.

The confirmatory mixed model is a separate step: pip install -r analysis/requirements.txt && python analysis/mixed_model.py --results data/results/runs-<id>.jsonl.

Contributing your results (PR-back)

• data/raw/ (raw transcripts, stdout, diffs) is git-ignored and stays on your machine. score.py writes a sanitized data/results/runs-<id>.jsonl + .md.
• Review data/results/ before you open a PR, then PR it to this repo.

Learn more

• ARCHITECTURE.md — the design, the conformance vector, the metrics.
• DESIGN_DECISIONS.md — why things are the way they are.
• runner/, prompts/, variants/, analysis/ each have a local README.