Skip to content

Commit 06d47b3

Browse files
LinuxDev9002claude
andcommitted
docs: add AI-assisted policy exploration case study and supporting analysis
- case_study_agent_exploration.md: SOSP-ready case study (containment claim, 50 safety events / 0 kernel panics, 3 detailed policy arcs, 75% failure rate) - agent_session_analysis.md: 42 sessions, 448 MB transcript analysis - agent_codebase_analysis.md: 55 BPF files, 46 AI-attributed commits, 59+ configs - agent/q1-q5: git archaeology, safety taxonomy, 3 case studies, session log, session safety event extraction Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent c120348 commit 06d47b3

8 files changed

Lines changed: 2898 additions & 0 deletions

docs/gpu-ext/eval/agent/q1_git_archaeology.md

Lines changed: 577 additions & 0 deletions
Large diffs are not rendered by default.

docs/gpu-ext/eval/agent/q2_safety_taxonomy.md

Lines changed: 107 additions & 0 deletions
Large diffs are not rendered by default.

docs/gpu-ext/eval/agent/q3_case_studies.md

Lines changed: 254 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Q4 Session Exploration Log
2+
3+
Methodology: top-level `~/.claude/projects/-home-yunwei37-workspace-gpu-gpu-ext/*.jsonl` only; dates are the first event timestamp converted to `America/Vancouver`; `Errors` is `error` + `fail`; `Xid/crash` is `xid` + `crash|hang|panic`; `Subagents` counts `Agent` and `Task` tool uses. Topic/outcome text is a manual summary from the first/last 200 lines, with snapshot-only files called out explicitly.
4+
5+
| Session ID (first 8 chars) | Date | Lines | Topic (1 sentence) | BPF touches | Errors | Verifier hits | Xid/crash | Benchmarks | Edits | Bash cmds | Subagents | Outcome |
6+
|---|---|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---|
7+
| `9d654c47` | 2026-02-23 | 12087 | xCoord doc cleanup expanded into MSched comparison, doc pruning, and policy-related refactoring. | 3022 | 1882 | 292 | 272 | 4406 | 67 | 457 | 8 | Committed and pushed updated xCoord/MSched docs plus a new improvement section. |
8+
| `00b4c113` | 2026-02-24 | 9 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
9+
| `1499f02c` | 2026-02-24 | 3 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
10+
| `16156e61` | 2026-02-24 | 5183 | Downloaded and analyzed the MSched paper, fixed submodule/gitmodules issues, and explored whether gpu_ext could reproduce the algorithm. | 836 | 1090 | 220 | 434 | 1576 | 18 | 299 | 7 | Added the paper reference and ended with a concrete reproduction/research direction. |
11+
| `32c1ebfa` | 2026-02-24 | 3 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
12+
| `84403ce2` | 2026-02-24 | 10 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
13+
| `cec13454` | 2026-02-24 | 4 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
14+
| `ee60df57` | 2026-02-24 | 8 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
15+
| `3b3897bc` | 2026-02-25 | 1 | Single-line snapshot stub; no substantive prompt or conclusion was preserved. | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
16+
| `9dbdebcd` | 2026-02-25 | 8 | Snapshot-only session file with a few residual BPF path mentions but no dialog. | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
17+
| `d662e303` | 2026-02-25 | 94 | Located the existing MSched reproduction plan and extended it into a layer-aware combined-policy execution plan. | 95 | 49 | 27 | 59 | 141 | 0 | 0 | 3 | Produced a concrete plan covering VA analysis, baselines, BPF hooks, and experiment variants. |
18+
| `ab2202db` | 2026-02-25 | 5 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
19+
| `6b21980a` | 2026-02-26 | 190190 | Long-running MSched and cross-block exploration covering NVBit retries, template alternatives, policy capability audits, and repeated pivots. | 5035 | 4078 | 1266 | 1478 | 30495 | 201 | 676 | 21 | Collapsed the exploration into a new `docs/msched_reproduction_plan.md` Section 21 after extensive iteration. |
20+
| `ce9a1079` | 2026-02-26 | 179 | Audited the repo and surrounding caches for large temporary files and safe cleanup targets. | 0 | 39 | 1 | 0 | 66 | 0 | 36 | 1 | Freed about 133 GB across old traces, `node_modules`, editor caches, `uv`, and model caches. |
21+
| `de6eabd4` | 2026-02-26 | 4665 | Reviewed the MSched plan, retried the NVBit-based route, and decided on next-step experiments plus fallback paths. | 1132 | 1446 | 393 | 422 | 2364 | 37 | 307 | 13 | Finalized a plan centered on a 120B NVBit retry with a no-NVBit fallback. |
22+
| `6c0aa1fb` | 2026-02-27 | 1073 | Removed kernel-modification ideas from the MSched plan and implemented a proactive layer-migration policy entirely in BPF and user space. | 862 | 1986 | 173 | 196 | 1937 | 22 | 64 | 2 | Benchmarked `prefetch_proactive_layer` and recorded no meaningful speedup, then appended the results to the plan. |
23+
| `e19dd100` | 2026-02-27 | 22197 | Continued xCoord implementation with scheduler/QoS policy edits, environment recovery, and concurrent benchmark campaigns. | 3705 | 6371 | 699 | 1031 | 22203 | 234 | 1248 | 19 | Reached a 290 tok/s baseline and left the FAISS+QoS run in progress. |
24+
| `f9e903b2` | 2026-03-01 | 1 | Single-line snapshot stub; no substantive prompt or conclusion was preserved. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
25+
| `b1e7bc20` | 2026-03-03 | 4886 | Explored preempt-via-kfunc instead of workqueue, built test scaffolding, and reorganized the preempt tooling/docs. | 2194 | 3376 | 583 | 1966 | 942 | 141 | 484 | 13 | Finished with the preempt materials moved under `scripts/extension/preempt/` and path references repaired. |
26+
| `1ffa360b` | 2026-03-04 | 21355 | Continued the cross-block prefetch plan with many new policies, baseline runs, subagents, and repeated tuning attempts. | 9656 | 7196 | 2521 | 1973 | 25499 | 222 | 1510 | 65 | Ended in repeated API 400 tool-concurrency failures after a very long iteration chain. |
27+
| `648091fb` | 2026-03-04 | 6188 | Started as a documentation consistency audit and then turned into resubmission-readiness fixes plus cross-block benchmark reruns. | 1291 | 1007 | 423 | 484 | 5388 | 36 | 290 | 28 | Completed Config A at 57.84 tok/s and moved on to Config B. |
28+
| `7cf7718d` | 2026-03-04 | 2 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
29+
| `7d3f43c8` | 2026-03-04 | 2 | Snapshot-only session file with no substantive dialog in the first or last 200 lines. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
30+
| `e557f3b2` | 2026-03-04 | 1 | Single-line snapshot stub; no substantive prompt or conclusion was preserved. | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | No conclusion recorded. |
31+
| `05039828` | 2026-03-26 | 171 | Evaluated resubmission intro framing and then launched several background analysis/report-writing tasks from session data. | 40 | 162 | 241 | 257 | 102 | 4 | 20 | 2 | Queued five report tasks, including this Q4 exploration log and the Q5 safety-events report. |
32+
33+
## Answers
34+
35+
- Total sessions with BPF policy work, excluding pure docs/planning: **7**. Evidence-based set: `16156e61`, `9d654c47`, `6b21980a`, `6c0aa1fb`, `e19dd100`, `b1e7bc20`, `1ffa360b`. These are the sessions where `Edit` events touched `extension/*.bpf.c` or closely coupled policy loader files.
36+
- Total sessions with safety events by raw transcript grep (`verifier` or `xid` or `crash|hang|panic` > 0): **12**. Those sessions are `9d654c47`, `16156e61`, `d662e303`, `6b21980a`, `ce9a1079`, `de6eabd4`, `6c0aa1fb`, `e19dd100`, `b1e7bc20`, `1ffa360b`, `648091fb`, and `05039828`.
37+
- Sessions with the most `errors + failures`, and therefore the richest case-study candidates: `1ffa360b` (7196), `e19dd100` (6371), `6b21980a` (4078), `b1e7bc20` (3376), and `6c0aa1fb` (1986).
38+
- Sessions with the longest iteration chains by session length: `6b21980a` (190190 lines), `e19dd100` (22197 lines), `1ffa360b` (21355 lines), `9d654c47` (12087 lines), and `648091fb` (6188 lines).
39+
- Sessions with the longest iteration chains by action density (`Edits + Bash cmds + Subagents`): `1ffa360b` (1797), `e19dd100` (1501), `6b21980a` (898), `b1e7bc20` (638), and `9d654c47` (532).

0 commit comments

Comments
 (0)