|
| 1 | +# End-to-end terminal recordings (VHS) |
| 2 | + |
| 3 | +This directory drives the **real `skillz` binary through its TUI** — one |
| 4 | +[VHS](https://github.com/charmbracelet/vhs) tape per command flow — and produces |
| 5 | +two artifacts per flow from a single recording: |
| 6 | + |
| 7 | +| Artifact | Purpose | |
| 8 | +| --- | --- | |
| 9 | +| `<flow>-flow.gif` | Animated demo for PRs / README | |
| 10 | +| `<flow>-flow.golden.txt` | Final-frame text snapshot, diffed in CI as an integration test | |
| 11 | + |
| 12 | + |
| 13 | + |
| 14 | +The same recording is both the demo *and* the assertion: one tape, one run. |
| 15 | + |
| 16 | +## Why a recording (we already have snapshot tests) |
| 17 | + |
| 18 | +The in-process tests (`Skillz.Tests`) drive the prompts through Spectre's |
| 19 | +`TestConsole` — fast and deterministic, but they never exercise the *compiled* |
| 20 | +binary: `Program.cs`, DI wiring, `System.CommandLine` parsing, or real terminal |
| 21 | +rendering. This tier does, end to end, through a real PTY. It is the slow, |
| 22 | +high-confidence layer — a handful of representative flows, not exhaustive. |
| 23 | + |
| 24 | +## The flows |
| 25 | + |
| 26 | +| Flow | Drives | Validates | |
| 27 | +| --- | --- | --- | |
| 28 | +| `add` | interactive: skill multi-select → searchable agent picker → scope → method → confirm | the full interactive install, **symlink** path | |
| 29 | +| `copy` | `add --agent claude-code --copy --skill … -y` | non-interactive install, **copy** path + `Copied:` summary | |
| 30 | +| `global` | `add --agent claude-code --global --skill … -y` | **global scope** (`$HOME`-rooted paths) | |
| 31 | +| `init` | `init my-skill` | skill scaffolding + next-steps output | |
| 32 | +| `list` | `list` (after a hidden install) | the installed-skills table | |
| 33 | +| `remove` | interactive: multi-select → y/n confirm → summary | **interactive removal** + lock/symlink cleanup | |
| 34 | +| `update` | interactive scope picker → Both | the update check (hermetic: local skills are never network-checked) | |
| 35 | +| `error` | `add ./missing` | the **failure UX**: error message + non-zero exit | |
| 36 | + |
| 37 | +Each flow's tape is the source of truth; keep `MARKERS`/`ALL_FLOWS` in |
| 38 | +[`run.sh`](run.sh) in sync with the tape set. |
| 39 | + |
| 40 | +## How it works |
| 41 | + |
| 42 | +``` |
| 43 | +<flow>-flow.tape ──▶ VHS container (ttyd + ffmpeg) ──▶ <flow>-flow.gif |
| 44 | + └──▶ <flow>-flow.txt ──▶ extract-frame.sh ──▶ diff vs golden |
| 45 | +``` |
| 46 | + |
| 47 | +1. **`<flow>-flow.tape`** is a VHS script. Interactive flows gate key transitions |
| 48 | + on `Wait+Screen /.../` sentinels, so the recording syncs on **state** rather |
| 49 | + than wall-clock timing (short `Sleep`s are still used to let the UI settle). |
| 50 | + A hidden setup block puts the published binary on `PATH`, works in a throwaway |
| 51 | + `/tmp/work`, and (for `list`/`remove`/`update`) pre-installs fixtures so the |
| 52 | + demo has real state. |
| 53 | +2. **`run.sh`** publishes a self-contained `linux-x64` binary once, then for each |
| 54 | + flow mounts the repo read-only into the pinned VHS container and records. |
| 55 | +3. **`extract-frame.sh`** reduces VHS's multi-frame `.txt` capture to the final |
| 56 | + completed frame, keyed on a per-flow marker (e.g. `Done!`, `Successfully removed`). |
| 57 | +4. Each frame is diffed against `<flow>-flow.golden.txt`. |
| 58 | + |
| 59 | +## Running it |
| 60 | + |
| 61 | +```bash |
| 62 | +./test/e2e/run.sh # record + verify EVERY flow (what CI does) |
| 63 | +./test/e2e/run.sh remove update # record + verify only the named flows |
| 64 | +./test/e2e/run.sh --update # accept new output: refresh all goldens + GIFs |
| 65 | +./test/e2e/run.sh --update init # refresh a single flow |
| 66 | +REBUILD=1 ./test/e2e/run.sh # force re-publish of the binary first |
| 67 | +``` |
| 68 | + |
| 69 | +`run.sh` exits non-zero if any flow's frame differs from its golden (**FAIL**) or |
| 70 | +has no golden yet (**NEW**), and collects the changed/new GIFs, frames, and diffs |
| 71 | +under `out/report/` for CI. |
| 72 | + |
| 73 | +Requirements: `docker` + the .NET SDK. Nothing else — `ttyd`, `ffmpeg`, and the |
| 74 | +fonts are baked into the pinned container. |
| 75 | + |
| 76 | +**In this repo's devcontainer**, both are provided as features (Docker via |
| 77 | +`docker-in-docker`, .NET 10 via the `dotnet` feature) — run **Dev Containers: |
| 78 | +Rebuild Container** once, then `./test/e2e/run.sh` works as above. Docker-in-Docker |
| 79 | +needs a host that permits privileged containers. |
| 80 | + |
| 81 | +## What makes it deterministic |
| 82 | + |
| 83 | +Two independent runs produce a **byte-identical** final frame. The levers: |
| 84 | + |
| 85 | +- **Assert on text, never on the GIF.** GIF bytes go through ffmpeg/gifski and are |
| 86 | + not stable across versions/platforms. The character grid (`.txt`) is. |
| 87 | +- **Final frame only.** Intermediate frames vary with timing; the end state does not. |
| 88 | +- **Pinned VHS image** (by digest) — a new VHS release can't silently reflow output. |
| 89 | +- **Fixed geometry / theme / `CursorBlink false`** in every tape. |
| 90 | +- **Hermetic inputs**: a local fixture (no network), a fixed `/tmp/work` cwd, and a |
| 91 | + pinned `$HOME` where it appears in output, so every path is constant. A clean |
| 92 | + container has no agent env (`AI_AGENT`, `CLAUDECODE`, …) and no agent config, so |
| 93 | + skillz renders the real interactive prompts with stable defaults. |
| 94 | +- **One skill where order matters.** The install report lists skills in discovery |
| 95 | + order (filesystem-dependent), so the non-interactive `copy`/`global`/`list` flows |
| 96 | + pin a single skill with `--skill`. `remove` lists skills sorted, and `update` |
| 97 | + never enumerates local skills, so those use all three fixtures. |
| 98 | + |
| 99 | +If skillz legitimately changes its output (e.g. a new universal agent), the diff |
| 100 | +fails — that's the test working. Re-run with `--update <flow>` and commit the new |
| 101 | +golden + GIF. |
| 102 | + |
| 103 | +## CI: artifacts + PR comment |
| 104 | + |
| 105 | +The [`e2e-demo`](../../.github/workflows/e2e-demo.yml) workflow records and verifies |
| 106 | +every flow on pushes and PRs that touch the CLI, the tapes, or the fixtures, and: |
| 107 | + |
| 108 | +- **Uploads** all GIFs plus `out/report/**` (changed frames, diffs, per-flow status) |
| 109 | + as the `skillz-e2e-snapshots` build artifact. |
| 110 | +- On a **PR**, posts a single collapsed comment — one expandable `<details>` per |
| 111 | + flow that **changed** (🔴) or is **new** (🆕) — with the recording inline. The |
| 112 | + GIFs are hosted on an `e2e-snapshots` side branch so GitHub renders them in the |
| 113 | + comment. When everything matches, the comment resets to a ✅ line. |
| 114 | +- **Fails the job** (after uploading + commenting) if any flow changed or is new. |
| 115 | + |
| 116 | +> Inline hosting needs write access, so it is skipped for **fork** PRs (the GIFs |
| 117 | +> are still in the artifact). Same-repo PRs get the inline previews. |
| 118 | +
|
| 119 | +## Files |
| 120 | + |
| 121 | +| File | | |
| 122 | +| --- | --- | |
| 123 | +| `<flow>-flow.tape` | The VHS script for a flow (source of truth) | |
| 124 | +| `<flow>-flow.golden.txt` | Committed final-frame snapshot | |
| 125 | +| `<flow>-flow.gif` | Committed demo (regenerate with `run.sh --update <flow>`) | |
| 126 | +| `run.sh` | Publish → record each flow → extract → verify/update + build report | |
| 127 | +| `extract-frame.sh` | VHS `.txt` → final frame (per-flow marker) | |
| 128 | +| `bin/`, `out/` | gitignored: published binary, raw recordings, report | |
| 129 | + |
| 130 | +Fixture skills live in [`../fixtures/sample-skills`](../fixtures/sample-skills). |
0 commit comments