Skip to content

Commit 0f2ffc9

Browse files
claudeclaude
committed
fix(mutants): the full sweep has never completed, and said it had
Queue item was "record the first real mutation-sweep result once a Monday sweep has run." Checking whether one had produced a finding instead: **no full sweep has ever completed**, and the one attempt reported success anyway. Verified against the Actions API, not assumed. mutants.yml has exactly one scheduled run in its history — 30236603180, 2026-07-27, against b416c1a, run conclusion `cancelled`. Nine of its eleven sweep jobs succeeded; a2a-server shards 3/8 and 4/8 were cancelled at the 120-minute job timeout. The shards that did finish took 86-113 minutes, so those two were always only minutes from the limit. `Mutants Summary` concluded **success** regardless. A cancelled shard still runs its upload step, so it contributes an empty mutants.out; the aggregator's count() reads zero missed from it, and the job's only failure condition was TOTAL_MISSED > 0. Two elevenths of the workspace went unmutated and the gate said every mutant was caught — a green check over work that did not happen. The hazard was already understood here for the *incremental* PR job, whose header says a job that overruns "gets cancelled, so the gate silently stops enforcing", and which was sharded for exactly that reason. The full sweep had the identical hole and no guard. Both halves fixed: * A shard-completeness gate runs before aggregation and fails unless the matrix result is `success` AND the expected number of per-shard reports arrived. Two checks because they fail differently: a shard can end non-success, or its artifact can vanish while the job reports success. A workflow_dispatch scoped to one package is exempt from the count, since it skips the other shards by design — caught while writing the guard, which would otherwise have failed every scoped run. * a2a-server is sharded 12 ways instead of 8, cutting per-shard load by a third so the slowest finishes with headroom rather than four minutes of it. Raising the timeout instead would have been a guess: the cancelled shards' true duration is unknown precisely because they were cancelled. Expect the next scheduled sweep to be red if anything is genuinely incomplete. That is the point — it was silently green before. mutation-history.md records why it is still empty and that the 2026-07-27 score must NOT be backfilled: it was computed from partial data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4Eq5Yinogz6qNGpAe9Ns1
1 parent 1a173eb commit 0f2ffc9

2 files changed

Lines changed: 157 additions & 28 deletions

File tree

‎.github/workflows/mutants.yml‎

Lines changed: 117 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,10 @@
88
# incremental (diff-only) job instead — see the `on.schedule` cron below.
99
#
1010
# Architecture:
11-
# - Full sweep splits into 4 parallel jobs (one per library crate) to avoid
12-
# the 120-minute GitHub Actions timeout. a2a-server alone can take 100+
13-
# minutes when run sequentially.
11+
# - Full sweep splits into 15 parallel jobs: one each for a2a-types,
12+
# a2a-client and a2a-sdk, plus 12 shards of a2a-server. Sharding exists
13+
# to stay inside the 120-minute GitHub Actions job timeout; at 8 shards
14+
# two of them hit it (run 30236603180, 2026-07-27) and were cancelled.
1415
# - A lightweight aggregation job merges per-crate reports and produces the
1516
# combined score.
1617
# - Incremental mode (PRs) mutates only PR-diff changes (--in-diff) in one job.
@@ -59,9 +60,9 @@ env:
5960
jobs:
6061
# ── Per-crate mutation sweep (parallel) ──────────────────────────────────────
6162
#
62-
# Splits the workspace into 4 parallel runners — one per library crate.
63-
# This avoids the 120-minute timeout that the monolithic --workspace run hit,
64-
# since a2a-server alone can generate 200-400 mutants at ~90s each.
63+
# Splits the workspace across parallel runners. This avoids the 120-minute
64+
# timeout that the monolithic --workspace run hit, since a2a-server alone
65+
# generates > 1,800 mutants.
6566
#
6667
# When a specific package is provided via workflow_dispatch, only that
6768
# package's job runs (the others are skipped via the `if` condition).
@@ -79,11 +80,18 @@ jobs:
7980
matrix:
8081
# a2a-server has > 1,800 mutants and cannot finish inside a single
8182
# 2-hour job even at 4-way intra-crate parallelism — so we shard it
82-
# across 8 separate runners (wall-clock ≈ the slowest shard; runs
83-
# use nextest + a debuginfo-free profile, see mutants.toml). The `shard` value is passed verbatim to
84-
# cargo-mutants' `--shard K/N` flag (cargo-mutants will split mutants
85-
# deterministically across shards). Other crates run as a single
86-
# shard (K=0/N=1 ≡ no sharding).
83+
# across 12 separate runners. Wall-clock ≈ the slowest shard; runs
84+
# use nextest + a debuginfo-free profile (see mutants.toml).
85+
#
86+
# It was 8, and 8 was not enough: in run 30236603180 (2026-07-27) the
87+
# shards that finished took 86-113 minutes, and shards 3/8 and 4/8
88+
# were cancelled at the 120-minute job timeout. 12 shards cut the
89+
# per-shard load by a third, putting the slowest inside the limit
90+
# with real headroom rather than four minutes of it.
91+
#
92+
# The `shard` value is passed verbatim to cargo-mutants' `--shard K/N`
93+
# flag, which splits mutants deterministically across shards. Other
94+
# crates run as a single shard (K=0/N=1 ≡ no sharding).
8795
include:
8896
- crate: a2a-protocol-types
8997
short: a2a-types
@@ -95,36 +103,52 @@ jobs:
95103
shard: "0/1"
96104
- crate: a2a-protocol-server
97105
short: a2a-server
98-
name: a2a-server shard 1/8
99-
shard: "0/8"
106+
name: a2a-server shard 1/12
107+
shard: "0/12"
100108
- crate: a2a-protocol-server
101109
short: a2a-server
102-
name: a2a-server shard 2/8
103-
shard: "1/8"
110+
name: a2a-server shard 2/12
111+
shard: "1/12"
104112
- crate: a2a-protocol-server
105113
short: a2a-server
106-
name: a2a-server shard 3/8
107-
shard: "2/8"
114+
name: a2a-server shard 3/12
115+
shard: "2/12"
108116
- crate: a2a-protocol-server
109117
short: a2a-server
110-
name: a2a-server shard 4/8
111-
shard: "3/8"
118+
name: a2a-server shard 4/12
119+
shard: "3/12"
112120
- crate: a2a-protocol-server
113121
short: a2a-server
114-
name: a2a-server shard 5/8
115-
shard: "4/8"
122+
name: a2a-server shard 5/12
123+
shard: "4/12"
116124
- crate: a2a-protocol-server
117125
short: a2a-server
118-
name: a2a-server shard 6/8
119-
shard: "5/8"
126+
name: a2a-server shard 6/12
127+
shard: "5/12"
120128
- crate: a2a-protocol-server
121129
short: a2a-server
122-
name: a2a-server shard 7/8
123-
shard: "6/8"
130+
name: a2a-server shard 7/12
131+
shard: "6/12"
124132
- crate: a2a-protocol-server
125133
short: a2a-server
126-
name: a2a-server shard 8/8
127-
shard: "7/8"
134+
name: a2a-server shard 8/12
135+
shard: "7/12"
136+
- crate: a2a-protocol-server
137+
short: a2a-server
138+
name: a2a-server shard 9/12
139+
shard: "8/12"
140+
- crate: a2a-protocol-server
141+
short: a2a-server
142+
name: a2a-server shard 10/12
143+
shard: "9/12"
144+
- crate: a2a-protocol-server
145+
short: a2a-server
146+
name: a2a-server shard 11/12
147+
shard: "10/12"
148+
- crate: a2a-protocol-server
149+
short: a2a-server
150+
name: a2a-server shard 12/12
151+
shard: "11/12"
128152
- crate: a2a-protocol-sdk
129153
short: a2a-sdk
130154
name: a2a-sdk
@@ -298,6 +322,72 @@ jobs:
298322
pattern: mutation-report-*
299323
path: reports/
300324

325+
# THE SHARD-COMPLETENESS GATE.
326+
#
327+
# This job aggregates whatever artifacts happen to be present and then
328+
# fails only on `TOTAL_MISSED > 0`. That is unsound on its own: a shard
329+
# that times out still runs its upload step, so it contributes an empty
330+
# or partial `mutants.out`, `count()` reads 0 missed from it, and the
331+
# aggregate reports a clean sweep over a shrunken denominator.
332+
#
333+
# This is not hypothetical. Run 30236603180 (2026-07-27, the first and
334+
# so far only scheduled full sweep) hit exactly this: `a2a-server`
335+
# shards 3/8 and 4/8 were cancelled at the 120-minute job timeout, and
336+
# "Mutants Summary" still concluded **success**. Two elevenths of the
337+
# workspace went unmutated and the gate said everything was caught.
338+
#
339+
# The incremental job below already documents this same hazard in its
340+
# own header — "it gets cancelled, so the gate silently stops
341+
# enforcing" — and was sharded to avoid it. The full sweep had the
342+
# identical hole and no equivalent guard. This is that guard.
343+
#
344+
# Two independent checks, because they fail differently: a shard can
345+
# end non-success (timeout/failure/cancel), or its artifact can go
346+
# missing while the job still reports success.
347+
- name: Require every shard to have completed
348+
env:
349+
# Matrix result: 'success' only when every matrix job succeeded.
350+
SHARD_RESULT: ${{ needs.mutants-crate.result }}
351+
# Must equal the number of `include:` entries in mutants-crate
352+
# (12 a2a-server shards + types + client + sdk). Deliberately a
353+
# literal: if the matrix grows and this does not, the sweep fails
354+
# loudly here rather than quietly measuring less.
355+
EXPECTED_SHARDS: 15
356+
# A workflow_dispatch scoped to one package skips the other matrix
357+
# entries by design, so they legitimately upload nothing. Only the
358+
# unscoped sweep — the one whose score means "the whole workspace" —
359+
# is held to the full count.
360+
PACKAGE_FILTER: ${{ inputs.package }}
361+
run: |
362+
fail=0
363+
364+
if [ "$SHARD_RESULT" != "success" ]; then
365+
echo "::error::mutation shards did not all succeed (matrix result: ${SHARD_RESULT})."
366+
echo "A cancelled or timed-out shard produces an empty report, which this"
367+
echo "job would otherwise aggregate as 'zero surviving mutants'."
368+
fail=1
369+
fi
370+
371+
found=$(find reports -maxdepth 1 -type d -name 'mutation-report-*' | wc -l)
372+
if [ -n "$PACKAGE_FILTER" ]; then
373+
echo "Scoped run (package='${PACKAGE_FILTER}'): ${found} report(s); count not enforced."
374+
elif [ "$found" -ne "$EXPECTED_SHARDS" ]; then
375+
echo "::error::expected ${EXPECTED_SHARDS} per-shard reports, found ${found}."
376+
echo "Reports present:"
377+
find reports -maxdepth 1 -type d -name 'mutation-report-*' -printf ' %f\n' | sort
378+
fail=1
379+
fi
380+
381+
if [ "$fail" -ne 0 ]; then
382+
echo ""
383+
echo "The sweep is incomplete, so its score is not a measurement of the"
384+
echo "whole workspace. Re-run it, or raise the per-shard timeout /"
385+
echo "shard count, before trusting or recording any number from it."
386+
exit 1
387+
fi
388+
389+
echo "All ${EXPECTED_SHARDS} shards completed and reported."
390+
301391
- name: Aggregate results
302392
run: |
303393
count() { if [ -f "$1" ]; then grep -c . "$1" 2>/dev/null || echo 0; else echo 0; fi; }

‎book/src/reference/mutation-history.md‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,47 @@ row below, dated to the run, with the commit it ran against. A clean sweep
2626
(zero missed) is still worth recording — "still zero" is signal too, not a
2727
no-op.
2828

29+
## Why there is still no row (verified 2026-07-31)
30+
31+
The ledger below is empty, and that is not an oversight — **no full sweep has
32+
ever completed.** Checked against the Actions API rather than assumed:
33+
34+
* `mutants.yml` has exactly **one** scheduled run in its history:
35+
[30236603180](https://github.com/tomtom215/a2a-rust/actions/runs/30236603180),
36+
2026-07-27, against `b416c1a`. Run conclusion: `cancelled`.
37+
* Of its 11 sweep jobs, nine succeeded. `a2a-server` shards **3/8 and 4/8**
38+
were cancelled at the 120-minute job timeout — the shards that did finish
39+
took 86-113 minutes, so those two were only ever a few minutes from the
40+
limit.
41+
* **`Mutants Summary` nonetheless concluded `success`.** A cancelled shard
42+
still runs its upload step, so it contributes an empty `mutants.out`; the
43+
aggregator's `count()` reads zero missed mutants from it, and the job's only
44+
failure condition was `TOTAL_MISSED > 0`. Two elevenths of the workspace
45+
went unmutated and the gate reported everything caught.
46+
47+
That is the failure mode this project refuses everywhere else: a green check
48+
over work that did not happen. The same hazard was already understood for the
49+
*incremental* PR job — its header says a job that overruns "gets cancelled, so
50+
the gate silently stops enforcing" — but the full sweep had no equivalent
51+
guard.
52+
53+
Both halves are now fixed in `mutants.yml`:
54+
55+
1. **A shard-completeness gate** runs before aggregation and fails unless the
56+
matrix result is `success` *and* the expected number of per-shard reports
57+
arrived. A scoped `workflow_dispatch` (one package) is exempt from the
58+
count, since it skips the other shards by design.
59+
2. **`a2a-server` is sharded 12 ways instead of 8**, cutting per-shard load by
60+
a third so the slowest shard finishes with headroom rather than four
61+
minutes to spare.
62+
63+
So the first real row still has to come from a real, complete sweep — the next
64+
Monday run, or an on-demand `workflow_dispatch`. **Do not backfill a number
65+
from run 30236603180**: its score was computed from partial data and is not a
66+
measurement of the workspace.
67+
2968
## History
3069

3170
| Date | Commit | Overall Score | Caught | Missed | Timeout | Notes |
3271
|------|--------|---------------|-------:|-------:|--------:|-------|
33-
| _(none recorded yet)_ | | | | | | This ledger was created 2026-07-30, retroactively, before any full sweep's result had been captured into it. The next scheduled Monday sweep (or an on-demand `workflow_dispatch` run) is the first opportunity to populate a real row — do not backfill a number that wasn't actually measured. |
72+
| _(none recorded yet)_ | | | | | | No full sweep has completed. The only scheduled run (2026-07-27, `b416c1a`) lost two `a2a-server` shards to the 120-minute timeout while its summary job still reported success — see the section above. The completeness gate and 12-way sharding that fix this landed 2026-07-31; the first complete sweep after that date is the first eligible row. |

0 commit comments

Comments
 (0)