fix(mtp): run static mutants in a dedicated test-server process

[npnelson]

Sorry if there is too much AI slop in here. This is mostly Claude Fable's work with GPT 5.5 review. I think I understand most of what's going on here. Hopefully it fits in with the bigger picture and I'm not wasting your time.

The preview MTP runner reuses one test-host process per runner for all mutant sessions, switching the active mutant through the control file. Static state in the code under test initializes once per process — under whichever mutant is active when the type first loads — which breaks verdicts in both directions:

• False Killed (score inflation): a static-initializer mutant active during the first session is baked into the host's static state for the process lifetime. The tests it breaks keep failing in every later session, so later mutants on that runner are reported Killed — including provably surviving ones and mutants in code no test executes. Because MTP coverage currently assesses every mutant against every test (#3629), one baked-in failure kills everything that follows. Observed as reproducible 98–100% scores at --concurrency 1 and as run-to-run score wobble at default concurrency.
• False Survived: a static mutant tested later on a warm host can never activate at all — its type is already initialized.

This is the static-state cross-pollution risk raised in the #3094 design discussion before the runner was built (warning: "if you get rid of per-process isolation there is no isolation left, and you cannot guarantee that mutations to static state won't bleed into next iterations"), together with the intended answer (proposed carve-out: "we know when mutants are in a static context. We can offer different levels of isolation with the least isolated method reusing the testrunner for all mutants except those that have to be isolated"). The MTP path currently consults neither IsStaticValue nor MustBeTestedInIsolation; this PR implements that carve-out.

Reproduction

Self-checking ~30-line repro (script + sources): https://gist.github.com/npnelson/0945e1a7d9bf5887c70d764a868005da — one static dictionary, IsPositive(x) => x > 0, four facts, none probing 0. Output on 4.16.0:

  RESULT: this run scored 100.00 %  (a correct run scores 75.00 %)

id  mutation          killing test            truth      this run

0  "A" -> ""         Lookup("A") == 1        Killed     Killed     ok

1  "B" -> ""         Lookup("B") == 2        Killed     Killed     ok

2  x > 0 -> x < 0    IsPositive(5)           Killed     Killed     ok

3  x > 0 -> x >= 0   none (x == 0 untested)  Survived   Killed     <-- WRONG

VERDICT: BUG REPRODUCED

Step 2: Stryker (MTP) reported id 3 Killed, score 100.00%.

Step 3: all 4 tests passed with that exact mutation applied ...

On a larger project (46 xunit.v3 tests, 53 testable mutants, 8 in static initializers; per-mutant ground truth established by hand-applying each disputed mutant and running the plain suite):

config	4.16.0 / master	this branch
default	90.57–92.45% (varies run to run)	84.91%
--concurrency 1	98.11% (reproducible)	84.91%
concurrency 1 + coverage-analysis: off	100.00% (reproducible)	84.91%

84.91% is the hand-verified true score; with the fix every per-mutant status matches it, bit-identically across configs and repeated runs. RPC traces confirmed the mechanism: before the fix, once a static mutant's session had run, every later runTests call on that host reported the same baseline failures regardless of the active mutant.

What the fix does

Mutants flagged IsStaticValue (set at compile time by mutation placement, so this also works with coverage-analysis: off) or MustBeTestedInIsolation (set by coverage analysis) run one-per-session in a dedicated test-server process:

• servers reset before the run — the fresh host reads the mutant id from the control file at startup, so the mutation is active during static initialization (the VsTest runner's early-activation semantics);
• control file cleared and servers reset after — neither the baked-in state nor a stale mutant id can leak into whatever starts a server next;
• isolated batches are split, one mutant per session — the single-id control channel would otherwise run the batch unmutated; if a split session crashes or times out, the merged batch result is flagged but carries empty test lists, so the executor's single-mutant retry path classifies the affected mutant instead of siblings' results deciding its verdict;
• non-static mutants keep full process reuse. Cost: one extra host spawn per static-mutant session; no change on the hot path.

Validation

• Stryker.vstest.slnf unit suites: 1,826 passed. MTP unit project: 202 passed, including 11 new regression tests that observe reset ordering and the activated mutant id through the real control file (crash and timeout flavors of the split-batch merge each covered).
• All five MTP integration categories (MSTestMTP, XUnitMTP, NUnitMTP, TUnit, MTPSolution) pass with baselines unchanged.
• The gist script doubles as fix verification: against this branch it prints VERDICT: no bug (75.00%, id 3 Survived).

Notes for reviewers

• ResetServerAsync became virtual as the regression-test seam.
• Interaction with per-test coverage (#3689 / #3516): complementary, different phases. Per-test(-in-isolation) capture isolates the coverage run, where no mutant is ever active; this PR isolates mutation-phase sessions. Per-test attribution alone does not stop the poisoning — a baked-in failure still kills every mutant whose covering tests include the poisoned tests (in the larger project above, the poisoned tests were exactly the disputed mutants' covering tests), and unisolated static mutants flip to reliably false-Survived instead — which mirrors why VsTest ended up with both per-test coverage and static-mutant isolation. The two also reinforce each other: per-test-in-isolation capture re-runs static initializers in every test process, so MustBeTestedInIsolation gets set more completely and more of this class is routed into isolation.
• TestMultipleMutantsAsync_RegularBatch_CurrentlyRunsUnmutated is a deliberate canary, not an endorsement: real per-test coverage is exactly what makes disjoint assessing sets — and therefore real multi-mutant batches — possible, and the single-id channel would report every batched mutant falsely Survived. The test's comment carries instructions for both futures; static-containing batches are already handled here by splitting.
• Known gap, out of scope here: there is no in-repo integration fixture exercising static-initializer mutants, so none of the integration categories would catch a regression of this bug. Adding static state to the shared TargetProject would ripple through the exact-count baselines of many categories; a dedicated category looks like the clean home for such a fixture if maintainers want one. Until then, the gist's self-checking repro covers the end-to-end behavior (it prints VERDICT: no bug against a fixed build).

Related

• #3094 — MTP epic; the design discussion that predicted this failure mode and sketched this fix
• #3629 (closed as duplicate of #3689) — aggregate coverage assesses every mutant against every test; the amplifier that turns one baked-in failure into mass false kills
• #3516 — per-test coverage for MTP (complementary; see the interaction and canary notes above)
• #3117 — VsTest runner broken on xunit.v3; context for impact, since MTP is currently the only working runner on that stack
• #3563 — separate MTP verdict-correctness report (instance-method mutant, different shape; mechanism not established by this investigation)

[Copilot]

Pull request overview

This PR addresses a correctness flaw in the Microsoft Testing Platform (MTP) runner where reusing a single test-host process across mutant sessions can “poison” static initialization state, leading to false Killed or false Survived verdicts. The change introduces a carve-out so mutants that require early activation (static initializer context / MustBeTestedInIsolation) run in a fresh dedicated test-server process per mutant session, while keeping process reuse for the normal path.

Changes:

• Route mutants flagged IsStaticValue or MustBeTestedInIsolation through a per-mutant isolated execution path that resets servers before/after each run and clears the mutant control file.
• Split isolated batches to one-mutant-per-session (single-id control channel), and merge split results while preserving executor semantics for flagged sessions.
• Add unit-test coverage that asserts reset ordering, activated mutant id, and crash/timeout merge behavior; make ResetServerAsync virtual to enable test seams.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/Stryker.TestRunner.MicrosoftTestPlatform/SingleMicrosoftTestPlatformRunner.cs	Adds “process isolation” execution path for static/isolated mutants (reset-before/after, control-file clearing) and merged-result behavior; makes ResetServerAsync virtual for testability.
src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/SingleMicrosoftTestPlatformRunnerIsolationTests.cs	Adds regression tests validating isolated execution ordering, control-file activation/clearing, batch splitting, and crash/timeout merge semantics.