Skip to content

fix: report baseline compile failures instead of an internal error - #3715

Open
anagnorisis2peripeteia wants to merge 1 commit into
stryker-mutator:masterfrom
anagnorisis2peripeteia:fix/report-baseline-compile-failure
Open

fix: report baseline compile failures instead of an internal error#3715
anagnorisis2peripeteia wants to merge 1 commit into
stryker-mutator:masterfrom
anagnorisis2peripeteia:fix/report-baseline-compile-failure

Conversation

@anagnorisis2peripeteia

@anagnorisis2peripeteia anagnorisis2peripeteia commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

When a source generator (e.g. Microsoft.Windows.CsWin32) produces no output during Stryker's mutation compilation, the unmutated file that consumes the generated symbols fails to compile. Rollback can't attribute those errors to any mutant, so Stryker aborts — but the message it prints is "Stryker.NET could not compile the project after mutation. This is probably an error for Stryker.NET and not your project.", which points people at the wrong thing. That's the reporting gap noted in #3698.

The cause: in CSharpRollbackProcess.RollbackMutationsInError, once rollback can't recover a compile error it always throws "Internal error due to compile error." and logs the "probably a Stryker.NET error" critical — even when the erroring file contains no mutants at all, i.e. the project simply doesn't build independently of mutation.

The change: when an erroring syntax tree carries no mutants (MutantPlacer.GetAllMutations(...) is empty), report the failure naming the file and diagnostic codes, and point at the likely cause (a generator that emitted nothing), without asserting the user's project is broken — keeping a "this may be a Stryker.NET issue worth reporting" fallback.

Proving it's actually mutation-independent: before concluding a mutant-free failure is a baseline problem, Stryker now rebuilds a genuinely uninstrumented baseline and recompiles it (ICompilationContent.CompilesWithoutMutations()). "Uninstrumented" means reversing not just the mutations but also the structural rewrites Stryker applies to host them (e.g. converting an expression-bodied member to a block body), then re-running the source generators on that clean source so a syntax-sensitive generator sees the original code. If that baseline builds, the failure was mutation-induced (rollback just couldn't attribute it) and stays an internal error; only a failure that persists with no instrumentation at all is reported as a baseline failure. With today's mutators this always confirms "baseline" here — Stryker only places runtime-wrapped mutations and never mutates the compile-time constructs that could break a different file (const/enum/attribute/parameter-default) — but the recompile keeps the classification sound if that ever changes.

Injected helpers: Stryker's own injected helper files (MutantControl etc.) are also mutant-free, but a compile failure there really is an internal Stryker problem, so those keep the internal-error message (an internal, immutable check, not public API). Trees that still carry mutants keep the existing internal-error path unchanged.

Real-behaviour proof (the CsWin32 source-generator repro from #3698). A minimal project that uses Microsoft.Windows.CsWin32 to generate a P/Invoke consumed by Win32Anchor.cs. It builds fine with dotnet build, but under Stryker's mutation compilation CsWin32 emits nothing, so Win32Anchor.cs (which has no mutants) fails to compile. Packing Stryker.CLI from master and from this branch, then running each against the repro test project with the identical command:

# from repros/tests/CsWin32Repro.Tests
dotnet <packed-Stryker.CLI.dll> --mutate '**/Target.cs' --reporter json

Base (433fde6a) — misdiagnoses it as Stryker's fault, preceded by contradictory warnings:

[WRN] An unidentified mutation in Win32Anchor.cs resulted in a compile error (id: CS0234) ...
[INF] Safe Mode! Stryker will remove all mutations in ...
[FTL] Stryker.NET could not compile the project after mutation. This is probably an error for Stryker.NET and not your project. Please report this issue on github with the previous error message.

This branch — reverses all instrumentation, re-runs the generators on the clean source, sees Win32Anchor.cs still fails (mutation-independent), and reports it plainly with no contradictory warnings:

[ERR] Compilation failed in 'Win32Anchor.cs' (CS0234, CS0246, CS0103), which has no mutants, so these errors are not caused by mutation. This usually means a build input is missing under Stryker's compilation (for example a source generator produced no output; check analyzer configs / generator setup). If it builds fine outside Stryker.NET, this may be a Stryker.NET issue worth reporting on github.

Both runs were captured to files; the exact before/after transcripts (and their sha256) are recorded in the branch review-history gist below, alongside the local CI, mutation, and generator-baseline test results.

Testing: unit tests in CSharpRollbackProcessTests cover the branch — an unmutated file that fails to compile gets the baseline message (with joined diagnostic codes, and a generic label when it has no FilePath); a mutant-carrying tree still gets the internal-error message; an injected-helper file gets the internal-error message; a mutant-free tree co-erroring with a mutant-carrying tree is deferred rather than misreported; and CompilesWithoutMutations is covered both ways (true when removing mutations fixes the build, false when the failure is mutation-independent) plus the branch where a baseline that compiles keeps the internal error. Verified end-to-end against the CsWin32 repro (the before/after above).

Refs #3698. AI-assisted, but the before/after is reproducible so you don't have to take my word for it.

Review history: https://gist.github.com/387d71b3928dfead781013182e7286ec

@anagnorisis2peripeteia
anagnorisis2peripeteia force-pushed the fix/report-baseline-compile-failure branch 10 times, most recently from 1208a5b to bed7629 Compare July 16, 2026 17:07
When rollback cannot attribute a compile error to a mutant and the erroring
file contains no mutants, the failure is not a mutation problem: the project
does not compile independently of Stryker (e.g. a source generator produced
no output). Report that as a baseline failure naming the file and diagnostic
codes, rather than the misleading "internal error for Stryker.NET".

Before concluding a mutant-free failure is baseline, prove it: rebuild the
user's original source only - drop generated trees and Stryker's injected
helper trees, reverse every mutation AND the structural rewrites Stryker
applies to host them (e.g. expression-body to block-body), then re-run the
source generators on that clean source and recompile. If it builds, the
failure was mutation-induced (rollback just could not attribute it) and stays
an internal error; only a failure that persists on the fully-uninstrumented
baseline is reported as baseline. This baseline capability is an internal
interface (no new public API); content that cannot provide it keeps the
conservative internal-error path.

Stryker's own injected helper files (MutantControl etc.) are also mutant-free,
but a compile failure there is an internal Stryker problem, so those keep the
internal-error path. Helper detection is an internal, immutable check.

Refs stryker-mutator#3698
@anagnorisis2peripeteia
anagnorisis2peripeteia force-pushed the fix/report-baseline-compile-failure branch from bed7629 to 26a6f46 Compare July 16, 2026 18:01
anagnorisis2peripeteia added a commit to anagnorisis2peripeteia/marmorkrebs that referenced this pull request Jul 17, 2026
Recognise the concrete, reliable equivalent-mutant classes from the
Stryker.NET PR (stryker-mutator/stryker-net#3715) so they stop surfacing as
false survivors / reviewer noise:

- logging-only: a string/removal mutation on a bare, result-discarded
  ILogger/ILog-style call (message not asserted -> no observable change)
- attribute context: a mutation on an attribute-only line (no runtime effect)
- manual: an in-source `// marmorkrebs-ok[: reason]` reviewer directive

Classification is report/source-driven (reads the mutant's own source span
from the mutation-testing-elements `source`), not a Roslyn AST pass — that
belongs in the Stryker.NET fork. This keeps it self-contained, lane-extensible
and unit-testable.

New `--classify-equivalent <off|annotate|suppress>` (default annotate):
- annotate: flag survivors via SurvivingMutant.likelyEquivalent, score unchanged
- suppress: drop heuristic matches from the threshold denominator
- a `// marmorkrebs-ok` directive is authoritative and suppresses in any mode but off

Fail-closed: heuristic guesses never silently inflate a score (annotate is the
default; only explicit suppress or a human directive removes a survivor), and an
all-equivalent-suppressed run still trips the vacuous-run guard rather than
passing 100%. Distinct from the existing stryker-cxx `--equivalent-suppression`
binary passthrough.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant