feat(timeouts): Calculate timeout values based on actual mutant runtimes - #3731
Open
richardwerkman wants to merge 5 commits into
Open
feat(timeouts): Calculate timeout values based on actual mutant runtimes#3731richardwerkman wants to merge 5 commits into
richardwerkman wants to merge 5 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates timeout estimation to be based on the runtime of the tests that actually cover the mutant(s) being executed (instead of the full test suite), and introduces a configurable timeout multiplier (timeout-ratio) to tune how aggressively Stryker cancels long-running mutants.
Changes:
- Refine VsTest + MTP timeout estimation to use the union of assessing tests for the mutant group being executed.
- Add
TimeoutRatio/timeout-ratioconfiguration support and thread it intoTimeoutValueCalculator. - Update help-text formatting (invariant culture) and documentation/tests to reflect the new timeout model and defaults.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/Stryker.TestRunner.VsTest/VsTestRunner.cs | Estimate timeouts using distinct covering tests for the current mutant session. |
| src/Stryker.TestRunner.MicrosoftTestPlatform/SingleMicrosoftTestPlatformRunner.cs | Estimate assembly timeout from assessing tests (and pass mutants through runner pipeline). |
| src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/SingleMicrosoftTestPlatformRunnerTests.cs | Add unit tests validating per-mutant/per-group timeout estimation behavior. |
| src/Stryker.Core/Stryker.Core/Initialisation/TimeoutValueCalculator.cs | Make timeout ratio configurable (constructor parameter) and expose a default ratio constant. |
| src/Stryker.Core/Stryker.Core/Initialisation/InitialTestProcess.cs | Pass configured TimeoutRatio into TimeoutValueCalculator. |
| src/Stryker.Core/Stryker.Core.UnitTest/Options/StrykerInputsTests.cs | Ensure TimeoutRatioInput is included in inputs wiring tests. |
| src/Stryker.Core/Stryker.Core.UnitTest/Options/Inputs/AdditionalTimeoutMsInputTests.cs | Update additional-timeout default/help-text expectations. |
| src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/TimeoutValueCalculatorTests.cs | Add coverage for configured ratio affecting computed timeout. |
| src/Stryker.Configuration/Options/StrykerOptions.cs | Add TimeoutRatio option to the validated options model. |
| src/Stryker.Configuration/Options/StrykerInputs.cs | Wire TimeoutRatioInput into validation pipeline and input surface. |
| src/Stryker.Configuration/Options/Inputs/AdditionalTimeoutInput.cs | Update default additional-timeout value and description text. |
| src/Stryker.Configuration/Options/InputDefinition.cs | Format defaults using invariant culture (stabilize help text for fractional defaults). |
| src/Stryker.CLI/Stryker.CLI/FileConfigWriter.cs | Persist timeout-ratio into generated config output. |
| src/Stryker.CLI/Stryker.CLI/FileConfigReader.cs | Read timeout-ratio from config into inputs. |
| src/Stryker.CLI/Stryker.CLI/FileBasedInput.cs | Add timeout-ratio JSON/YAML config field. |
| src/Stryker.CLI/Stryker.CLI.UnitTest/StrykerCLIInitCommandTests.cs | Assert init config includes default timeout-ratio. |
| src/Stryker.CLI/Stryker.CLI.UnitTest/filled-stryker-config.yaml | Add timeout-ratio to filled-config test fixture (YAML). |
| src/Stryker.CLI/Stryker.CLI.UnitTest/filled-stryker-config.json | Add timeout-ratio to filled-config test fixture (JSON). |
| src/Stryker.CLI/Stryker.CLI.UnitTest/FileConfigReaderTests.cs | Validate timeout-ratio is read from JSON/YAML configs. |
| src/Stryker.CLI/Stryker.CLI.UnitTest/ConfigBuilderTests.cs | Ensure mocked inputs include TimeoutRatioInput. |
| src/Stryker.Abstractions/Options/IStrykerOptions.cs | Expose TimeoutRatio on the public options interface. |
| docs/configuration.md | Document per-mutant timeout calculation and new timeout-ratio setting. |
Comments suppressed due to low confidence (1)
src/Stryker.TestRunner.MicrosoftTestPlatform/SingleMicrosoftTestPlatformRunner.cs:643
CalculateAssemblyTimeoutnow estimates runtime from only the mutants’ assessing tests, butRunAssemblyTestsAsyncstill executes all discovered tests by passingtestUidFilter: nullintoRunAssemblyTestsInternalAsync. That can underestimate the timeout and introduce spurious timeouts when non-assessing tests are slow.
To align execution with the estimate (and the PR’s intent), build a testUidFilter from mutants.AssessingTests and pass it to RunAssemblyTestsInternalAsync when not EveryTest.
var discoveredTests = GetDiscoveredTests(assembly);
TimeSpan? timeout = null;
if (timeoutCalc is not null && discoveredTests is not null)
{
timeout = CalculateAssemblyTimeout(discoveredTests, timeoutCalc, assembly, mutants);
}
var (testResults, timedOut) = await RunAssemblyTestsInternalAsync(assembly, null, timeout).ConfigureAwait(false);
richardwerkman
commented
Jul 19, 2026
|
|
||
| ### `additional-timeout` <`number`> | ||
|
|
||
| Default: `5000` |
Member
Author
There was a problem hiding this comment.
I'm not sure if we should lower this. Timeout values will already be lower because of the actual mutants and this is also significantly lower. It could increase the chance for false timeouts
richardwerkman
changed the title
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #3653
Changes: