Describe the bug
The MTP (Microsoft Testing Platform) test runner serializes the test selection of a testing/runTests request under the JSON property name testCases, but the MTP server-mode protocol names it tests. The server therefore never sees the selection and falls back to "no selection", i.e. it runs the entire test suite.
src/Stryker.TestRunner.MicrosoftTestPlatform/Models/RunTestsRequest.cs:
public sealed record RunTestsRequest(
[property:JsonPropertyName("runId")]
Guid RunId,
[property:JsonPropertyName("testCases")] // <-- should be "tests"
TestNode[]? TestCases = null);
The platform side is unambiguous on all three counts:
- Protocol spec (
001-protocol-intro.md): interface RunTestsParams { tests?: TestNode[], runId: GUID }
- Wire constant (
ServerMode/JsonRpc/JsonRpcMethods.cs): public const string Tests = "tests";
- Deserializer (
ServerMode/JsonRpc/SerializerUtilities.Deserializers.cs): GetOptionalPropertyFromJson(properties, JsonRpcStrings.Tests)
The likely origin is the testfx samples/Playground/ServerMode sample this client was ported from, where the C# member is named TestCases but the wire name is tests:
public sealed record RunRequest(
[property:JsonProperty("tests")]
TestNode[]? TestCases,
...
Because tests is an optional property server-side, an unrecognised testCases is dropped silently — no error, no warning, no protocol violation reported. The run just quietly executes every test.
This has been present since the MTP runner was introduced in #3404.
Why it has gone unnoticed
Two reasons:
SingleMicrosoftTestPlatformRunner.RunAssemblyTestsAsync currently always passes testUidFilter: null, so no selection is ever sent in practice. The wrong name has never actually been exercised against a real server.
- The existing round-trip test (
TestingPlatformClientTests.FakeTestServer) deserializes using Stryker's own RunTestsRequest record, so the property name matches on both sides and the test passes regardless of what the name is.
Second, related defect (blocks the same fix)
RpcJsonSerializerOptions.Default sets no DefaultIgnoreCondition, and TestNode has nullable location.* members. System.Text.Json therefore emits explicit "location.file": null for tests the framework reported without location info.
testfx's TestNode deserializer uses TryGetValue, which succeeds on an explicit JSON null, and then asserts non-null:
if (properties.TryGetValue("location.file", out object? location_file))
{
ApplicationStateGuard.Ensure(location_file is not null); // throws InvalidOperationException
This is latent today because TestNodes only ever flow server -> client. The moment Stryker starts sending them back (i.e. as soon as the tests rename makes selection actually work), any test without location info will fail the run request. Both parts need fixing together.
Impact
- Every mutant test run executes the full test suite, regardless of any selection Stryker computes.
- This is a hard blocker for per-mutant test selection and for per-test coverage analysis under the MTP runner. Combined with the current cumulative-coverage model,
CoverageBasedTest yields no test reduction at all on MTP.
Expected behavior
The test selection is serialized as tests, the server honours it and runs only the selected tests, and TestNodes serialized back to the server omit absent location properties rather than emitting them as explicit null.
Desktop:
- OS: Windows
- Type of project: core
- Framework Version: net10.0
- Stryker Version: main (
4.16.0)
Additional context
Found while assessing overall MTP runner support. Fixing this is a prerequisite for addressing the larger per-test coverage limitation, so I'd like to get it in on its own first. PR to follow.
Describe the bug
The MTP (Microsoft Testing Platform) test runner serializes the test selection of a
testing/runTestsrequest under the JSON property nametestCases, but the MTP server-mode protocol names ittests. The server therefore never sees the selection and falls back to "no selection", i.e. it runs the entire test suite.src/Stryker.TestRunner.MicrosoftTestPlatform/Models/RunTestsRequest.cs:The platform side is unambiguous on all three counts:
001-protocol-intro.md):interface RunTestsParams { tests?: TestNode[], runId: GUID }ServerMode/JsonRpc/JsonRpcMethods.cs):public const string Tests = "tests";ServerMode/JsonRpc/SerializerUtilities.Deserializers.cs):GetOptionalPropertyFromJson(properties, JsonRpcStrings.Tests)The likely origin is the testfx
samples/Playground/ServerModesample this client was ported from, where the C# member is namedTestCasesbut the wire name istests:Because
testsis an optional property server-side, an unrecognisedtestCasesis dropped silently — no error, no warning, no protocol violation reported. The run just quietly executes every test.This has been present since the MTP runner was introduced in #3404.
Why it has gone unnoticed
Two reasons:
SingleMicrosoftTestPlatformRunner.RunAssemblyTestsAsynccurrently always passestestUidFilter: null, so no selection is ever sent in practice. The wrong name has never actually been exercised against a real server.TestingPlatformClientTests.FakeTestServer) deserializes using Stryker's ownRunTestsRequestrecord, so the property name matches on both sides and the test passes regardless of what the name is.Second, related defect (blocks the same fix)
RpcJsonSerializerOptions.Defaultsets noDefaultIgnoreCondition, andTestNodehas nullablelocation.*members.System.Text.Jsontherefore emits explicit"location.file": nullfor tests the framework reported without location info.testfx's
TestNodedeserializer usesTryGetValue, which succeeds on an explicit JSON null, and then asserts non-null:This is latent today because
TestNodes only ever flow server -> client. The moment Stryker starts sending them back (i.e. as soon as thetestsrename makes selection actually work), any test without location info will fail the run request. Both parts need fixing together.Impact
CoverageBasedTestyields no test reduction at all on MTP.Expected behavior
The test selection is serialized as
tests, the server honours it and runs only the selected tests, andTestNodes serialized back to the server omit absent location properties rather than emitting them as explicitnull.Desktop:
4.16.0)Additional context
Found while assessing overall MTP runner support. Fixing this is a prerequisite for addressing the larger per-test coverage limitation, so I'd like to get it in on its own first. PR to follow.