Skip to content

MTP test runner sends run request test selection as testCases instead of tests, so selection is silently ignored #3754

Description

@Evangelink

Describe the bug

The MTP (Microsoft Testing Platform) test runner serializes the test selection of a testing/runTests request under the JSON property name testCases, but the MTP server-mode protocol names it tests. The server therefore never sees the selection and falls back to "no selection", i.e. it runs the entire test suite.

src/Stryker.TestRunner.MicrosoftTestPlatform/Models/RunTestsRequest.cs:

public sealed record RunTestsRequest(
    [property:JsonPropertyName("runId")]
    Guid RunId,
    [property:JsonPropertyName("testCases")]   // <-- should be "tests"
    TestNode[]? TestCases = null);

The platform side is unambiguous on all three counts:

  • Protocol spec (001-protocol-intro.md): interface RunTestsParams { tests?: TestNode[], runId: GUID }
  • Wire constant (ServerMode/JsonRpc/JsonRpcMethods.cs): public const string Tests = "tests";
  • Deserializer (ServerMode/JsonRpc/SerializerUtilities.Deserializers.cs): GetOptionalPropertyFromJson(properties, JsonRpcStrings.Tests)

The likely origin is the testfx samples/Playground/ServerMode sample this client was ported from, where the C# member is named TestCases but the wire name is tests:

public sealed record RunRequest(
    [property:JsonProperty("tests")]
    TestNode[]? TestCases,
    ...

Because tests is an optional property server-side, an unrecognised testCases is dropped silently — no error, no warning, no protocol violation reported. The run just quietly executes every test.

This has been present since the MTP runner was introduced in #3404.

Why it has gone unnoticed

Two reasons:

  1. SingleMicrosoftTestPlatformRunner.RunAssemblyTestsAsync currently always passes testUidFilter: null, so no selection is ever sent in practice. The wrong name has never actually been exercised against a real server.
  2. The existing round-trip test (TestingPlatformClientTests.FakeTestServer) deserializes using Stryker's own RunTestsRequest record, so the property name matches on both sides and the test passes regardless of what the name is.

Second, related defect (blocks the same fix)

RpcJsonSerializerOptions.Default sets no DefaultIgnoreCondition, and TestNode has nullable location.* members. System.Text.Json therefore emits explicit "location.file": null for tests the framework reported without location info.

testfx's TestNode deserializer uses TryGetValue, which succeeds on an explicit JSON null, and then asserts non-null:

if (properties.TryGetValue("location.file", out object? location_file))
{
    ApplicationStateGuard.Ensure(location_file is not null);   // throws InvalidOperationException

This is latent today because TestNodes only ever flow server -> client. The moment Stryker starts sending them back (i.e. as soon as the tests rename makes selection actually work), any test without location info will fail the run request. Both parts need fixing together.

Impact

  • Every mutant test run executes the full test suite, regardless of any selection Stryker computes.
  • This is a hard blocker for per-mutant test selection and for per-test coverage analysis under the MTP runner. Combined with the current cumulative-coverage model, CoverageBasedTest yields no test reduction at all on MTP.

Expected behavior

The test selection is serialized as tests, the server honours it and runs only the selected tests, and TestNodes serialized back to the server omit absent location properties rather than emitting them as explicit null.

Desktop:

  • OS: Windows
  • Type of project: core
  • Framework Version: net10.0
  • Stryker Version: main (4.16.0)

Additional context

Found while assessing overall MTP runner support. Fixing this is a prerequisite for addressing the larger per-test coverage limitation, so I'd like to get it in on its own first. PR to follow.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions