Make Reqnroll tests discoverable in release pipelines

[jangroendus]

currently, a release pipeline in Azure Devops is not able to discover Reqnroll tests (or at least I have not found a way to do so). It would be really helpful, so that tests can also be run from Release pipelines, not only build pipelines, with a connection to a manual test in Test Plan. (running the automated tests from a test plan in that release pipeline). I hope this is a feature that can be added to Reqnroll, and needs no technical changes on the MS side). Thanks for considering

[304NotModified]

Could you please elaborate on this? Do you mean the classic release pipelines, as nowadays there is only one type (multi stage) pipelines in Azure DevOps recommended? Are you getting an error?

It's just running dotnet test in the pipeline. Copy the output files (e.g dll and config files) to/from the pipeline artifacts.

I use it daily as I'm running post deployment tests with Reqnroll & Reqnroll

[jangroendus]

Sorry about formatting, not sure what happened.

Hi, I try to run reqnroll test from a testplan in a pipeline, so my vstest task is:

task: VSTest@3
  displayName: 'Test run for Test plans'
  inputs:
    testSelector: testPlan
    testPlan: 95302
    testSuite: 95323
    testConfiguration: 10
    runInParallel: true
    runTestsInIsolation: false
    codeCoverageEnabled: false
  continueOnError: true
  retryCountOnTaskFailure: 1

There are 4 tests in the test plan. 2 of them are associated with automated playwright tests in the same project, 1 is associated with a reqnroll test in the same project and 1 from another project (which cannot run, so I expect the last one not to be exucuted).
The result is:

TestExecutionHost.ProcessCommand. Start Command handled
Slice with id = 11, of type = 'Execution' received.
Count of test sources found: 2
=================================================================
Discovering tests from sources
Using new test platform for test execution
Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
Number of testcases discovered : 21
Discovered tests 21 from sources
=================================================================
[RunStatistics]This execution slice with id '11', received '4' testcases to execute out of which '2' is discovered.

The test runner discovers my 23 automated tests in the 2 test projects. One of these 23 is the associated reqnroll test. But the last line says only 2 are discovered for execution and these 2 are the only ones that are run and where results are fed back to test manager.

[gasparnagy]

I don't think we can do anything for this on Reqnroll side. We just generate standard MsTest/NUnit/etc tests, like the ones you could generate yourself.

Some things that you can check:
Are you sure that the linking is done correctly? Do you use normal scenarios (not outlines)? Have you tried disabling friendly name generation in Reqnroll using the disableFriendlyTestNames setting?

I'm also the creator of a tool SpecSync and because of that tool I was fighting with the TestPlan/TestSuite based execution feature of VSTest, but that infrastructure seems to be pretty unreliable, especially for data-driven tests, like the ones Reqnroll generates from scenario outlines, so that I had to make a workaround and substituted with a proper test result publishing mechanism in SpecSync. And Microsoft does not seem to have improved that infrastructure for many years...

[jangroendus]

tried to disablefriendlytestnames and the same result. double checked the linking, and that is correct. And I do not use outlines. So far no luck...

[jangroendus]

also tried to move back to vstest (from Microsoft.testing.platform) but that also did not work

[jangroendus]

Hello again, still no luck. If this is not a reqnroll problem, could I get suggestions on where to address this? whose problem is it?

[gasparnagy]

I would try first at https://github.com/microsoft/vstest/issues

[304NotModified]

Another approach is to check the generated testcode (by reqnroll, you could do that in Visual Studio) , move it to a new file and change until it works and/or remove all reqnroll references.

That could make it more clear why it isn't working and/or is this is a reqnroll issue.

[jangroendus]

Hello, I somehow managed to get it working (coincidentally without trying to anymore).
my pipeline tasks now look like this:

• task: VisualStudioTestPlatformInstaller@1
  inputs:
  versionSelector: 'latestStable'
• task: VSTest@3
  displayName: 'Test run for Test plans'
  inputs:
  vsTestVersion: 'toolsInstaller'
  testSelector: testPlan
  testPlan: 95302
  testSuite: 95323
  testConfiguration: 10
  searchFolder: '$(System.ArtifactsDirectory)'
  resultsFolder: '$(Agent.TempDirectory)/VSTestResults'

It did not work after I added the first task, I can rule that out.
I also removed some inputs and added the resultsFodler (the split into stages and adding the searchFolder came later where it was already working). So closed for now, but still really strange.