Console Logger Verbosity

[rburnham-def]

When we Run our Tests we noticed that Reqnroll outputs the entire scenario and steps to the console. This is helpful for devs but it also means running dotnet test in your CI/CD pipeline will create a lot of noise in the build logs.

it seems like you should be able to control the dotnet test verbosity for the console logger like this

dotnet test --logger "console;verbosity=detailed"

Full Name	Short Form	Description
quiet	q	Suppresses most output, showing only the test summary.
minimal	m	The default level, showing limited information.
normal	n	Shows standard output, including which tests are being run.
detailed	d	Provides more detailed information about the test execution.
diagnostic	diag	The highest level of verbosity, providing detailed logs for debugging the test platform itself.

I tried Minimal and quite and this had no impact, i feel like minimal should maybe just output the scenario name and if it passed/failed.

Is there another way to control the console log level?

[304NotModified]

Related
https://github.com/orgs/reqnroll/discussions/805

[304NotModified]

Related file for xUnit (not sure if this is xUnit)

Reqnroll/Plugins/Reqnroll.xUnit.ReqnrollPlugin/XUnitTraceListener.cs

Line 9 in aeb2cc4

class XUnitTraceListener : AsyncTraceListener

[clrudolphi]

I believe the short answer to be 'no'.

The verbosity setting drives the behavior of the console logger, which is operating on the set of test progress events given it by the executing test framework (MsTest, NUnit, xUnit, TUnit). The test generated by Reqnroll uses the capabilities offered by the test frameworks. To my knowledge, none of them expose an API that allows Reqnroll to emit structured test progress information that would then be subject to the verbosity filtering setting of the logger.

Reqnroll's output, both that provided by Reqnroll's runtime (such as the text of Scenarios and Steps) and that of the test's themselves (via the IReqnrollOutputHelper), eventually find their way to the mechanism provided by the test framework (usually via a TestContext or similar API) for raw text output -- usually the equivalent to a console.writeline(). The usual outcome of such output is that it is not filtered by the logger's verbosity setting.

[Code-Grump]

We technically could classify all our output according to verbosity and not emit it to the test runner, but I don't believe there's an easy way to accept a log level argument from the command-line (passing values to test runners is generally painful).

Best I can imagine is via the config file (yuck) or use an environment variable (awkward.)

[rburnham-def]

Is there anything you could do with the Reqnroll.json to configure a Release build verbosity level maybe?

[gasparnagy]

@rburnham-def What you can try is to make a plugin that (conditionally) overrides the implementation of the ITraceListener interface that is set by the unit test providers (this is how MsTest sets it for example).

The only challenge is that as far as I see MsTest currently sets it in the "CustomizeTestThreadDependencies" event, and you would need to ensure somehow that your plugin is setting it after that. That might need a bit of tricks (e.g. subscribe to the event during the "RegisterGlobalDependencies" event handler, so it is the last one in the event handler list).

Try to make a first version and do some experimentation with it. Once it works and you have gathered the first experience, we can think about how it could be made more easy.

[clrudolphi]

That will work when one would like to omit all of the WriteToolOutput (all or none).
What @Code-Grump and I were attempting to say is that for a thorough implementation, we would have to assign a verbosity level to each call made to WriteToolOutput throughout the codebase in order to be able to filter by verbosity level. (And that verbosity level would be distinct from the one assigned to the test framework logger(s).)

Apologies if we (I) over-complicated it.
@rburnham-def please let us know if you run into any challenges; we're here to help.

[rburnham52]

I thought i would have a play around with what adding a trace level filter might look like. #1025. any feedback would be apricated, i still need to test it manually though