24.1.1: --baseline=skip and more mutations of expressions!

[sourcefrog]

• New: Mutate +, -, *, /, %, &, ^, |, <<, >> binary ops, and their corresponding assignment ops like +=.

• New: --baseline=skip option to skip running tests in an unmutated tree, when they're already been checked externally.

• Changed: Stop generating mutations of || and && to != and ||, because it seems to raise too many low-value false positives that may be hard to test.

• Fixed: Colors in command-line help and error messages.

[ASuciuX]

hello @sourcefrog , is there a way to toggle off the mutation of +, -, *, /, %, &, ^, |, <<, >> binary ops? They are creating a lot more cases on our end and the time required/results rapport is not necessarily the best. There is a great chance we will want to disable it as it increases a lot the workflow times. And maybe in the future, we'll implement something to somehow have it disabled in the first workflows and after a workflow is fully caught without them, run also one with the binary ops mutations.

[sourcefrog]

You can filter them out with --exclude-regex. I might add a more specific feature to get the whole genre in future.

[sourcefrog]

hello @sourcefrog , is there a way to toggle off the mutation of +, -, *, /, %, &, ^, |, <<, >> binary ops? They are creating a lot more cases on our end and the time required/results rapport is not necessarily the best. There is a great chance we will want to disable it as it increases a lot the workflow times. And maybe in the future, we'll implement something to somehow have it disabled in the first workflows and after a workflow is fully caught without them, run also one with the binary ops mutations.

I filed #288 to add a concept of "tags" on mutants as an extensible way to filter things like this.

[ASuciuX]

also, having --baseline=skip is great, but it now requires the first mutant to build the project. we were thinking of having a single workflow that just builds the project and tests i, then creates N shards which take that cache from the build and take less when doing the mutants (like it does now when it is run without baseline-skip). In that manner it takes less time on the shards, without having to run a redundant check of mutants baseline(build&test) on all of them, especially as it takes 25-30 minutes on each of them.

Ref here: #209

also, it is not a huge win, on big PRs is something like: instead of running 120-180 minutes on 8 shards, run 25 minutes first shard, then 95-155 minutes on the 8 shards. the cumulated time is 7*25=175 which is like saving 1-1.5 shards power, but not sure how much it affects irl.

[sourcefrog]

With the --in-place option that's now merged but unreleased, you should be able to do that: build in one machine and do the baseline tests, then copy that directory to many machines to run the shards.

If you did not yet try using larger workers I'd certainly recommend it

[sourcefrog]

See #262