Develop a testing solution for a standalone `base`

[jeltsch]

The testing of base is currently done together with the testing of GHC, using GHC’s test suite driver. When base is maintained separately from GHC, it should also have its own test suite, which should be tuned towards testing a library, not a compiler.

General goals

The following goals seem worthwhile:

• The test suite uses ordinary Haskell testing technology as much as feasible.
• The test suite avoids any unnecessary complexity. In particular, using different testing “ways”, as it is done in the GHC test suite, might be overkill.

Categories of base tests

The current tests for base seem to fall into the following categories:

1. Tests that check pure code
2. Tests that check impure code whose results are predictable (in particular code about I/O and concurrency)
3. Tests that check compilation

Some tests expect the code under test or the compilation to fail in a certain manner an check for this.

It seems that a large majority of the tests is of the first category and only a tiny part of the third.

Possibilities for transforming the different base tests

It is likely that most of the tests of the first two categories can be replaced by tests that use Tasty. In particular, the following seems to hold:

• Said tests can likely be reimplemented using tasty-hunit or tasty-golden.
• Many of the tests of the first category can likely be replaced by tests that use tasty-quickcheck, which could lead to simpler tests that at the same time provide better test coverage.
• Testing for expected exceptions should better not be performed by checking exit codes and error output, as it is done at the moment, but by potentially catching and then checking exceptions within Haskell code.

Tests of the third category are more difficult to deal with but may largely or even fully belong in the GHC test suite actually.

Unfortunately, many of the current tests use special arrangements. These are mostly of the following kinds:

1. Treatment as fragile
2. Use of compiler options to pick particular optimization levels
3. Use of RTS options regulating memory use (-A, -K, -M)
   • Some tests seem to employ the -K option for making code that should use constant-bound memory fail if it uses a lot of memory.
4. Execution only when not in fast mode
5. Normalization of output to shim over platform-specific quirks
   • Apparently, this is used only for a single test: T4006.
6. Timeout adjustments
7. Platform-dependent configuration

All arrangements starting with the fourth can likely be handled by Tasty, by underlying testing frameworks like HUnit, or within the tests.

[TeofilC]

Another question here is how we would test base against the moving target of GHC-HEAD?

Two possibilities come to my mind:

• we test against the latest nightly
• we have a pinned nightly in our CI config, which gets bumped in an automated way once a week via something like dependabot