[WIP] Simulation tests for coverage + p-values

[johnmyleswhite]

This is a work-in-progress project to add compute-intensive simulation testing to HypothesisTests.jl to assess the coverage probabilities of nominal 95% CI's and the uniform distribution of p-values under the null.

So far, the results are as follows:

• Currently passing tests
  • Kolmogorov-Smirnov tests
  • Kruskal-Wallis tests
  • Mann-Whitney tests
  • t-tests
  • z-tests (although increasing the number of simulations reveals expected small failures)
• Currently failing tests
  • Anderson-Darling tests (known to have problems)
  • Binomial tests (known to be too conservative)
• Still untested
  • Circular tests
  • Fisher exact tests
  • Power-divergence tests
  • Wilcoxon tests

Thoughts on the design of the simulations and action items based on the results would be very appreciated.

[johnmyleswhite]

One question that arises is what properties we really want to guarantee for our CI's. In the literature, there's a divide between conservative procedures (for which the empirical coverage probability is higher than the nominal coverage), anti-conservative procedures (which are essentially just broken as they don't provide at least the expected coverage) and calibrated procedures, which produce the expected coverage. We could check that CI's are not anti-conservative by checking that empirical coverage is always above nominal coverage (up to noise). That would get the Bernoulli stuff passing, although I think we'll still produce non-uniform p-values and I think we won't fix the Anderson-Darling test, which is known to not work well in practice.