Github driven automated testing

[MrBunsy]

This is a slightly vague investigation ticket as we need to capture some requirements before writing a more detailed set of tasks.

Goal: Each PR will be required to pass an automated test to be merged.

TODO:

• Define what is tested
• Write tests (assume Robotframework + Playwright as per Novartis, worth investigating current selenium tests)
• Link up Jenkins and Github (I know it can be done, uncertain of exact mechanisms for public github and private jenkins instance)

General thoughts:

PR will be best with only a minimal, quick, level of "smoke test" - check all the major features to ensure nothing is completely broken. The aim is to be more rigorous than what we'd usually do to investigate a PR on a dev box.

Decide if we want one test with a scope argument, or multiple tests. Could have multiple test suites and decide which of those are run by specific jenkins jobs?

[MrBunsy]

Current thoughts on testing: Playwright is great and definitely better than Selenium (it generally detects when changes are occurring and waits. Sleeps are only needed in extremely rare circumstances). However I've grown less impressed with RobotFramework after using it more. It's hard to add custom code, so writing lots of similar-but-slightly-different tests involves a lot of copy-pasting code. There is an occasional bug where it corrupts its output, and all the bug reports on the project are blamed on users.

On the other hand, I (and Ed) have used pytest before and found it far more capable and reliable. It's also relatively easy to debug, being plain python. The downside is a bit more boilerplate and setup code required, but as soon as the tests grow beyond simple I think it's less work overal.

I'd recommend PyTest + Playwright.