Add weekly GitHub Action for Python command coverage - #3351
Open
Lestropie wants to merge 6 commits into
Open
Conversation
Introduces .github/workflows/weekly_python_coverage.yml, a scheduled workflow that builds MRtrix3, runs the full suite of Python script tests under coverage instrumentation, and uploads an HTML report as a GitHub artifact. Coverage is collected via the COVERAGE_PROCESS_START + sitecustomize.py mechanism so that every Python subprocess spawned by ctest automatically participates in measurement. FSL and ANTs are installed from conda-forge to satisfy the majority of external neuroimaging dependencies; tests requiring AFNI, HD-BET, or SynthStrip are allowed to fail gracefully, still contributing partial coverage. The ctest invocation is explicitly serialised (-j 1) because all script tests share the script_test_data root as their working directory and write to overlapping temporary paths. Session prompts: 1. > Implement a GitHub Action that will produce a coverage report for > Python commands. It must first build the software using cmake. The > tests themselves are typically executed using ctest, specifically > those tests to which the "script" label is ascribed. Some of these > commands rely on external neuroimaging software, which will need to > be installed. Evaluate whether it is preferable to make use of > ctest and attach the required Python modules for coverage tracking, > or to instead parse the testing structure directly, explicitly > downloading the "script_test_data" repository at the relevant > commit. The resulting coverage report should be uploaded to GitHub > as an artifact. The Action should run weekly. 2. > The Bash tests in testinig/scripts/tests/ are all executed from > the root of the "script_test_data" repository, and may delete or > overwrite contents of that directory. If this makes it unsafe to > execute tests in parallel in this environment, disable parallelism. Generated-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Trigger on Pull Request for initial evaluation - Do not install GUI dependencies or build GUI commands - Rename build directory to reflect release build (debugging symbols for C++ binaries not required in this context)
CI Action fails when trying to set up micromamba for FSL and ANTs. So for now, only get faithful test coverage of commands that have zero third-party neuroimaging dependencies; these can all be added together at a later date.
Lestropie
force-pushed
the
python_coverage_master
branch
from
May 12, 2026 11:47
56276f2 to
b0bcaa0
Compare
Member
Author
Report generated via Action:I'd like to get more detail than just overall per-file statistics. Eg. Knowledge of which lines are missed could direct creation of additional tests. |
- Include evaluation of both branches for if statements - Print line ranges not executed
Member
Author
|
OK, that looks much more useful now. Index page for generated report:Each file contains a hyperlink to another page that highlights the per-line hits and misses. Overall percentage is low right now, partly because anything with an external dependency is going to error out immediately as the Action environment is not yet set up to install them. Entirely feasible to add the workflow first, and then deal with inclusion of external dependencies later (eg. integrating the use of #3073). |
- Rename file (to eventually be more consistent with C++ coverage analysis workflow). - Disable trigger on Pull Request. - Enable manual triggering.
Member
|
Not sure I feel competent to approve this, I'm not familiar with the concept. If the point is purely to generate a report and doesn't otherwise affect the software, generated artefacts, or block merging, I suggest we merge and see how it goes...? |
Member
Author
|
Generates new artifacts, doesn't modify existing ones. On Github go to Actions, on left side hit "Show more workflows" and select "Python coverage". Select the most recent, navigate to the individual workflow, and expand "Upload coverage report", and there's a URL for the download. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes #2773 in conjunction with #2786.
Comprehensiveness of testing can be improved once #3073 is completed. There, for each external neuroimaging software dependency, a list of only those files required to execute MRtrix3 scripts will be created. Consequently, just as the container build process will install those softwares then strip out all unnecessary content, this workflow could do the same. It might also be possible to cache them just for the Action.