Skip to content

Commit 5408d69

Browse files
dsymedsyme
authored
Merge pull request #61 from fslaborg/cov
automate coverage in CI/CD
2 parents 3b06c3e + 0d0fbce commit 5408d69

File tree

9 files changed

+279
-7
lines changed

9 files changed

+279
-7
lines changed

‎.coverletrc‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"exclude": [
3+
"[*]*.Program",
4+
"[*]*Tests*",
5+
"[*]*Test*",
6+
"[*]*.*Tests",
7+
"[*]*AssemblyInfo*"
8+
],
9+
"excludebyattribute": [
10+
"Obsolete",
11+
"GeneratedCodeAttribute",
12+
"CompilerGeneratedAttribute"
13+
],
14+
"excludebyfile": [
15+
"**/bin/**/*.*",
16+
"**/obj/**/*.*"
17+
]
18+
}

‎.github/workflows/build-and-test.yml‎

Lines changed: 46 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,9 @@ on:
66
pull_request:
77
branches: [ main ]
88

9+
permissions:
10+
contents: read
11+
pull-requests: write
912

1013
jobs:
1114
build-and-test-linux:
@@ -14,15 +17,54 @@ jobs:
1417

1518
steps:
1619
- uses: actions/checkout@v4
20+
with:
21+
fetch-depth: 0
1722
- name: Setup .NET
1823
uses: actions/setup-dotnet@v4
1924
with:
2025
dotnet-version: 8.x.x
2126
- name: make script executable
2227
run: chmod u+x build.sh
23-
- name: Build and test
28+
- name: Build and test with coverage
2429
working-directory: ./
25-
run: ./build.sh runtests
30+
run: ./build.sh runtestswithcoverage
31+
32+
- name: Install ReportGenerator
33+
run: dotnet tool install -g dotnet-reportgenerator-globaltool
34+
35+
- name: Generate coverage report
36+
run: |
37+
reportgenerator \
38+
-reports:"TestResults/**/coverage.cobertura.xml" \
39+
-targetdir:"TestResults/CoverageReport" \
40+
-reporttypes:"Html;Cobertura;JsonSummary" \
41+
-verbosity:"Info"
42+
43+
- name: Code Coverage Report
44+
uses: irongut/CodeCoverageSummary@v1.3.0
45+
with:
46+
filename: TestResults/**/coverage.cobertura.xml
47+
badge: true
48+
fail_below_min: false
49+
format: markdown
50+
hide_branch_rate: false
51+
hide_complexity: true
52+
indicators: true
53+
output: both
54+
thresholds: '60 80'
55+
56+
- name: Add Coverage PR Comment
57+
uses: marocchino/sticky-pull-request-comment@v2
58+
if: github.event_name == 'pull_request'
59+
with:
60+
recreate: true
61+
path: code-coverage-results.md
62+
63+
- name: Upload coverage report
64+
uses: actions/upload-artifact@v4
65+
with:
66+
name: coverage-report
67+
path: TestResults/CoverageReport/
2668

2769
build-and-test-windows:
2870

@@ -34,6 +76,6 @@ jobs:
3476
uses: actions/setup-dotnet@v4
3577
with:
3678
dotnet-version: 8.x.x
37-
- name: Build and test
79+
- name: Build and test with coverage
3880
working-directory: ./
39-
run: ./build.cmd runtests
81+
run: ./build.cmd runtestswithcoverage

‎README.md‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
## FsMath - Fast & Friendly Numerical Foundation for F\#
22

3+
[![Build and test](https://github.com/fslaborg/FsMath/actions/workflows/build-and-test.yml/badge.svg)](https://github.com/fslaborg/FsMath/actions/workflows/build-and-test.yml)
4+
35
**FsMath** is a lightweight maths library designed for modern F# workflows.
46
It focuses on **zero-friction interop** with existing array-centric libraries ( *FSharp.Stats*, *Math.NET Numerics*, *libtorch* via TorchSharp, etc.) while giving you the performance head-room of **SIMD-accelerated** kernels and a clean, idiomatic F# API.
57

@@ -12,7 +14,7 @@ It focuses on **zero-friction interop** with existing array-centric libraries (
1214
| **Interop first** | Core types are just aliases or thin wrappers around native F# `'T []` arrays -> you can pass data to any array-based .NET or C library without conversion or pinning.|
1315
| **Predictable memory layout** | Matrices are stored **row-major** (C-style) to line up with libtorch / TorchSharp, modern BLAS back-ends, and GPU kernels. |
1416
| **SIMD out-of-the-box** | Tight loops (dot, outer product, reductions ...) auto-detect hardware with SIMD support and pure-managed fallback otherwise. |
15-
| **F#-idiomatic API** | Pipelining, module functions, inline operators (`.+`, `.*`, `transpose`) and SRTP “type-class” dispatch keep your notebooks & scripts concise. |
17+
| **F#-idiomatic API** | Pipelining, module functions, inline operators (`.+`, `.*`, `transpose`) and SRTP �type-class� dispatch keep your notebooks & scripts concise. |
1618

1719
---
1820

@@ -72,16 +74,40 @@ Check the [build project](./build) to take a look at the build targets.
7274
build.sh
7375
```
7476

75-
## running and writing tests
77+
## run tests
7678

7779
```shell
7880
# Windows
7981
./build.cmd runtests
8082

8183
# Linux/mac
82-
build.sh runtests
84+
./build.sh runtests
85+
```
86+
87+
## run tests with coverage
88+
89+
To generate code coverage reports locally:
90+
91+
```shell
92+
# Windows
93+
./build.cmd runtestswithcoverage
94+
95+
# Linux/mac
96+
./build.sh runtestswithcoverage
8397
```
8498

99+
For a complete coverage report with HTML output, use the provided scripts:
100+
101+
```shell
102+
# Windows
103+
./scripts/generate-coverage.ps1
104+
105+
# Linux/mac
106+
./scripts/generate-coverage.sh
107+
```
108+
109+
This will generate coverage reports in the `TestResults` directory and create an HTML report you can open in your browser. Coverage is automatically collected and reported in CI/CD pipelines, with coverage summaries displayed in pull requests and detailed HTML reports available as downloadable artifacts.
110+
85111
## docs
86112

87113
The docs are contained in `.fsx` and `.md` files in the `docs` folder. To develop docs on a local server with hot reload, run the following in the root of the project:

‎build/TestTasks.fs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,19 @@ let runTests = BuildTask.create "RunTests" [clean; buildSolution] {
2323
)
2424
}
2525

26+
let runTestsWithCoverage = BuildTask.create "RunTestsWithCoverage" [clean; buildSolution] {
27+
testProjects
28+
|> Seq.iter (fun testProject ->
29+
testProject
30+
|> Fake.DotNet.DotNet.test (fun testParams ->
31+
{ testParams with
32+
Logger = Some "console;verbosity=detailed"
33+
Configuration = DotNet.BuildConfiguration.fromString configuration
34+
NoBuild = true
35+
MSBuildParams = { testParams.MSBuildParams with DisableInternalBinLog = true }
36+
}
37+
|> DotNet.Options.withCustomParams (Some "--collect:\"XPlat Code Coverage\" --results-directory ./TestResults --settings ./coverlet.runsettings --logger trx --logger \"console;verbosity=detailed\"")
38+
)
39+
)
40+
}
41+

‎coverlet.runsettings‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
<?xml version="1.0" encoding="utf-8" ?>
2+
<RunSettings>
3+
<DataCollectionRunSettings>
4+
<DataCollectors>
5+
<DataCollector friendlyName="XPlat code coverage">
6+
<Configuration>
7+
<Format>cobertura</Format>
8+
<Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude>
9+
<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
10+
<ExcludeByFile>**/Views/**/*.cs,**/wwwroot/**/*.js</ExcludeByFile>
11+
<SingleHit>false</SingleHit>
12+
<UseSourceLink>true</UseSourceLink>
13+
<IncludeTestAssembly>false</IncludeTestAssembly>
14+
<SkipAutoProps>true</SkipAutoProps>
15+
<DeterministicReport>false</DeterministicReport>
16+
</Configuration>
17+
</DataCollector>
18+
</DataCollectors>
19+
</DataCollectionRunSettings>
20+
</RunSettings>

‎docs/coverage.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Coverage Analysis Setup
2+
3+
This document describes the code coverage analysis and reporting setup for FsMath.
4+
5+
## Overview
6+
7+
The project uses [Coverlet](https://github.com/coverlet-coverage/coverlet) for code coverage collection and reporting. Coverage reports are generated automatically during CI/CD builds with detailed HTML reports available as artifacts and coverage summaries in pull requests.
8+
9+
## Tools Used
10+
11+
- **Coverlet**: .NET code coverage library
12+
- **ReportGenerator**: Tool for generating HTML/XML coverage reports
13+
- **GitHub Actions**: CI/CD automation with coverage integration
14+
15+
## Local Development
16+
17+
### Running Tests with Coverage
18+
19+
```bash
20+
# Basic coverage collection
21+
./build.sh runtestswithcoverage # Linux/macOS
22+
./build.cmd runtestswithcoverage # Windows
23+
24+
# Full coverage report generation
25+
./scripts/generate-coverage.sh # Linux/macOS
26+
./scripts/generate-coverage.ps1 # Windows
27+
```
28+
29+
### Coverage Configuration
30+
31+
The coverage setup is configured through several files:
32+
33+
- `coverlet.runsettings`: MSBuild test settings for coverage collection
34+
- `.coverletrc`: Coverlet-specific configuration for exclusions
35+
- Test project files include `coverlet.msbuild` and `coverlet.collector` packages
36+
37+
### Exclusions
38+
39+
The following items are excluded from coverage analysis:
40+
- Test assemblies and test-related code
41+
- Generated code (marked with attributes)
42+
- Program entry points
43+
- Obsolete code
44+
45+
## CI/CD Integration
46+
47+
### GitHub Actions Workflow
48+
49+
The `build-and-test.yml` workflow includes:
50+
51+
1. **Test Execution**: Runs tests with coverage collection using XPlat Code Coverage
52+
2. **Report Generation**: Creates HTML and Cobertura reports using ReportGenerator
53+
3. **PR Comments**: Adds coverage summary comments to pull requests
54+
4. **Artifact Storage**: Stores HTML coverage reports as build artifacts
55+
56+
### Coverage Thresholds
57+
58+
- **Warning threshold**: 60% coverage
59+
- **Good threshold**: 80% coverage
60+
- **CI doesn't fail** on coverage below thresholds (informational only)
61+
62+
## Coverage Reports
63+
64+
### Local Reports
65+
66+
- **Location**: `TestResults/CoverageReport/`
67+
- **Format**: HTML (main), Cobertura XML, JSON summary
68+
- **View**: Open `TestResults/CoverageReport/index.html` in browser
69+
70+
### GitHub Integration
71+
72+
- **Coverage Badge**: Generated in pull request comments
73+
- **PR Integration**: Coverage changes shown in pull request comments
74+
- **Artifact Downloads**: Full HTML reports available from GitHub Actions
75+
76+
## Troubleshooting
77+
78+
### Common Issues
79+
80+
1. **No coverage data**: Ensure tests are running and `coverlet.collector` is installed
81+
2. **Low coverage**: Check exclusions in `.coverletrc` and `coverlet.runsettings`
82+
3. **Report generation fails**: Verify ReportGenerator is installed globally
83+
84+
### Debug Coverage Collection
85+
86+
```bash
87+
# Run with verbose logging
88+
dotnet test --collect:"XPlat Code Coverage" --logger "console;verbosity=detailed"
89+
90+
# Check for coverage files
91+
find TestResults -name "*.cobertura.xml"
92+
```
93+
94+
## Maintenance
95+
96+
The coverage setup should be reviewed periodically to:
97+
98+
- Update exclusion patterns as the codebase evolves
99+
- Adjust coverage thresholds based on project maturity
100+
- Update tool versions (Coverlet, ReportGenerator)
101+
- Review GitHub Actions workflow and coverage reporting

‎scripts/generate-coverage.ps1‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Coverage Report Generator for FsMath
2+
# PowerShell script for Windows
3+
4+
Write-Host "🧪 Running tests with coverage collection..." -ForegroundColor Green
5+
.\build.cmd runtestswithcoverage
6+
7+
Write-Host "📊 Installing ReportGenerator..." -ForegroundColor Yellow
8+
try {
9+
dotnet tool install --global dotnet-reportgenerator-globaltool 2>$null
10+
} catch {
11+
Write-Host "ReportGenerator already installed" -ForegroundColor Yellow
12+
}
13+
14+
Write-Host "📈 Generating coverage report..." -ForegroundColor Green
15+
& reportgenerator `
16+
-reports:"TestResults/**/coverage.cobertura.xml" `
17+
-targetdir:"TestResults/CoverageReport" `
18+
-reporttypes:"Html;Cobertura;JsonSummary;MarkdownSummary" `
19+
-verbosity:"Info"
20+
21+
Write-Host "✅ Coverage report generated!" -ForegroundColor Green
22+
Write-Host "📂 Report location: TestResults/CoverageReport/index.html" -ForegroundColor White
23+
Write-Host ""
24+
Write-Host "🌐 To view the report, open the HTML file in your browser:" -ForegroundColor Cyan
25+
Write-Host " file://$(Get-Location)/TestResults/CoverageReport/index.html" -ForegroundColor White

‎scripts/generate-coverage.sh‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
# Coverage Report Generator for FsMath
3+
4+
set -e
5+
6+
echo "🧪 Running tests with coverage collection..."
7+
./build.sh runtestswithcoverage
8+
9+
echo "📊 Installing ReportGenerator..."
10+
dotnet tool install --global dotnet-reportgenerator-globaltool 2>/dev/null || echo "ReportGenerator already installed"
11+
12+
echo "📈 Generating coverage report..."
13+
reportgenerator \
14+
-reports:"TestResults/**/coverage.cobertura.xml" \
15+
-targetdir:"TestResults/CoverageReport" \
16+
-reporttypes:"Html;Cobertura;JsonSummary;MarkdownSummary" \
17+
-verbosity:"Info"
18+
19+
echo "✅ Coverage report generated!"
20+
echo "📂 Report location: TestResults/CoverageReport/index.html"
21+
echo ""
22+
echo "🌐 To view the report, open the HTML file in your browser:"
23+
echo " file://$(pwd)/TestResults/CoverageReport/index.html"

‎tests/FsMath.Tests/FsMath.Tests.fsproj‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
<PrivateAssets>all</PrivateAssets>
4545
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
4646
</PackageReference>
47+
<PackageReference Include="coverlet.msbuild" Version="6.0.4" />
4748
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />
4849
<PackageReference Include="xunit" Version="2.9.3" />
4950
<PackageReference Include="xunit.runner.visualstudio" Version="3.0.2">

0 commit comments

Comments
 (0)