Parallelize coverage tests in client build pipeline

[frankmueller-msft]

Summary

• Split the single sequential Coverage_tests job into parallel per-test jobs (Coverage_MochaTest, Coverage_RealsvcLocalTest) that run concurrently after the build
• Added a Merge_coverage job that downloads raw coverage data from both parallel jobs, merges it, generates the combined Cobertura report, and runs the flub report codeCoverage comparison
• Each parallel job publishes its raw nyc/.nyc_output/ as a pipeline artifact; the merge job combines them via file copy (V8 coverage files have unique UUIDs, so no conflicts)
• No changes to build-client.yml — the coverageTests parameter format is unchanged; the ${{ each }} loop handles any number of entries, enabling follow-up sharding (see Shard mocha coverage into DDS and non-DDS shards #26562)

Pipeline structure

Build → Coverage_MochaTest       ─┐
      → Coverage_RealsvcLocalTest ─┤→ Merge_coverage
      → Test_*                     │

Observed timings (build 380200)

Job	Duration
Build	20m58s
Coverage MochaTest	24m7s
Coverage RealsvcLocalTest	7m4s
Merge Coverage Reports	6m55s

The coverage critical path is Build (21m) + MochaTest (24m) + Merge (7m) = ~52m. The MochaTest job is slower than the old sequential run due to per-job setup overhead (checkout, pnpm install, artifact extraction). However, this creates the structural foundation for sharding mocha tests (#26562), which splits MochaTest into DDS (~11m) and non-DDS (~3m) shards — bringing the coverage critical path down significantly.

Key design decisions

• Merge_coverage runs with condition: succeededOrFailed() so coverage data is still collected and reported even when tests fail
• continueOnError: true on artifact download steps so a dead agent in one coverage job doesn't block the merge of the other job's data
• Explicit --temp-directory and --report-dir on c8 report rather than relying on package.json config resolution
• Merge script validates that coverage files were actually merged before proceeding to report generation — fails explicitly if the merge directory is empty

Validated

• YAML syntax validated (PyYAML safe_load)
• ADO Pipelines Preview API confirmed correct ${{ each }} expansion:
  • Coverage_MochaTest depends on build
  • Coverage_RealsvcLocalTest depends on build
  • Merge_coverage depends on build, Coverage_MochaTest, Coverage_RealsvcLocalTest
• Build 380200: all parallel coverage jobs succeeded, artifacts downloaded, coverage files merged, report generated

Test plan

• Observe two parallel coverage jobs start after Build completes
• Observe Merge_coverage starts after both coverage jobs complete
• Verify Coverage tab in ADO shows merged results (Publish Code Coverage succeeded)
• Verify flub report codeCoverage runs in merge job (Report Code Coverage Comparison succeeded)
• Compare coverage numbers with a recent main build to confirm no coverage loss

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This pull request refactors the client build pipeline to parallelize coverage test execution, splitting the sequential Coverage_tests job into separate parallel jobs for each test type, with a subsequent merge job to combine results.

Changes:

• Created parallel Coverage_${{ test.jobName }} jobs that run mocha and realsvc:local tests concurrently after the build completes
• Added Merge_coverage job that downloads coverage data from all parallel jobs, merges the raw V8 coverage files, generates a combined Cobertura report, and runs code coverage comparison
• Moved test result publishing to individual parallel jobs; merge job only handles coverage aggregation and reporting

[Copilot]

The Merge_coverage job is missing a condition to ensure it runs even if the parallel coverage jobs fail. In Azure Pipelines, by default a job only runs if all its dependencies succeed. However, coverage artifacts are published with succeededOrFailed() condition (line 672), meaning they will be available even if tests fail. The merge job should have condition: succeededOrFailed() added to ensure coverage data is collected and reported even when tests fail. Without this, if any parallel coverage job fails due to test failures, the merge job won't run and all coverage data will be lost.

[frankmueller-msft]

Already present — the Merge_coverage job has condition: succeededOrFailed() (visible in the diff at line 684).

[Copilot]

The shell command has a critical bug that will prevent coverage data from being merged correctly. The pattern "$dir"* attempts to use a glob pattern inside quotes, which prevents shell expansion. Since $dir already ends with a trailing slash (from the loop pattern nyc/.nyc_output_*/), the correct command should be cp -r "$dir"* nyc/.nyc_output/ changed to cp -r "${dir}"* nyc/.nyc_output/ or better yet cp -r "${dir%/}/"* nyc/.nyc_output/ to ensure proper path handling. However, the most robust solution would be to copy the directory contents directly using cp -r "$dir"/* nyc/.nyc_output/ which properly handles the trailing slash and doesn't rely on glob expansion outside quotes.

Suggested change

	cp -r "$dir"* nyc/.nyc_output/ 2>/dev/null || true
	cp -r "$dir"/* nyc/.nyc_output/ 2>/dev/null || true

[frankmueller-msft]

False positive. The actual code uses cp -r "$dir"/* nyc/.nyc_output/ (with /* after the variable), which is what this suggestion recommends. The glob expands correctly outside the quotes.

[Copilot]

The || true at the end of the copy command suppresses all errors, which combined with 2>/dev/null means failures will be completely silent. This is problematic because if the copy operation fails (which it will due to the shell bug on the same line), the merge job will proceed with an empty or incomplete .nyc_output directory without any warning. If the intent is to handle empty coverage directories gracefully, a better approach would be to check if the directory has contents before attempting to copy, or add validation after the merge loop to verify that files were actually copied (e.g., check that .nyc_output is not empty).

Suggested change

	echo "Merging coverage data from $dir"
	cp -r "$dir"* nyc/.nyc_output/ 2>/dev/null || true
	fi
	done
	if compgen -G "$dir"* > /dev/null; then
	echo "Merging coverage data from $dir"
	cp -r "$dir"* nyc/.nyc_output/
	else
	echo "No coverage files found in $dir; skipping."
	fi
	fi
	done
	if ! compgen -G "nyc/.nyc_output/*" > /dev/null; then
	echo "Error: No coverage files were merged into nyc/.nyc_output."
	exit 1
	fi

[frankmueller-msft]

False positive. The code doesn't use || true — it already has the exact pattern suggested here: compgen -G guards the copy, and a final non-empty check on nyc/.nyc_output/* exits with error if no files were merged.