Skip to content

Commit c467d2b

Browse files
radicalCopilot
radical
and
Copilot
committed
Reduce PR validation time by decoupling CLI archive dependencies (#15023)
* Build CLI archive bundle dependencies independently Let the CLI archive workflow produce its own Dashboard and DCP bundle dependencies so it can run in parallel with build_packages without waiting on that job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim build_packages work during CLI archive builds Skip redundant extension, Dashboard, DCP, and Aspire.Cli work from build_packages when the CLI archive workflow is already producing those artifacts in parallel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Split CLI archive builds by platform Break the CLI archive build into linux and non-linux targets so linux-only consumers can start sooner, and remove the unnecessary dependency on the non-linux archive job from tests_requires_nugets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use runner.os for CLI archive build conditionals Switch the CLI archive workflow conditionals over to runner.os so they still evaluate correctly after the matrix was reshaped around per-target objects. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Split requires-nugets tests by archive platform Route requires-nugets tests to the platform-specific CLI archive jobs and teach the matrix splitting script about the per-OS archive keys they now depend on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix infrastructure tests for per-platform archive keys Update the infrastructure tests so they validate the new per-OS requires-nugets keys and compile cleanly against the split archive matrix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove stale test summary timing scaffolding Drop the unused infra timing and dependency map code from tests.yml now that it no longer reflects the current CI pipeline behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Route RID-specific NuGets through CLI archive jobs Upload the RID-specific DCP and Dashboard packages from the CLI archive workflow itself, and remove the dead staging and fan-out logic from build_packages now that those artifacts are no longer produced there. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Document the CI pipeline optimizations Explain why the CLI archive build was decoupled and split by platform, how the workflow now behaves, and what latency benefits the pipeline gets from the change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Ensure specialized tests get RID-specific NuGets Run the CLI archive workflow alongside build_packages in the specialized test runner so requiresNugets lanes still receive the arch-specific bundle artifacts that run-tests.yml downloads. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Scope bundle dependency builds to requested RIDs Restrict BuildBundleDepsOnly to the DCP and Dashboard pack projects for the requested TargetRids, and update the Build.props property descriptions to match the actual build behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 570921d commit c467d2b

File tree

10 files changed

+472
-167
lines changed

10 files changed

+472
-167
lines changed

‎.github/workflows/build-cli-native-archives.yml‎

Lines changed: 58 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,18 @@ on:
1010
required: false
1111
type: string
1212
default: Debug
13+
targets:
14+
description: >
15+
JSON array of target objects to build. Each object must have 'os', 'runner', and 'rids' keys.
16+
Defaults to all platforms (linux-x64, win-x64, osx-arm64).
17+
required: false
18+
type: string
19+
default: >-
20+
[
21+
{"os": "ubuntu-latest", "runner": "8-core-ubuntu-latest", "rids": "linux-x64"},
22+
{"os": "windows-latest", "runner": "windows-latest", "rids": "win-x64"},
23+
{"os": "macos-latest", "runner": "macos-latest", "rids": "osx-arm64"}
24+
]
1325
1426
jobs:
1527

@@ -18,30 +30,58 @@ jobs:
1830
runs-on: ${{ matrix.targets.runner }}
1931
strategy:
2032
matrix:
21-
targets:
22-
- os: ubuntu-latest
23-
runner: 8-core-ubuntu-latest
24-
rids: linux-x64
25-
- os: windows-latest
26-
runner: windows-latest
27-
rids: win-x64
28-
- os: macos-latest
29-
runner: macos-latest
30-
rids: osx-arm64
33+
targets: ${{ fromJson(inputs.targets) }}
3134

3235
steps:
3336
- name: Checkout code
3437
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
3538

36-
# Download RID-specific NuGet packages (for DCP) used by Bundle.proj
37-
- name: Download RID-specific NuGets
38-
uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
39+
# Build RID-specific NuGet packages (DCP + Dashboard) locally instead of
40+
# downloading them from the build_packages job. This allows the CLI archive
41+
# build to run in parallel with build_packages.
42+
- name: Build RID-specific packages (Windows)
43+
if: ${{ runner.os == 'Windows' }}
44+
shell: pwsh
45+
run: >
46+
.\build.cmd
47+
-ci
48+
-restore
49+
-build
50+
-pack
51+
-configuration ${{ inputs.configuration }}
52+
/bl:${{ github.workspace }}/artifacts/log/${{ inputs.configuration }}/BuildBundleDeps.binlog
53+
/p:ContinuousIntegrationBuild=true
54+
/p:BuildBundleDepsOnly=true
55+
${{ inputs.versionOverrideArg }}
56+
57+
- name: Build RID-specific packages (Unix)
58+
if: ${{ runner.os != 'Windows' }}
59+
shell: bash
60+
run: >
61+
./build.sh
62+
--ci
63+
--restore
64+
--build
65+
--pack
66+
--configuration ${{ inputs.configuration }}
67+
/bl:${{ github.workspace }}/artifacts/log/${{ inputs.configuration }}/BuildBundleDeps.binlog
68+
/p:ContinuousIntegrationBuild=true
69+
/p:BuildBundleDepsOnly=true
70+
${{ inputs.versionOverrideArg }}
71+
72+
# Upload DCP + Dashboard NuGets so test/polyglot jobs can download them
73+
- name: Upload RID-specific NuGets
74+
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
3975
with:
4076
name: built-nugets-for-${{ matrix.targets.rids }}
41-
path: artifacts/packages/Release/Shipping
77+
path: |
78+
artifacts/packages/${{ inputs.configuration }}/Shipping/Aspire.Hosting.Orchestration.${{ matrix.targets.rids }}.*.nupkg
79+
artifacts/packages/${{ inputs.configuration }}/Shipping/Aspire.Dashboard.Sdk.${{ matrix.targets.rids }}.*.nupkg
80+
if-no-files-found: error
81+
retention-days: 15
4282

4383
- name: Build bundle payload archive (Windows)
44-
if: ${{ matrix.targets.os == 'windows-latest' }}
84+
if: ${{ runner.os == 'Windows' }}
4585
shell: pwsh
4686
run: >
4787
.\dotnet.cmd
@@ -57,7 +97,7 @@ jobs:
5797
${{ inputs.versionOverrideArg }}
5898
5999
- name: Build bundle payload archive (Unix)
60-
if: ${{ matrix.targets.os != 'windows-latest' }}
100+
if: ${{ runner.os != 'Windows' }}
61101
shell: bash
62102
run: >
63103
./dotnet.sh
@@ -73,7 +113,7 @@ jobs:
73113
${{ inputs.versionOverrideArg }}
74114
75115
- name: Build CLI packages (Windows)
76-
if: ${{ matrix.targets.os == 'windows-latest' }}
116+
if: ${{ runner.os == 'Windows' }}
77117
shell: pwsh
78118
run: >
79119
.\build.cmd
@@ -89,7 +129,7 @@ jobs:
89129
${{ inputs.versionOverrideArg }}
90130
91131
- name: Build CLI packages (Unix)
92-
if: ${{ matrix.targets.os != 'windows-latest' }}
132+
if: ${{ runner.os != 'Windows' }}
93133
shell: bash
94134
run: >
95135
./build.sh

‎.github/workflows/build-packages.yml‎

Lines changed: 3 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -6,94 +6,31 @@ on:
66
versionOverrideArg:
77
required: false
88
type: string
9-
outputs:
10-
arch_rids:
11-
description: JSON array of architecture-specific RIDs discovered during packaging
12-
value: ${{ jobs.build_packages.outputs.arch_rids }}
139

1410
jobs:
1511
build_packages:
1612
name: Build packages
1713
runs-on: 8-core-ubuntu-latest
18-
outputs:
19-
arch_rids: ${{ steps.stage_rid_specific.outputs.rids }}
2014
steps:
2115
- name: Checkout code
2216
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
2317

24-
- name: Install VSCE tool
25-
run: npm i -g @vscode/vsce@3.7.1
26-
2718
- name: Build with packages
28-
run: ./build.sh -restore -build -build-extension -ci -pack -bl -p:InstallBrowsersForPlaywright=false -p:SkipTestProjects=true -p:SkipPlaygroundProjects=true ${{ inputs.versionOverrideArg }}
19+
run: ./build.sh -restore -build -ci -pack -bl -p:InstallBrowsersForPlaywright=false -p:SkipTestProjects=true -p:SkipPlaygroundProjects=true -p:SkipBundleDeps=true ${{ inputs.versionOverrideArg }}
2920

3021
- name: Clean up artifacts
3122
run: |
3223
rm -rf artifacts/bin
3324
rm -rf artifacts/obj
3425
35-
- name: Stage RID-specific NuGets and remaining packages
36-
id: stage_rid_specific
37-
shell: bash
38-
run: |
39-
set -euo pipefail
40-
set -x
41-
shopt -s nullglob
42-
mkdir -p staging/built-nugets staging/rid
43-
# Copy full packages tree first (so structure expected by downstream tests is preserved)
44-
rsync -a artifacts/packages/ staging/built-nugets/
45-
46-
declare -A RID_SET=()
47-
# Find target packages (dcp and dashboard)
48-
while IFS= read -r -d '' pkg; do
49-
bn="$(basename "$pkg")"
50-
rid=""
51-
if [[ $bn =~ ^Aspire\.Hosting\.Orchestration\.([^.]+)\..*\.nupkg$ ]]; then
52-
rid="${BASH_REMATCH[1]}"
53-
elif [[ $bn =~ ^Aspire\.Dashboard\.Sdk\.([^.]+)\..*\.nupkg$ ]]; then
54-
rid="${BASH_REMATCH[1]}"
55-
else
56-
continue
57-
fi
58-
if [[ -n $rid ]]; then
59-
RID_SET["$rid"]=1
60-
mkdir -p "staging/rid/$rid"
61-
cp "$pkg" "staging/rid/$rid/"
62-
# Remove from built-nugets staging copy so it is excluded there
63-
rel="${pkg#artifacts/packages/}"
64-
rm -f "staging/built-nugets/$rel" || true
65-
fi
66-
done < <(find artifacts/packages -type f \( -name 'Aspire.Hosting.Orchestration.*.nupkg' -o -name 'Aspire.Dashboard.Sdk.*.nupkg' \) -print0)
67-
68-
# Build JSON array of RIDs
69-
if (( ${#RID_SET[@]} )); then
70-
printf '%s\n' "${!RID_SET[@]}" | sort -u > /tmp/rids.txt
71-
# Build a compact single-line JSON array (avoid pretty-print newlines which break $GITHUB_OUTPUT)
72-
json=$(jq -R . < /tmp/rids.txt | jq -s -c .)
73-
else
74-
json='[]'
75-
fi
76-
echo "Discovered RIDs: $json"
77-
# Use printf to safely write (single line) to GITHUB_OUTPUT
78-
printf 'rids=%s\n' "$json" >> "$GITHUB_OUTPUT"
79-
80-
- name: Upload built NuGets (excluding RID-specific orchestration/dashboard)
26+
- name: Upload built NuGets
8127
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
8228
with:
8329
name: built-nugets
84-
path: staging/built-nugets
30+
path: artifacts/packages
8531
if-no-files-found: error
8632
retention-days: 15
8733

88-
- name: Upload consolidated RID-specific NuGets
89-
if: ${{ steps.stage_rid_specific.outputs.rids != '[]' }}
90-
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
91-
with:
92-
name: built-nugets-for-rid-all
93-
path: staging/rid
94-
if-no-files-found: error
95-
retention-days: 1
96-
9734
# Always upload logs, even if the build fails, to enable debugging
9835
- name: Upload logs
9936
if: always()
@@ -103,27 +40,3 @@ jobs:
10340
path: artifacts/log
10441
retention-days: 5
10542

106-
upload_arch_specific_nugets:
107-
name: Upload arch-specific NuGets
108-
needs: build_packages
109-
if: ${{ needs.build_packages.outputs.arch_rids != '[]' }}
110-
runs-on: ubuntu-latest
111-
strategy:
112-
fail-fast: false
113-
matrix:
114-
rid: ${{ fromJson(needs.build_packages.outputs.arch_rids) }}
115-
steps:
116-
- name: Download consolidated RID-specific NuGets
117-
uses: actions/download-artifact@cc203385981b70ca67e1cc392babf9cc229d5806 # v4.1.9
118-
with:
119-
name: built-nugets-for-rid-all
120-
path: rid-nugets
121-
122-
- name: Upload per-RID NuGets
123-
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
124-
with:
125-
name: built-nugets-for-${{ matrix.rid }}
126-
path: rid-nugets/${{ matrix.rid }}/*.nupkg
127-
if-no-files-found: error
128-
retention-days: 15
129-

‎.github/workflows/specialized-test-runner.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -101,9 +101,15 @@ jobs:
101101
if: ${{ github.repository_owner == 'dotnet' && needs.generate_tests_matrix.outputs.requiresNugets == 'true' }}
102102
uses: ./.github/workflows/build-packages.yml
103103

104+
build_cli_archives:
105+
name: Build native CLI archives
106+
needs: [generate_tests_matrix]
107+
if: ${{ github.repository_owner == 'dotnet' && needs.generate_tests_matrix.outputs.requiresNugets == 'true' }}
108+
uses: ./.github/workflows/build-cli-native-archives.yml
109+
104110
run_tests:
105111
name: ${{ matrix.tests.project }}
106-
needs: [generate_tests_matrix, build_packages]
112+
needs: [generate_tests_matrix, build_packages, build_cli_archives]
107113
strategy:
108114
fail-fast: false
109115
matrix:
@@ -123,7 +129,7 @@ jobs:
123129
if: ${{ always() && github.repository_owner == 'dotnet' }}
124130
runs-on: ubuntu-latest
125131
name: Final Results
126-
needs: [generate_tests_matrix, build_packages, run_tests]
132+
needs: [generate_tests_matrix, build_packages, build_cli_archives, run_tests]
127133
steps:
128134
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
129135

@@ -184,7 +190,8 @@ jobs:
184190
- name: Fail if any dependency failed
185191
# 'skipped' can be when a transitive dependency fails and the dependent job gets 'skipped'.
186192
# For example, one of setup_* jobs failing and the Integration test jobs getting 'skipped'
187-
# Note: build_packages being skipped is expected when requiresNugets is false, so we check it separately
193+
# Note: build_packages/build_cli_archives being skipped is expected when requiresNugets is false,
194+
# so we check them separately.
188195
if: >-
189196
${{
190197
always() && (
@@ -194,7 +201,9 @@ jobs:
194201
needs.run_tests.result == 'cancelled' ||
195202
needs.run_tests.result == 'skipped' ||
196203
(needs.build_packages.result == 'failure') ||
197-
(needs.build_packages.result == 'cancelled')
204+
(needs.build_packages.result == 'cancelled') ||
205+
(needs.build_cli_archives.result == 'failure') ||
206+
(needs.build_cli_archives.result == 'cancelled')
198207
)
199208
}}
200209
run: |

0 commit comments

Comments
 (0)