feat(ci): split release builds across multiple runners; delete develop; stable becomes mainnet#2592
Open
danceratopz wants to merge 23 commits intoforks/amsterdamfrom
Open
feat(ci): split release builds across multiple runners; delete develop; stable becomes mainnet#2592danceratopz wants to merge 23 commits intoforks/amsterdamfrom
danceratopz wants to merge 23 commits intoforks/amsterdamfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #2592 +/- ##
================================================
Coverage 86.24% 86.24%
================================================
Files 599 599
Lines 36984 36984
Branches 3795 3795
================================================
Hits 31895 31895
Misses 4525 4525
Partials 564 564
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
c7711ef to
86ffd56
Compare
4 tasks
b2bec0b to
b32d8ac
Compare
…evelop Add a top-level fork-ranges list for splitting multi-fork releases across parallel runners. Bump stable to --until=Osaka (latest mainnet fork) and develop to --until=Amsterdam (current fork in development).
b32d8ac to
61875b9
Compare
…nners Split multi-fork fixture releases (stable, develop) across parallel runners using the shared fork-ranges in feature.yaml. Each runner fills its fork range to a directory, then a combine job merges the results into a single release tarball. Features without --until (e.g., benchmark with --fork) continue to build on a single runner with unchanged behavior. Also adds integration tests for the new scripts and a single-fork-range parameter set to the fill output directory tests.
Benchmark fixtures have never been included in full releases (verified against v5.3.0 and v5.4.0 release assets). Mark the feature as feature_only so it is only built via feature release tags (tests-benchmark@v*), matching actual release practice.
The setup and combine jobs use uv run but were missing the setup-uv step, causing "uv: command not found" on ubuntu-latest runners.
Use explicit name fields so GitHub Actions shows concise labels like "build (pre-cancun)" instead of dumping all matrix values.
BPO forks cannot be specified directly via --from/--until because explicit BPO forks collect too many tests, and transition fork classes don't filter correctly. Work around this by using fill-args to specify --from OsakaToBPO1AtTime15k --until Amsterdam -k "not fork_Amsterdam". Add fill_args support to fork-range entries and the build-fixtures action. Also remove BPO3-BPO5 from fork ordering (actual path is Osaka->BPO1->BPO2->Amsterdam).
The pipefail shell option causes find's SIGPIPE (from head closing the pipe after 30 lines) to fail the step. Append || true since this is just debug output.
The `fixtures_*` pattern matched both final tarballs (`fixtures_stable`) and intermediate split artifacts (`fixtures__cancun`). Use `fixtures_[!_]*` to skip double-underscore names.
Limit stable and develop to tests/istanbul/eip1344_chainid/ to quickly validate the multi-runner release pipeline end-to-end. Revert before merge.
Allow exit code 5 (NO_TESTS_COLLECTED) from fill so fork ranges with no applicable tests don't fail the build. Skip artifact upload when no fixtures are generated. Handle missing artifacts gracefully in the combine step.
The \`gh run download -p "fixtures_[!_]*"\` pattern doesn't work with the \`gh\` CLI's glob implementation. Download all \`fixtures_*\` artifacts then \`rm -rf ./artifacts/fixtures__*/\` to remove split artifacts. Also restrict release upload glob to \`*.tar.gz\` files.
The transition fork fix in #2607 makes --from=BPO1 --until=BPO2 work correctly. Remove the fill-args/fill-k override plumbing that was needed to work around the transition fork boundary bug.
61875b9 to
2960d34
Compare
… testing" This reverts commit 1fc9ee8.
Full releases previously built two tarballs (stable + develop) via a dedicated workflow. Since develop is already covered by the `bal` feature-only release for devnets, the full release reduces to a single mainnet tarball -- making it just another feature release. Changes: - Remove `develop` feature and rename `stable` to `mainnet`. - Unify evm-type: both `mainnet` and `bal` now use `evm-type: eels` directly, replacing the `stable`/`develop` indirection in evm.yaml. - Delete `release_fixture_full.yaml`; mainnet releases now use the feature workflow via `tests-mainnet@v*` tags. - Enable `fail-fast: true` on the build matrix so expensive runners are cancelled when one fork range fails.
Each release builds exactly one feature, so the combine step never iterates over multiple features. Replace \`combine_matrix\` with \`feature_name\` and \`combine_labels\` outputs. Also remove the \`--all\` mode and \`get_releasable_features()\` (dead code since the full release workflow was deleted).
Mainnet fixture releases are stable and should not be marked as pre-releases. Other features (bal, benchmark) keep \`--prerelease\`.
This reverts commit 31e0d15.
Member
Author
|
Ah @spencer-tb, this is probably missing an |
When multi-fork releases are split across parallel runners, each runner produces its own .meta/index.json. Add a classmethod that merges multiple IndexFile instances by concatenating test cases, unioning forks and fixture formats, and recomputing the root hash.
Standalone script that loads .meta/index.json from each split fixture directory and merges them via IndexFile.merge(). Called by the combine workflow step to produce a correct combined index.
Run merge_index_files.py after downloading split artifacts so the combined tarball contains a correct .meta/index.json covering all fork ranges, not just the last one copied.
Delete split_artifacts/ after merging index files to avoid running out of disk space on ubuntu-latest during tarball creation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Description
Split multi-fork fixture releases across parallel runners by fork range, then simplify the release strategy by collapsing the two release workflows into one.
Summary of changes
mainnet) are split across parallel self-hosted runners by fork range (pre-cancun, cancun, prague, osaka, ...), then combined into a single tarball.developfeature is removed;stableis renamed tomainnet.release_fixture_full.yaml) is deleted. All releases now userelease_fixture_feature.yamlwithtests-<feature>@v*tags.How we got here
The initial goal was to reduce release CI wall-clock time by splitting expensive fill jobs across parallel runners, following the pattern from #2529.
While implementing the split, we noticed that
stableanddevelopwould share identical fork-range builds (same EELS implementation, same fill-params minus--until). The deduplication logic to handle this was fragile: it silently assumed both evm-types produce the same output, and would break if they ever diverged.This led to a simpler realization: the
developrelease is redundant. Thebalfeature already fills Amsterdam for devnets, and the full release workflow exists only to bundlestable+develop. Removingdevelopeliminates the cross-feature deduplication problem entirely and lets us collapse the full and feature release workflows into one. A mainnet release is just another feature release (tests-mainnet@v6.0.0).Workflow structure
Before:
After:
The
setupjob readsfeature.yamland emits a flat build matrix of fork ranges. Each runner fills its range to a directory. Thecombinejob merges the directories and creates the final tarball. Features without fork ranges (e.g.,benchmark) skip the combine step and produce the tarball directly.Config changes
feature.yaml: top-levelfork-rangeslist defines how multi-fork features are split. Features with--untilin theirfill-paramsare automatically split across applicable ranges.evm.yaml:stable/developentries replaced with singleeelsentry.feature_only: trueflag excludes features (benchmark, bal) from--allmode (now removed, but the flag remains useful as documentation).New scripts
generate_build_matrix.py: readsfeature.yaml, emits build matrix and combine metadata for GitHub Actions.create_release_tarball.py: creates a release tarball from a merged fixture directory, usingpigzfor parallel compression when available..github/scripts/tests/(runnable viajust test-ci-scripts).PRs and Actions
py3tox env.stable+develop): validates the fork-range splitting approach before the simplification to mainnet-only.mainnetbuild WIP.✅ Checklist
toxchecks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:uvx tox -e statictype(scope):.Cute Animal Picture