Skip to content

feat(ci): split release builds across multiple runners; delete develop; stable becomes mainnet#2592

Open
danceratopz wants to merge 23 commits intoforks/amsterdamfrom
experiments/multi-runner-releases
Open

feat(ci): split release builds across multiple runners; delete develop; stable becomes mainnet#2592
danceratopz wants to merge 23 commits intoforks/amsterdamfrom
experiments/multi-runner-releases

Conversation

@danceratopz
Copy link
Copy Markdown
Member

@danceratopz danceratopz commented Mar 30, 2026

🗒️ Description

Split multi-fork fixture releases across parallel runners by fork range, then simplify the release strategy by collapsing the two release workflows into one.

Summary of changes

  • Multi-fork releases (e.g., mainnet) are split across parallel self-hosted runners by fork range (pre-cancun, cancun, prague, osaka, ...), then combined into a single tarball.
  • The develop feature is removed; stable is renamed to mainnet.
  • The full release workflow (release_fixture_full.yaml) is deleted. All releases now use release_fixture_feature.yaml with tests-<feature>@v* tags.
  • Mainnet releases are marked as regular releases (not pre-releases). Other features (bal, benchmark) remain pre-releases.

How we got here

The initial goal was to reduce release CI wall-clock time by splitting expensive fill jobs across parallel runners, following the pattern from #2529.

While implementing the split, we noticed that stable and develop would share identical fork-range builds (same EELS implementation, same fill-params minus --until). The deduplication logic to handle this was fragile: it silently assumed both evm-types produce the same output, and would break if they ever diverged.

This led to a simpler realization: the develop release is redundant. The bal feature already fills Amsterdam for devnets, and the full release workflow exists only to bundle stable + develop. Removing develop eliminates the cross-feature deduplication problem entirely and lets us collapse the full and feature release workflows into one. A mainnet release is just another feature release (tests-mainnet@v6.0.0).

Workflow structure

Before:

features -> build (matrix: feature) -> release

After:

setup -> build (matrix: fork-range) -> combine -> release

The setup job reads feature.yaml and emits a flat build matrix of fork ranges. Each runner fills its range to a directory. The combine job merges the directories and creates the final tarball. Features without fork ranges (e.g., benchmark) skip the combine step and produce the tarball directly.

Config changes

  • feature.yaml: top-level fork-ranges list defines how multi-fork features are split. Features with --until in their fill-params are automatically split across applicable ranges.
  • evm.yaml: stable/develop entries replaced with single eels entry.
  • New feature_only: true flag excludes features (benchmark, bal) from --all mode (now removed, but the flag remains useful as documentation).

New scripts

  • generate_build_matrix.py: reads feature.yaml, emits build matrix and combine metadata for GitHub Actions.
  • create_release_tarball.py: creates a release tarball from a merged fixture directory, using pigz for parallel compression when available.
  • Both scripts have integration tests in .github/scripts/tests/ (runnable via just test-ci-scripts).

PRs and Actions

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx tox -e static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

Cute Animal Picture

image

@danceratopz danceratopz marked this pull request as draft March 30, 2026 15:28
@danceratopz danceratopz added C-feat Category: an improvement or new feature A-ci Area: Continuous Integration labels Mar 30, 2026
@danceratopz danceratopz changed the title chore(ci): split release fixture builds across multiple fork-range runners feat(ci): split release fixture builds across multiple fork-range runners Mar 30, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.24%. Comparing base (1e49423) to head (56b3910).
⚠️ Report is 2 commits behind head on forks/amsterdam.

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #2592   +/-   ##
================================================
  Coverage            86.24%   86.24%           
================================================
  Files                  599      599           
  Lines                36984    36984           
  Branches              3795     3795           
================================================
  Hits                 31895    31895           
  Misses                4525     4525           
  Partials               564      564           
Flag Coverage Δ
unittests 86.24% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@danceratopz danceratopz force-pushed the experiments/multi-runner-releases branch 4 times, most recently from c7711ef to 86ffd56 Compare March 31, 2026 15:33
@danceratopz danceratopz force-pushed the experiments/multi-runner-releases branch 2 times, most recently from b2bec0b to b32d8ac Compare April 1, 2026 14:09
…evelop

Add a top-level fork-ranges list for splitting multi-fork releases
across parallel runners. Bump stable to --until=Osaka (latest mainnet
fork) and develop to --until=Amsterdam (current fork in development).
@danceratopz danceratopz force-pushed the experiments/multi-runner-releases branch from b32d8ac to 61875b9 Compare April 1, 2026 19:25
…nners

Split multi-fork fixture releases (stable, develop) across parallel
runners using the shared fork-ranges in feature.yaml. Each runner
fills its fork range to a directory, then a combine job merges the
results into a single release tarball.

Features without --until (e.g., benchmark with --fork) continue to
build on a single runner with unchanged behavior.

Also adds integration tests for the new scripts and a single-fork-range
parameter set to the fill output directory tests.
Benchmark fixtures have never been included in full releases (verified
against v5.3.0 and v5.4.0 release assets). Mark the feature as
feature_only so it is only built via feature release tags
(tests-benchmark@v*), matching actual release practice.
The setup and combine jobs use uv run but were missing the setup-uv
step, causing "uv: command not found" on ubuntu-latest runners.
Use explicit name fields so GitHub Actions shows concise labels like
"build (pre-cancun)" instead of dumping all matrix values.
BPO forks cannot be specified directly via --from/--until because
explicit BPO forks collect too many tests, and transition fork classes
don't filter correctly. Work around this by using fill-args to specify
--from OsakaToBPO1AtTime15k --until Amsterdam -k "not fork_Amsterdam".

Add fill_args support to fork-range entries and the build-fixtures
action. Also remove BPO3-BPO5 from fork ordering (actual path is
Osaka->BPO1->BPO2->Amsterdam).
The pipefail shell option causes find's SIGPIPE (from head closing the
pipe after 30 lines) to fail the step. Append || true since this is
just debug output.
The `fixtures_*` pattern matched both final tarballs (`fixtures_stable`)
and intermediate split artifacts (`fixtures__cancun`). Use
`fixtures_[!_]*` to skip double-underscore names.
Limit stable and develop to tests/istanbul/eip1344_chainid/ to quickly
validate the multi-runner release pipeline end-to-end. Revert before
merge.
Allow exit code 5 (NO_TESTS_COLLECTED) from fill so fork ranges with
no applicable tests don't fail the build. Skip artifact upload when no
fixtures are generated. Handle missing artifacts gracefully in the
combine step.
The \`gh run download -p "fixtures_[!_]*"\` pattern doesn't work
with the \`gh\` CLI's glob implementation. Download all \`fixtures_*\`
artifacts then \`rm -rf ./artifacts/fixtures__*/\` to remove split
artifacts. Also restrict release upload glob to \`*.tar.gz\` files.
The transition fork fix in #2607 makes --from=BPO1 --until=BPO2 work
correctly. Remove the fill-args/fill-k override plumbing that was
needed to work around the transition fork boundary bug.
@danceratopz danceratopz force-pushed the experiments/multi-runner-releases branch from 61875b9 to 2960d34 Compare April 1, 2026 19:29
Full releases previously built two tarballs (stable + develop) via a
dedicated workflow. Since develop is already covered by the `bal`
feature-only release for devnets, the full release reduces to a single
mainnet tarball -- making it just another feature release.

Changes:
- Remove `develop` feature and rename `stable` to `mainnet`.
- Unify evm-type: both `mainnet` and `bal` now use `evm-type: eels`
  directly, replacing the `stable`/`develop` indirection in evm.yaml.
- Delete `release_fixture_full.yaml`; mainnet releases now use the
  feature workflow via `tests-mainnet@v*` tags.
- Enable `fail-fast: true` on the build matrix so expensive runners
  are cancelled when one fork range fails.
Each release builds exactly one feature, so the combine step never
iterates over multiple features. Replace \`combine_matrix\` with
\`feature_name\` and \`combine_labels\` outputs. Also remove the
\`--all\` mode and \`get_releasable_features()\` (dead code since the
full release workflow was deleted).
Mainnet fixture releases are stable and should not be marked as
pre-releases. Other features (bal, benchmark) keep \`--prerelease\`.
@danceratopz danceratopz changed the title feat(ci): split release fixture builds across multiple fork-range runners feat(ci): split release builds across multiple runners; delete develop; stable becomes mainnet Apr 1, 2026
@danceratopz danceratopz marked this pull request as ready for review April 1, 2026 22:44
@danceratopz danceratopz requested a review from spencer-tb April 1, 2026 23:00
@danceratopz
Copy link
Copy Markdown
Member Author

Ah @spencer-tb, this is probably missing an index.json!!!

When multi-fork releases are split across parallel runners, each
runner produces its own .meta/index.json. Add a classmethod that
merges multiple IndexFile instances by concatenating test cases,
unioning forks and fixture formats, and recomputing the root hash.
Standalone script that loads .meta/index.json from each split fixture
directory and merges them via IndexFile.merge(). Called by the combine
workflow step to produce a correct combined index.
Run merge_index_files.py after downloading split artifacts so the
combined tarball contains a correct .meta/index.json covering all
fork ranges, not just the last one copied.
Delete split_artifacts/ after merging index files to avoid running
out of disk space on ubuntu-latest during tarball creation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci Area: Continuous Integration C-feat Category: an improvement or new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant