Simplify reproducible builds: single version pin, add binary tarball + AppImage outputs#9147
Simplify reproducible builds: single version pin, add binary tarball + AppImage outputs#9147
Conversation
…+ AppImage outputs - Replace docker-reproducible.yml with reproducible.yml which produces three artifacts per arch: Docker image, binary tarball, and AppImage - Use a single multi-arch index digest in Dockerfile.reproducible as the sole version tag to maintain; Makefile and CI no longer carry their own per-arch image references - Add packaging/appimage/ template (AppRun, .desktop, lighthouse.svg) Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
… from builder The previous final stage used distroless/cc-debian12 (Bookworm) which carries no libssl and uses OpenSSL 3, making the Bullseye-built binary non-functional. - Switch to distroless/cc-debian11:nonroot (pinned by index digest) — same Bullseye ABI as the builder, already includes libc and libgcc - Copy libssl.so.1.1 and libcrypto.so.1.1 from the builder stage into /usr/lib/ so no package manager is invoked in the final image (stays fully pinned) - Normalise the arch-specific triplet lib path via a `find` into /libs/ so the COPY instructions work identically for both amd64 and arm64 builds Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
…ssl COPY ldd on the built binary shows only libz.so.1 is missing from distroless/cc-debian11; libssl/libcrypto are statically linked by this build and do not need to be copied. libstdc++.so.6 and libgcc_s.so.1 are already present in the distroless/cc variant. Also consolidates the mv + mkdir into a single RUN layer. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
- Rust builder: rust:1.88-bookworm (multi-arch index digest)
- Runtime: distroless/cc-debian12:nonroot (Bookworm, pinned by index digest)
- Build deps bumped to Bookworm versions:
libclang-dev 1:14.0-55.7~deb12u1
cmake 3.25.1-1
libjemalloc-dev 5.3.0-1
- libz.so.1 search path updated /lib → /usr/lib (moved in Bookworm)
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
rust:1.95-bookworm@sha256:225aa827... Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
- Pin appimagetool by SHA256 digest rather than floating 'continuous' tag. The tool has no stable releases; we verify the download hash before use. To update: download new binary, sha256sum it, bump the matrix value. - Add --clobber to gh release upload to handle reruns cleanly. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
All version pins are now visible at the top of their respective files: - Dockerfile.reproducible: Rust image, apt packages, distroless runtime - reproducible.yml: appimagetool SHA256s (APPIMAGETOOL_SHA256_AMD64/ARM64) Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
The double-build check ran both passes on the same runner/daemon/filesystem — any non-determinism it could catch is already eliminated by the build pins (SOURCE_DATE_EPOCH, compiler digest, pinned deps). Replace with a single build that prints the binary SHA256 for external verification. SVG: add Sigma Prime brand color background (#CC00A0), white logo mark. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
bakhtin
left a comment
bakhtin
left a comment
There was a problem hiding this comment.
Thanks for the refactor of original PR. I left some comments.
You pointed out that building twice in the same environment does not make sense which I agree with. But to confirm the build is really reproducible one should build on two different hosts that are sufficiently different. At Flashbots we're building on fresh runners with ubuntu22 and ubuntu24 and compare the hashes of a binary. They should match.
Reproducibility test wasn't part of the original PR either but something worth to follow up with later.
| jobs: | ||
| extract-version: | ||
| name: extract version | ||
| runs-on: ubuntu-22.04 |
There was a problem hiding this comment.
Unless you have a specific reason, why not running on ubuntu-24.04?
| RUN apt-get update && apt-get install -y libclang-dev=1:11.0-51+nmu5 cmake=3.18.4-2+deb11u1 libjemalloc-dev=5.2.1-3 | ||
| # Install pinned versions of the build dependencies | ||
| RUN apt-get update && apt-get install -y \ | ||
| libclang-dev=1:14.0-55.7~deb12u1 \ |
There was a problem hiding this comment.
Pinning to a specific package version does not guarantee the package will be there after some time. Debian tends to remove older versions of packages.
A better approach would be to use a package from the pinned snapshots Debian repo. Check out https://github.com/flashbots/rbuilder/blob/develop/docker/Dockerfile.reproducible for an example. With the pinned snapshot you also don't need to pin package versions.
| # This multi-arch index digest resolves to the correct arch-specific image at build time. | ||
| # To update: run `docker buildx imagetools inspect rust:X.Y-bookworm` and replace the digest below. | ||
| # rust:1.95-bookworm | ||
| ARG RUST_IMAGE="rust:1.95-bookworm@sha256:225aa827d55fae9816a0492284592827e794a5247c6c6a961c3b471b344295ec" |
There was a problem hiding this comment.
bookworm is nearing EOL soon. trixie is around since August last year so maybe switch to a newer version. A newer version will also fix an OpenSSL issue you're having.
|
|
||
| # Move the binary to a standard location | ||
| RUN mv /app/target/${RUST_TARGET}/release/lighthouse /lighthouse | ||
| # Move the binary and runtime libs to fixed paths for arch-independent copying below. |
There was a problem hiding this comment.
trixie distroless docker image comes with libz by default. If you switch you won't need to do this copying.
| - name: Download appimagetool | ||
| run: | | ||
| curl -fsSL \ | ||
| "https://github.com/AppImage/appimagetool/releases/download/continuous/appimagetool-${{ matrix.appimage_arch }}.AppImage" \ | ||
| -o appimagetool | ||
| # Verify against pinned SHA256 (see APPIMAGETOOL_SHA256_* env vars at top of file) | ||
| EXPECTED="APPIMAGETOOL_SHA256_$(echo '${{ matrix.appimage_arch }}' | tr '[:lower:]' '[:upper:]')" | ||
| echo "${!EXPECTED} appimagetool" | sha256sum --check | ||
| chmod +x appimagetool | ||
|
|
||
| - name: Assemble AppDir | ||
| run: | | ||
| mkdir -p AppDir/usr/bin | ||
| cp lighthouse-bin AppDir/usr/bin/lighthouse | ||
| cp packaging/appimage/AppRun AppDir/AppRun | ||
| chmod +x AppDir/AppRun | ||
| cp packaging/appimage/lighthouse.desktop AppDir/lighthouse.desktop | ||
| cp packaging/appimage/lighthouse.svg AppDir/lighthouse.svg | ||
|
|
||
| - name: Build AppImage | ||
| env: | ||
| VERSION: ${{ needs.extract-version.outputs.VERSION }} | ||
| # Deterministic squashfs: fixed modification times, no extra metadata | ||
| SOURCE_DATE_EPOCH: 0 | ||
| run: | | ||
| ./appimagetool \ | ||
| --comp xz \ | ||
| AppDir \ | ||
| lighthouse-${VERSION}-${{ matrix.appimage_arch }}.AppImage | ||
| sha256sum lighthouse-${VERSION}-${{ matrix.appimage_arch }}.AppImage \ | ||
| > lighthouse-${VERSION}-${{ matrix.appimage_arch }}.AppImage.sha256 | ||
|
|
There was a problem hiding this comment.
It came to my attention this is introduced and the cargo debian packaging from the original PR has been removed. Are there plans to add this here in case this PR is going to replace #7617 ?
Also it only uploads the built debian package with a default systemd service file after verifying it is reproducible.
| "https://github.com/AppImage/appimagetool/releases/download/continuous/appimagetool-${{ matrix.appimage_arch }}.AppImage" \ | ||
| -o appimagetool | ||
| # Verify against pinned SHA256 (see APPIMAGETOOL_SHA256_* env vars at top of file) | ||
| EXPECTED="APPIMAGETOOL_SHA256_$(echo '${{ matrix.appimage_arch }}' | tr '[:lower:]' '[:upper:]')" |
There was a problem hiding this comment.
There is a potential bug here because the env variables are named APPIMAGETOOL_SHA256_AMD64 and APPIMAGETOOL_SHA256_ARM64 mean while the appimage_arch in the matrix use x86_64 and aarch64.
I would recommend renaming the env variables to follow the same format of the matrix, or add a separate env variable in the matrix above like :
appimagetool_sha_env: APPIMAGETOOL_SHA256_AMD64
appimagetool_sha_env: APPIMAGETOOL_SHA256_ARM64
and replace EXPECTED here with:
EXPECTED="${{ matrix.appimagetool_sha_env }}"
I agree to some extent. The above mentioned suggestion by @bakhtin is a reasonable follow-up option to tackle this on different setup/environment. However, I still believe it has value to have this check to catch early on any changes that impacts reproducibility in the same environment/systems. Remember reproducibility was an issue before in the same environment. With such checks, you would be notified that the corresponding modification contributed to the reproducibility violation. |
4 participants
Rationale
Summary
docker-reproducible.ymlwithreproducible.yml— one workflow producing three artifacts per arch: Docker image, binary tarball, AppImageRUST_IMAGEARG inDockerfile.reproducible(multi-arch index digest). Makefile and CI no longer carry their own per-arch image referencesdistroless/cc-debian12(broken — wrong OpenSSL ABI) todistroless/cc-debian12:nonrootwith onlylibz.so.1copied from the builder;libssl/libcryptoare statically linked in this buildpackaging/appimage/template (AppRun,.desktop, SVG icon)Dockerfile.reproducible,appimagetoolSHA256s in the workflowenv:blockCaveats
continuousis the only release. We pin by SHA256 of the downloaded binary and verify before use. Updating requires downloading the new binary, runningsha256sum, and bumpingAPPIMAGETOOL_SHA256_AMD64/ARM64in the workflow env blockpublishjob assumes the GitHub release draft already exists (created byrelease.ymlrunning in parallel on the same tag). There is no cross-workflow ordering enforcement — ifrelease.ymlis significantly slower,gh release uploadwill fail. This matches the design of the existing PR feat: Add reproducible Debian package builds and distribution #7617 and is a known limitationTesting
Built and smoke-tested locally (
lighthouse:reproducible-arm64 --version) on arm64 (Bookworm/Rust 1.95).TODO
build twice and comparehas any value as we are building in the same environment on the same system