Implement `export-hashes` and `compare-hashes` subcommands for report processor by elle-j · Pull Request #250 · paritytech/revive-differential-tests

elle-j · 2026-03-10T14:19:26Z

Summary

Extends the report processor with support for extracting bytecode hashes from the reports generated by retester, and running hash comparisons between multiple reports.

The primary purpose is to be able to verify reproducible resolc builds by verifying that identical bytecode was generated by each build across the provided optimization modes.

Notes

Exporting hashes

The source paths from the reports are platform-specific, therefore export-hashes will generate normalized relative paths in the output file provided.

Example run command:

report-processor export-hashes \
    --report-path ./workdir/report.json \
    --output-path ./hashes/hashes-macos.json \
    --remove-prefix /path/to/resolc-compiler-tests/fixtures \
    --platform-label macos

Example of extracted hashes output:

{
  "platform": "macos",
  "hashes": {
    "Y M0 S+": {
      "solidity/simple/block/pyramid.sol": {
        "Test": "0x3126bf12aaf6024767da0dba42f40a7bd602c3899a705904e7e8d0606f505000"
      },
      "solidity/simple/call_chain/address_size1.sol": {
        "Test": "0xc552314bd6c7a6293f0e7f45dc1b339670ec7a4c53f11445c23db163a44f1083",
        "TestA": "0x411ce1cc977316c69a03acb0217b78d4c6ab74b1257283e721187cc27129320e"
      },
    },
    "Y M3 S+": {
      "solidity/simple/block/pyramid.sol": {
        "Test": "0x3126bf12aaf6024767da0dba42f40a7bd602c3899a705904e7e8d0606f505000"
      },
      "solidity/simple/call_chain/address_size1.sol": {
        "Test": "0xc552314bd6c7a6293f0e7f45dc1b339670ec7a4c53f11445c23db163a44f1083",
        "TestA": "0x411ce1cc977316c69a03acb0217b78d4c6ab74b1257283e721187cc27129320e"
      },
    },
    "Y Mz S+": {
      "solidity/simple/block/pyramid.sol": {
        "Test": "0x3126bf12aaf6024767da0dba42f40a7bd602c3899a705904e7e8d0606f505000"
      },
      "solidity/simple/call_chain/address_size1.sol": {
        "Test": "0xc552314bd6c7a6293f0e7f45dc1b339670ec7a4c53f11445c23db163a44f1083",
        "TestA": "0x411ce1cc977316c69a03acb0217b78d4c6ab74b1257283e721187cc27129320e"
      },
    }
  }
}

Comparing hashes

For each mode, the union of all source paths and contracts is traversed.
For each contract, all platforms' hashes are compared.
Any mismatch (either due to a different hash or missing hash) in any platform is reported.
The comparison result is written to the --output-path and a human readable summary is printed to stdout.
The above approach is used, rather than e.g. a diff-based approach, in order to generate more structured results across multi-platform comparisons.

Example run command:

If --mode is specified, only the provided modes will be compared, otherwise the union of all modes found in the files will be compared. (The mode format shares the same parsing logic as used for the retester.)

report-processor compare-hashes \
    --hash-path ./hashes/hashes-macos.json \
    --hash-path ./hashes/hashes-linux.json \
    --hash-path ./hashes/hashes-windows.json \
    --mode "Y M0 S+" \
    --mode "Y M3 S+" \
    --mode "Y Mz S+" \
    --output-path ./hash-comparison.json

Example comparison result and summary:

Expand to see an example comparison result

{
  "platforms": [
    "linux",
    "macos",
    "windows"
  ],
  "hash_counts": {
    "Y M0 S+": {
      "linux": 2978,
      "macos": 2978,
      "windows": 2978
    },
    "Y M3 S+": {
      "linux": 2978,
      "macos": 2978,
      "windows": 2968
    },
    "Y Mz S+": {
      "linux": 2978,
      "macos": 2978,
      "windows": 2978
    }
  },
  "mismatches": {
    "Y M0 S+": [],
    "Y M3 S+": [
      {
        "path": "fixtures/solidity/complex/create/create_in_library/main.sol",
        "contract_name": "Main",
        "hashes": {
          "linux": "0x7d9b1f1ce522121e54ca39a6bf0ffc1bdea31677e44be556faa2156212b1c6cf",
          "macos": "0xd5f5a8014f5326d9f027e35c9bf02635f6aefa4e6f5f499b6cf8c4219a6e06a5",
          "windows": "0x1d9c35bd92bacfa5b936e5a5c106b8f425720fe24f4852cf96fb94fb96cc1d92"
        }
      },
      {
        "path": "fixtures/solidity/complex/defi/UniswapV4/TickMath_loop.sol",
        "contract_name": "TickMathLoop",
        "hashes": {
          "linux": "0xe065e2b56c9247ec351dba8b0a3cf46dc445ec3235e595bf92e3b8dc8bf97366",
          "macos": "0xe065e2b56c9247ec351dba8b0a3cf46dc445ec3235e595bf92e3b8dc8bf97366",
          "windows": "0x137eb27e3d94e51185391fbc40753b9dce63a3b3bbfd18336ed8f6292f421b87"
        }
      },
      {
        "path": "fixtures/solidity/complex/indirect_recursion_fact/first.sol",
        "contract_name": "First",
        "hashes": {
          "linux": "0x9a3f8f95825d47314f2db875bc82e4a7a0922facf03896206d7073e1f06a457f",
          "macos": "0x9a3f8f95825d47314f2db875bc82e4a7a0922facf03896206d7073e1f06a457f",
          "windows": null
        }
      },
      // ...
    ],
    "Y Mz S+": []
  }
}

Expand to see an example stdout summary

===========================================
COMPARISONS
===========================================

-------------------------------------------
Mode: Y M0 S+
-------------------------------------------

Hash counts:
    - linux: 2978
    - macos: 2978
    - windows: 2978

Mismatches: ✅ 0

-------------------------------------------
Mode: Y M3 S+
-------------------------------------------

Hash counts:
    - linux: 2978
    - macos: 2978
    - windows: 2968

Mismatches: ❌ 64 (including 10 missing hashes)

    - path: fixtures/solidity/complex/create/create_in_library/main.sol
      contract: Main
      linux: 0x7d9b1f1ce522121e54ca39a6bf0ffc1bdea31677e44be556faa2156212b1c6cf
      macos: 0xd5f5a8014f5326d9f027e35c9bf02635f6aefa4e6f5f499b6cf8c4219a6e06a5
      windows: 0x1d9c35bd92bacfa5b936e5a5c106b8f425720fe24f4852cf96fb94fb96cc1d92

    - path: fixtures/solidity/complex/defi/UniswapV4/TickMath_loop.sol
      contract: TickMathLoop
      linux: 0xe065e2b56c9247ec351dba8b0a3cf46dc445ec3235e595bf92e3b8dc8bf97366
      macos: 0xe065e2b56c9247ec351dba8b0a3cf46dc445ec3235e595bf92e3b8dc8bf97366
      windows: 0x137eb27e3d94e51185391fbc40753b9dce63a3b3bbfd18336ed8f6292f421b87

    - path: fixtures/solidity/complex/indirect_recursion_fact/first.sol
      contract: First
      linux: 0x9a3f8f95825d47314f2db875bc82e4a7a0922facf03896206d7073e1f06a457f
      macos: 0x9a3f8f95825d47314f2db875bc82e4a7a0922facf03896206d7073e1f06a457f
      windows: MISSING

    ... and 61 more

-------------------------------------------
Mode: Y Mz S+
-------------------------------------------

Hash counts:
    - linux: 2978
    - macos: 2978
    - windows: 2978

Mismatches: ✅ 0

===========================================
SUMMARY
===========================================

* Platforms compared: 
    - linux
    - macos
    - windows

* Modes compared: 
    - Y M0 S+
    - Y M3 S+
    - Y Mz S+

Total mismatches for all modes: 64 (including 10 missing hashes)

❌ FAILURE: Mismatches found among the compared platforms!

===========================================

hashes.

Replace `std::process::exit(1)` with `bail!()` to ensure stdout is flushed before the process exits.

0xOmarA · 2026-03-16T03:34:51Z

Question, why do we need one of the files to be used as the reference platform? Could we not: get all of the files, extract all of the metadata file paths from them into a set and extract all of their modes into a set and then iterate through this super set comparing all of the items in the process?

crates/report-processor/src/main.rs

elle-j · 2026-03-17T10:28:00Z

crates/report-processor/src/compare_hashes.rs

+}
+
+/// Compares all platforms' hashes at the given `mode`.
+fn compare(all_hashes: &[&HashData], mode: &str) -> Vec<Mismatch> {


Question, why do we need one of the files to be used as the reference platform? Could we not: get all of the files, extract all of the metadata file paths from them into a set and extract all of their modes into a set and then iterate through this super set comparing all of the items in the process?

@0xOmarA , to answer your earlier comment ☝️ I'll create a thread here for easier communication.

All modes, file paths, and contracts were already extracted into sets. Selecting a reference platform is not needed per say, it's just one comparison approach. I've updated it now tho to show all platforms' hashes compared to each other rather than each one against the reference. The human parsing of the output becomes easier to understand, so thanks for this input :)

comparability. Source paths in the standard JSON compiler input become part of fully qualified library names embedded in bytecode. Different absolute paths per platform produced different bytecode for the same contract, preventing cross-platform hash comparison. On Windows, canonicalize() also produced \\?\ prefixed paths that broke solc import resolution for contracts with circular or multi-file imports, causing compilation failures. This normalizes source and library paths to relative paths (from the metadata directory) before passing them to solc/resolc, and re-absolutizes the compiler output source paths before canonicalizing.

xermicus · 2026-03-19T22:39:35Z

crates/common/src/fs/normalize_path.rs

+/// Returns an error if `base_path` is `Some` but is not a base directory of `path`.
+///
+/// Purpose of normalization:
+/// 1. **Cross-platform bytecode comparability**: Source paths passed via solc's standard JSON


Why is that? If the symbols are propagated into the PVM blob (why?), then we need to fix this in the compiler instead. But I don't see why and I can't reproduce this.

elle-j added 12 commits March 5, 2026 18:06

Add HashData.

5d8eea9

Implement ExportHashes subcommand.

fc711de

Update normalization to allow unit testing Windows paths on unix.

da43d40

Implement CompareHashes subcommand validation.

76f2f0b

Implement comparison.

cb5305d

Implement comparison analysis.

537c86a

Refactor tests.

f8629f8

Replace panic with bail during normalization.

5af6903

Support output_path CLI arg for the comparison result.

da7dd40

Support extracting hashes from all reports with pre-link compilation

2af4e02

hashes.

Merge branch 'main' into lj/compare-hashes

445e915

Minor updates.

5c9059a

elle-j requested review from 0xOmarA and xermicus March 10, 2026 14:27

elle-j marked this pull request as ready for review March 10, 2026 14:27

Minor updates.

72e7dc8

elle-j requested a review from kvpanch March 12, 2026 14:28

elle-j added 3 commits March 12, 2026 20:11

Support both relative and absolute path for remove-prefix.

2466f40

Fix stdout truncation on hash comparison failure.

68791c1

Replace `std::process::exit(1)` with `bail!()` to ensure stdout is flushed before the process exits.

Display only first 10 mismatches and refer to comparison result file.

41ead12

0xOmarA approved these changes Mar 16, 2026

View reviewed changes

crates/report-processor/src/main.rs Outdated Show resolved Hide resolved

elle-j added 4 commits March 16, 2026 12:07

Merge branch 'main' into lj/compare-hashes

f817984

Show all platforms' hashes vs each other rather than vs a reference.

1d9a6f4

Minor formatting update.

a8451a3

Remove unused method.

ddfe961

elle-j commented Mar 17, 2026

View reviewed changes

elle-j mentioned this pull request Mar 17, 2026

Implement GH actions for compiling, exporting and comparing hashes #251

Open

elle-j added 2 commits March 18, 2026 15:50

Commit missing normalization file.

dbd3e15

elle-j added 2 commits March 19, 2026 09:31

Show joined path in error message.

110c55f

Merge branch 'main' into lj/compare-hashes

2545f3e

elle-j mentioned this pull request Mar 19, 2026

[WIP] Verify reproducible builds and run differential E2E tests paritytech/revive#468

Draft

xermicus reviewed Mar 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `export-hashes` and `compare-hashes` subcommands for report processor#250

Implement `export-hashes` and `compare-hashes` subcommands for report processor#250
elle-j wants to merge 24 commits intomainfrom
lj/compare-hashes

elle-j commented Mar 10, 2026 •

edited

Loading

Uh oh!

0xOmarA commented Mar 16, 2026

Uh oh!

Uh oh!

elle-j Mar 17, 2026

Uh oh!

xermicus Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

elle-j commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notes

Exporting hashes

Comparing hashes

Uh oh!

0xOmarA commented Mar 16, 2026

Uh oh!

Uh oh!

elle-j Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

xermicus Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

elle-j commented Mar 10, 2026 •

edited

Loading