Implement flyclient prover functionality

[Metalcape]

This is what I've done so far on flyclient, as discussed on discord (sorry for the delay). I'm publishing this as draft mainly for discussion and feedback on the general implementation.

Update (Nov 1 2025): Added RPC methods for retrieving authdataroot and to support the difficulty-aware Flyclient protocol.

Motivation

Zcash already supports flyclient through the history tree/chainhistoryroot commitment, but zebrad is not currently able to generate a proof that allows a sublinear client to validate the blockchain with it. This PR implements the necessary state changes that zebrad needs to generate a flyclient proof using history tree nodes, and also a new RPC method that can be used by clients to request the nodes.

Solution

• zebra-state:
  • Change the database format to add a HistoryNode column which stores all MMR nodes, indexed by network upgrade and node index; this is to be kept consistent with the chaintip history tree
  • Add an automated database format upgrade for version 27.1.0 which generates history nodes from saved blocks
  • Add functions to read/write nodes and to rebuild a past history tree from saved nodes
  • Add history nodes to the non finalized state
  • Handle writing history nodes to the database when blocks are finalized, for both contextually verified and checkpoint verified blocks
  • Add state service requests/responses for both history nodes and history trees before chain tip
• zebra-chain:
  • Make HistoryTree::push return the list of entries added to the tree by the push
  • Add some functions used to navigate the MMR tree in its "array" form, using node index only
• zebra-rpc:
  • Add a new gethistorynode method to request a single MMR node from the database
  • Add the chainhistoryroot field to all getblock and getblockheader verbose responses
  • Add a new method (getauthdataroot) to download the auth data root of a block, used to validate its chainhistoryroot against the blockcommitments field.
  • Add new methods related to cumulative difficulty (gettotalwork and getfirstblockwithtotalwork) needed to implement difficulty-aware flyclient bootstrapping.

Existing tests have been updated to take these changes into account.

Tests

Testing mostly requires a full sync so I mainly tested manually using a client script that simulates the flyclient protocol bootstrap, sampling random blocks and then downloading MMR ancestry proofs for those blocks. I'm open to suggestions for automated tests.

Specifications & References

• ZIP-221
• The original flyclient paper

Follow-up Work

Needs a testing strategy for the new state changes and RPC methods. There is also a temporary workaround I did in the get_history_node method implementation, which can be replaced as soon as the zcash_history crate is updated with the changes I contributed in zcash/librustzcash#1809 .

PR Checklist

• The PR name is suitable for the release notes.
• The solution is tested.
• The documentation is up to date.

[conradoplg]

Thank you for the great contribution! It might take a bit until we're ready to review this, but we'll get to it eventually.

[Metalcape]

Updated to include the progress we've made in the last couple of months, as of now it should be more or less complete in terms of features (bare minimum that a client needs to validate the blockchain using the protocol).

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟠 ❄️ releae 4.0.0 [Scheduled Freeze]

Waiting for scheduled freeze to end.

A freeze on the repository is scheduled for the following reason: releae 4.0.0

• current-datetime < 2026-01-20T17:25:08[America/Sao_Paulo]

[Copilot]

Pull request overview

This pull request implements flyclient prover functionality for zebrad, enabling the generation of proofs that allow sublinear blockchain validation using the history tree/chainhistoryroot commitment. The implementation spans multiple layers of the Zebra architecture, adding database storage for history tree nodes, new state service operations, and RPC methods to support the difficulty-aware Flyclient protocol.

Changes:

• Added database storage for history tree MMR nodes with automated upgrade for version 27.1.0
• Modified HistoryTree::push to return the list of entries added, enabling tracking of new nodes
• Implemented four new RPC methods: gethistorynode, getauthdataroot, gettotalwork, and getfirstblockwithtotalwork
• Added chainhistoryroot field to all getblock and getblockheader verbose responses

Reviewed changes

Copilot reviewed 50 out of 51 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
zebra-test/src/command.rs	Style update: use format string interpolation
zebra-state/src/service/read/tree.rs	Add functions to read history trees by height, history nodes, and calculate total work
zebra-state/src/service/read/find.rs	Implement binary search to find first block with given total work threshold
zebra-state/src/service/read/block.rs	Add function to retrieve auth data root for blocks
zebra-state/src/service/read.rs	Export new read functions for history nodes and trees
zebra-state/src/service/non_finalized_state/tests/prop.rs	Update test to include history_nodes_by_height field
zebra-state/src/service/non_finalized_state/chain.rs	Add history node tracking to non-finalized chain state
zebra-state/src/service/finalized_state/zebra_db/shielded.rs	Write history nodes when finalizing blocks
zebra-state/src/service/finalized_state/zebra_db/chain.rs	Add history_node column family and methods to read/write/rebuild history trees
zebra-state/src/service/finalized_state/zebra_db/block.rs	Add method to retrieve auth_data_root from blocks
zebra-state/src/service/finalized_state/disk_format/upgrade/add_history_nodes.rs	Implement database upgrade to generate and store history nodes from existing blocks
zebra-state/src/service/finalized_state/disk_format/upgrade.rs	Register new database format upgrade for version 27.1.0
zebra-state/src/service/finalized_state/disk_format/tests/snapshots/*.snap	Update test snapshots to include new history_node column family
zebra-state/src/service/finalized_state/disk_format/chain.rs	Implement serialization for NetworkUpgrade, HistoryNodeIndex, and Entry types
zebra-state/src/service/finalized_state.rs	Update block finalization to capture and store new history nodes
zebra-state/src/service.rs	Add request handlers for new read operations (HistoryTree, HistoryNode, AuthDataRoot, TotalWork)
zebra-state/src/response.rs	Add response types for new read requests
zebra-state/src/request.rs	Add request types and update Treestate to include new_history_nodes
zebra-state/src/constants.rs	Bump database minor version to 27.1
zebra-rpc/tests/vectors/getblock_response_1.json	Add chainhistoryroot field to test vector
zebra-rpc/tests/serialization_tests.rs	Update tests to verify chainhistoryroot field in responses
zebra-rpc/src/methods/tests/vectors.rs	Update test helper and assertions to handle chainhistoryroot
zebra-rpc/src/methods/tests/snapshots/*.snap	Update all snapshot tests to include chainhistoryroot field
zebra-rpc/src/methods.rs	Implement four new RPC methods and add chainhistoryroot to block/header responses
zebra-rpc/Cargo.toml	Add dependencies for zcash_history and bytemuck
zebra-network/proptest-regressions/peer_set/set/tests/prop.txt	Add proptest regression seed
zebra-chain/src/primitives/zcash_history.rs	Add Entry::inner(), From<&Vec> for Entry, HistoryNodeIndex, and root_node_data() method
zebra-chain/src/history_tree.rs	Modify push() to return entries, add MMR navigation functions and HistoryNodeData type

[Copilot]

The binary search implementation has a potential integer overflow issue. On line 730, (start_height.as_usize() + end_height.as_usize()) / 2 could overflow if both heights are large (close to u32::MAX). While this is unlikely in practice for block heights, it's a pattern that should be avoided.

Consider using the overflow-safe pattern: start_height.as_usize() + (end_height.as_usize() - start_height.as_usize()) / 2, or verify that this overflow cannot occur with a comment explaining the invariant.

[Copilot]

The timer finish calls on lines 1847, 1872, 1896, 1915, and 1933 use incorrect labels. Lines 1896, 1915, and 1933 all say "ReadRequest::HistoryNode" but should be "ReadRequest::AuthDataRoot", "ReadRequest::FirstBlockWithTotalWork", and "ReadRequest::TotalWork" respectively. This will make performance debugging and metrics confusing.

[Copilot]

The panic message on line 2309 says "unexpected response to HistoryNode request" but this is actually handling an AuthDataRoot request. This will be confusing during debugging. It should say "unexpected response to AuthDataRoot request".

Suggested change

	panic!("unexpected response to HistoryNode request")
	panic!("unexpected response to AuthDataRoot request")

[Copilot]

The panic message says "unexpected response to HistoryNode request" but this is handling a FirstBlockWithTotalWork request. It should say "unexpected response to FirstBlockWithTotalWork request".

Suggested change

	panic!("unexpected response to HistoryNode request")
	panic!("unexpected response to FirstBlockWithTotalWork request")

[Copilot]

There are redundant Height conversions here - temp_tree_height is already a Height, so converting it .as_usize() as u32 and then back with Height::try_from() is unnecessary. You can use temp_tree_height directly in the function calls.

[Copilot]

The .expect() calls at lines 2220, 2227, 2253, and 2260 could panic if the database contains corrupted data. Since this data comes from the database (which could potentially be corrupted or malicious in some scenarios), these should be converted to proper error handling that returns an error to the RPC caller instead of panicking.

Consider replacing these expects with proper error handling using map_err or similar patterns to return an appropriate RPC error.

[Copilot]

The .unwrap() on line 2197 could panic if the network upgrade doesn't have a branch_id. While the current network upgrades in the UPGRADE_LIST all have branch IDs, this creates a potential future maintenance hazard. Consider handling this case explicitly with proper error handling.

[Copilot]

This method clones the entire history_nodes_by_height BTreeMap and then iterates over it, which is inefficient. The clone is unnecessary here - you can iterate directly over the BTreeMap values using .values() and then .flatten() or collect into the result vector.

Consider changing to: self.history_nodes_by_height.values().flatten().cloned().collect() to avoid the BTreeMap clone.

[Copilot]

Similarly to line 2309, the panic message here says "unexpected response to HistoryNode request" but this is actually handling a TotalWork request. It should say "unexpected response to TotalWork request".

[Copilot]

There's an extra space before FirstBlockWithTotalWork on line 545. It should be ReadResponse::FirstBlockWithTotalWork(_) without the double colon spacing.

Suggested change

	| ReadResponse:: FirstBlockWithTotalWork(_)
	| ReadResponse::FirstBlockWithTotalWork(_)