refactor(test): restructure zebrad tests into module-based architecture by gustavovalverde · Pull Request #10454 · ZcashFoundation/zebra

gustavovalverde · 2026-04-08T07:39:02Z

Motivation

The zebrad/tests/acceptance.rs file grew into a monolith containing 68 test functions that span unrelated concerns: CLI smoke tests, RPC endpoints, full chain sync, lightwalletd gRPC, regtest mode, checkpoint generation, and database migrations. The nextest configuration maintained 23 profiles, almost all targeting a single test function by exact name, plus a manually curated exclusion denylist of 14 test names. Adding a new stateful test required coordinated changes in 4 places: the test function, the nextest denylist, a new nextest profile, and the CI workflow.
This refactor replaces the monolith with a module-based architecture where test categorization is structural, not configurational.
Closes #10136

Solution

Structural categorization through module paths

Tests are organized into a single binary (autotests = false) with 3 module tiers that map directly to CI behavior:

Module	Scope	CI trigger	Duration
`unit::`	CLI, config, end-of-support	Every PR	<1 min
`integration::`	Launches zebrad, no cached state	Every PR	5-15 min
`stateful::`	Requires cached blockchain state	Weekly/label/GCP	30 min - days

Nextest filters on module paths using regex: test(/^stateful::/). The default and ci profiles exclude stateful tests automatically. Adding a new test requires placing a function in the right module; no configuration changes are needed.

Nextest simplification

The 23 per-test profiles are replaced by 4 semantic profiles:

default / ci: Runs unit:: + integration:: tests, excludes stateful:: by default-filter
ci-stateful: No default filter; GCP CI selects specific tests via --filter-expr / NEXTEST_FILTER
check-no-git-dependencies: Special release check (unchanged)
Per-test timeout overrides in ci-stateful handle varying runtime requirements (30 min to 20 days) without separate profiles.

Env var gate elimination

Scheduling-gate env vars (TEST_LARGE_CHECKPOINTS, TEST_SYNC_TO_CHECKPOINT, TEST_SYNC_PAST_CHECKPOINT) are removed from test functions. Tests are selected structurally by their module path. #[ignore] attributes remain as a safety net for cargo test without nextest.
TEST_LIGHTWALLETD is preserved in helper functions because it checks runtime binary availability, a fundamentally different concern from scheduling.

Feature flag preservation

Feature flags remain for tests that add compile-time dependencies, applied at the module level in stateful/mod.rs:

lightwalletd-grpc-tests (tonic/protobuf)
zebra-checkpoints (checkpoint binary)
indexer (indexer columns)

CI workflow changes

tests-unit.yml: --profile all-tests → --profile ci
zfnd-ci-integration-tests-gcp.yml: Per-test NEXTEST_PROFILE values → NEXTEST_PROFILE=ci-stateful with NEXTEST_FILTER=test(<test_name>)
docker/entrypoint.sh: Added NEXTEST_FILTER env var support

Specifications & References

Delete Cargo Integration Tests (compilation model rationale)
Nextest Per-Test Overrides
Nextest Filterset Reference

Follow-up Work

Revisit single-binary decision if the test suite grows beyond ~200 integration tests
Replace lightwalletd tests with Zaino-specific tests when ready

AI Disclosure

AI tools were used: Claude for test extraction, module creation, and CI workflow updates. All code was validated with cargo build, clippy, fmt, full nextest run (166 tests), and markdown linting.

PR Checklist

The PR name is suitable for the release notes.
The PR follows the contribution guidelines.
This change was discussed in an issue or with the team beforehand.
The solution is tested.
The documentation and changelogs are up to date.

Split the 4,440-line acceptance.rs monolith into a single test binary with three module tiers: unit::, integration::, and stateful::. This replaces 23 nextest profiles with 4 semantic profiles and eliminates the manually maintained exclusion denylist. Test categorization is now structural: adding a new test requires placing a function in the right module file with zero config changes. The nextest default-filter excludes stateful:: tests by module path, removing the fragile per-test-name denylist. Key changes: - Split acceptance.rs into unit/, integration/, stateful/ modules - Absorb config.rs and end_of_support.rs into unit/ modules - Replace 23 nextest profiles with 4 (default, ci, ci-stateful, check-no-git-dependencies) - Remove env var gates (TEST_LARGE_CHECKPOINTS, TEST_SYNC_TO_CHECKPOINT, TEST_SYNC_PAST_CHECKPOINT) from test functions - Update Docker entrypoint to support NEXTEST_FILTER env var - Update CI workflows to use new profiles and filter expressions - Preserve feature flags for external deps (lightwalletd, checkpoints) - Add ADR documenting the architectural decision Closes #10136

Document in the ADR that failure_messages.rs and cached_state.rs cannot move to zebra-test due to circular dependencies, and that TestType encodes infrastructure requirements rather than scheduling.

- Add cfg-gated imports in unit/config.rs for non_blocking_logger test (only compiles on non-macOS, uses RpcRequestClient and os_assigned_rpc_port_config) - Add cfg-gated imports in integration/rpc.rs for metrics_endpoint and tracing_endpoint (only compile with prometheus/filter-reload) - Add missing ServiceExt import in stateful/indexer.rs

…omment Move the 265-line lwd_integration_test helper to common/lightwalletd.rs (its logical home alongside spawn_lightwalletd_for_rpc) instead of duplicating it in both stateful/sync.rs and integration/network.rs. Also restore the 2 missing doc comment lines on non_blocking_logger that were dropped during the initial extraction.

- Remove TEST_LARGE_CHECKPOINTS from coverage.yml (no test checks it) - Update sync_full_mainnet/testnet doc comments (env var gate removed) - Fix stale lwd-sync-full profile name in lightwalletd error message - Update workflows README with new profile names and module tiers

The TEST_LIGHTWALLETD env var was accidentally removed from all GCP CI lightwalletd jobs when switching from per-test profiles to NEXTEST_FILTER. Unlike the scheduling gates (TEST_SYNC_TO_CHECKPOINT, etc.) which were intentionally removed, TEST_LIGHTWALLETD is a runtime binary availability check still enforced by zebra_skip_lightwalletd_tests(). Without it, all lwd_* tests silently skip with "set the TEST_LIGHTWALLETD variable". Also clean up the unused timeout_argument_name parameter in full_sync_test that was left behind when the env var gate was removed.

Copilot

Pull request overview

Refactors zebrad’s acceptance/integration test suite from a single large integration-test file into a single test binary with a module-based layout (unit::, integration::, stateful::), so CI selection can be driven by module paths and simplified nextest profiles.

Changes:

Introduces a single zebrad-tests integration test binary (autotests = false) with unit/, integration/, and stateful/ module tiers.
Simplifies .config/nextest.toml profiles and updates CI workflows + Docker entrypoint to use NEXTEST_PROFILE plus optional NEXTEST_FILTER.
Adds/updates documentation describing the new test architecture (ADR + repo docs).

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`zebrad/Cargo.toml`	Disables auto-discovered integration tests and registers the single `zebrad-tests` test target.
`zebrad/tests/main.rs`	New test-binary entrypoint and high-level docs for tiering/running tests.
`zebrad/tests/README.md`	Documents directory structure and recommended nextest commands.
`zebrad/tests/unit/mod.rs`	Declares unit submodules.
`zebrad/tests/unit/cli.rs`	CLI-focused tests moved under `unit::`.
`zebrad/tests/unit/config.rs`	Config/state-dir behavior tests moved under `unit::`.
`zebrad/tests/unit/end_of_support.rs`	End-of-support + lockfile git-source check tests moved under `unit::`.
`zebrad/tests/integration/mod.rs`	Declares integration submodules.
`zebrad/tests/integration/sync.rs`	Integration sync tests moved under `integration::`.
`zebrad/tests/integration/rpc.rs`	Integration RPC/endpoint tests moved under `integration::`.
`zebrad/tests/integration/database.rs`	Integration DB/state-format/conflict tests moved under `integration::`.
`zebrad/tests/integration/network.rs`	Network behavior/conflict tests moved under `integration::`.
`zebrad/tests/integration/mempool.rs`	Mempool activation test moved under `integration::`.
`zebrad/tests/integration/regtest.rs`	Regtest-focused integration tests moved under `integration::`.
`zebrad/tests/stateful/mod.rs`	Declares stateful submodules, feature-gated where needed.
`zebrad/tests/stateful/sync.rs`	Cached-state/full-sync tests moved under `stateful::`.
`zebrad/tests/stateful/rpc.rs`	Cached-state RPC tests moved under `stateful::`.
`zebrad/tests/stateful/lightwalletd.rs`	lightwalletd stateful tests moved under `stateful::` (feature/platform gated).
`zebrad/tests/stateful/checkpoints.rs`	Checkpoint generation tests moved under `stateful::` (feature gated).
`zebrad/tests/stateful/indexer.rs`	Indexer stateful tests moved under `stateful::` (feature gated).
`zebrad/tests/config.rs`	Removes the old standalone config integration test file (moved into module layout).
`zebrad/tests/common/mod.rs`	Adjusts common module exports/macros for the new test crate structure.
`zebrad/tests/common/test_type.rs`	Minor update (`#[allow(dead_code)]`) while preserving test infra behavior.
`zebrad/tests/common/lightwalletd.rs`	Expands shared lightwalletd helper logic to support the new structure/gating.
`.config/nextest.toml`	Replaces many per-test profiles with semantic profiles + module-path filters + stateful serial group.
`.github/workflows/tests-unit.yml`	Switches PR CI nextest profile to `ci`.
`.github/workflows/zfnd-ci-integration-tests-gcp.yml`	Moves GCP jobs to `ci-stateful` + `NEXTEST_FILTER` selection.
`.github/workflows/README.md`	Updates workflow docs for new nextest profiles/filtering.
`.github/workflows/coverage.yml`	Removes now-unneeded env var usage tied to old gating.
`docker/entrypoint.sh`	Adds support for passing `NEXTEST_FILTER` through to `cargo nextest run`.
`docs/decisions/testing/0005-test-architecture-refactor.md`	Adds an ADR documenting the architectural decision and tradeoffs.
`AGENTS.md`	Updates contributor guidance for running the new test tiers/profiles.

Comments suppressed due to low confidence (2)

zebrad/tests/unit/end_of_support.rs:31

This test unconditionally sleeps for 30 seconds, which slows CI/local runs and can still be flaky if startup is slower. Prefer waiting for the specific log line (with an explicit timeout) using the existing expect_* helpers, then killing the process once the invariant is observed.
zebrad/tests/unit/end_of_support.rs:57
../Cargo.lock assumes the test process is running with current directory zebrad/. In CI we typically invoke cargo nextest from the workspace root, so this relative path can fail. Prefer resolving from CARGO_MANIFEST_DIR (or a compile-time env!("CARGO_MANIFEST_DIR")) and joining to the workspace Cargo.lock, or searching upwards for the lockfile.

zebrad/tests/main.rs

zebrad/tests/README.md

zebrad/tests/stateful/sync.rs

zebrad/tests/stateful/lightwalletd.rs

Copilot · 2026-04-08T22:09:01Z

zebrad/tests/integration/rpc.rs

+/// See [`crate::common::get_block_template_rpcs::get_peer_info`] for more information.
+#[tokio::test]
+async fn get_peer_info() -> Result<()> {
+    crate::common::get_block_template_rpcs::get_peer_info::run().await


This test can be flaky because it calls getpeerinfo immediately after the RPC endpoint opens and asserts at least one peer, but peer connections are asynchronous and may not be established yet. Consider polling getpeerinfo with a timeout until at least one peer is present (or relaxing the assertion) to avoid intermittent CI failures.

Suggested change

crate::common::get_block_template_rpcs::get_peer_info::run().await

let mut last_error = None;

for attempt in 0..3 {

match crate::common::get_block_template_rpcs::get_peer_info::run().await {

Ok(()) => return Ok(()),

Err(error) => {

last_error = Some(error);

if attempt < 2 {

tokio::time::sleep(LAUNCH_DELAY).await;

}

}

}

}

Err(last_error.expect("the retry loop always runs at least once"))

Pre-existing behavior, identical code to main. Not changed by this refactor, only moved. Out of scope for this PR; unless we agree to fix it here.

- Fix nextest filter examples in main.rs, README.md, and CLAUDE.md to use contains match test(name) instead of exact match test(=name), and regex test(/^unit::/) instead of test(::unit::) - Add #[ignore] to sync_past_mandatory_checkpoint_mainnet, sync_past_mandatory_checkpoint_testnet, and lwd_sync_update which previously relied on env var gates as their safety net. Without #[ignore], cargo test would attempt these stateful tests and fail.

gustavovalverde temporarily deployed to dev April 8, 2026 07:39 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 07:41 — with GitHub Actions Inactive

docs: clarify why test helpers stay in zebrad/tests/common/

bae2641

Document in the ADR that failure_messages.rs and cached_state.rs cannot move to zebra-test due to circular dependencies, and that TestType encodes infrastructure requirements rather than scheduling.

gustavovalverde temporarily deployed to dev April 8, 2026 07:56 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 07:58 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 08:08 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 08:10 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 08:28 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 08:30 — with GitHub Actions Inactive

gustavovalverde self-assigned this Apr 8, 2026

gustavovalverde added the run-stateful-tests Allos to manually trigger a stateful tests run in GCP in PRs label Apr 8, 2026

gustavovalverde temporarily deployed to dev April 8, 2026 09:31 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 09:42 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 11:30 — with GitHub Actions Inactive

gustavovalverde added 2 commits April 8, 2026 12:47

gustavovalverde temporarily deployed to dev April 8, 2026 12:32 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 12:34 — with GitHub Actions Inactive

gustavovalverde added run-stateful-tests Allos to manually trigger a stateful tests run in GCP in PRs and removed run-stateful-tests Allos to manually trigger a stateful tests run in GCP in PRs labels Apr 8, 2026

gustavovalverde temporarily deployed to dev April 8, 2026 13:19 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 13:30 — with GitHub Actions Inactive

gustavovalverde temporarily deployed to dev April 8, 2026 13:31 — with GitHub Actions Inactive

Copilot AI reviewed Apr 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(test): restructure zebrad tests into module-based architecture#10454

refactor(test): restructure zebrad tests into module-based architecture#10454
gustavovalverde wants to merge 9 commits intomainfrom
refactor/test-architecture

gustavovalverde commented Apr 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

gustavovalverde Apr 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-    crate::common::get_block_template_rpcs::get_peer_info::run().await
+    let mut last_error = None;
+    for attempt in 0..3 {
+        match crate::common::get_block_template_rpcs::get_peer_info::run().await {
+            Ok(()) => return Ok(()),
+            Err(error) => {
+                last_error = Some(error);
+                if attempt < 2 {
+                    tokio::time::sleep(LAUNCH_DELAY).await;
+                }
+            }
+        }
+    }
+    Err(last_error.expect("the retry loop always runs at least once"))

Conversation

gustavovalverde commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

Structural categorization through module paths

Nextest simplification

Env var gate elimination

Feature flag preservation

CI workflow changes

Specifications & References

Follow-up Work

AI Disclosure

PR Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gustavovalverde Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gustavovalverde commented Apr 8, 2026 •

edited

Loading

gustavovalverde Apr 9, 2026 •

edited

Loading