Skip to content

devops: the cached-state image pool is shared, and the selector cannot distinguish producersΒ #11362

Description

@alchemydc

Describe the issue or request

The cached-state image pool in the dev GCP project is shared, and the selector that picks an image from it cannot tell which repository produced a given image.

scripts/gcp-get-cached-disks.sh selects the newest READY image matching prefix + branch-or-sha + state version + network + disk type, with a three-tier fallback (gcp-get-cached-disks.sh:51-55):

CACHED_DISK_NAME=$(find_cached_disk_image ".+-${GITHUB_REF}" "branch")
CACHED_DISK_NAME=${CACHED_DISK_NAME:-$(find_cached_disk_image "main-[0-9a-f]+" "main branch")}
CACHED_DISK_NAME=${CACHED_DISK_NAME:-$(find_cached_disk_image ".+-[0-9a-f]+" "any branch")}

The schedule: trigger fires in every repository that contains a copy of the integration workflow, and those runs publish into this same project. Every such run on a default branch produces a main-<sha> image, so tier 2 matches them all indistinguishably and the winner is simply whichever finished most recently.

Current Behavior

On 2026-08-28, three competing v28-testnet-tip images were published into the pool within 30 minutes. Selection was decided by a 2.5-minute ordering margin. Had the ordering differed, this repository's CI would have seeded a testnet run from state built elsewhere.

The consumers are broader than CI. ADR 0006 (docs/decisions/devops/0006-gcp-deployment-naming.md:28) states that cache images produced by integration tests are the cold-start mechanism for fresh node deploys, and #11108 removed the release skip from the cached-disk lookup, so the release path consults the pool too.

Scope, to be precise rather than alarming: the release/prod path resolves a different GCP project, so production is isolated. The exposure is the dev project β€” integration tests and the dev node fleet.

Expected Behavior

An image lookup should only return images this repository is entitled to consume.

Possible Solution

Not a one-line filter change, because the sharing we want is asymmetric: it is fine and desirable for other copies of the workflow to consume this repository's images; the reverse is not.

Options worth weighing:

  1. Scope the selector by producer. fix(ci): give GCP test resource names run identityΒ #11364 adds a repository label to every created image, making the pool auditable and a label-based filter possible. Needs care: a repository with no images of its own would find nothing and trigger a full regeneration.
  2. Separate GCP projects, so pools cannot overlap at all. Cleanest isolation, but each project then needs its own cache bootstrap β€” a full mainnet sync, a testnet sync and a lightwalletd sync β€” and carries its own 400 GB image set.
  3. Asymmetric read/write: publish to a per-repository project while reading this repository's pool cross-project via IAM. Removes contamination without any regeneration. gcp-get-cached-disks.sh calls gcloud compute images list with no --project, so it would need a flag.

Additional Information/Context

Until #11364 lands, images in the pool carry no producer attribution at all, so historical contamination cannot be audited β€” four images currently in the dev project cannot be attributed either way.

Related: #11226, #11202.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions