UI-driven resharding: metadata-store migrations with live operation log#914
Merged
Conversation
…gShard) Which cnpg shard a tenant's metadata lives on is currently only a composition-pinned status stamp (status.metadataStore.assignedShard) — not explicit anywhere in spec or the config store. charts PR #12918 adds an optional, schema-validated spec.metadataStore.cnpgShard field whose precedence is: spec override > pinned status > cnpg.activeShard. The provisioner now backfills that field on ready cnpg-shard ducklings with the CR's own pinned assignedShard — the same derived-output → durable-input move as the existing bucket-name backfill, and a pure no-op for the composition since the value equals the pin. Once every duckling carries it, the tenant's shard is explicit spec: the precondition for shard-migration cutovers, which patch this field to a DIFFERENT shard (an explicitly operator-driven step, never done by the backfill; an existing spec value is never overwritten). On a cluster whose Duckling XRD predates the field, the API server silently prunes the patch; the post-patch read-back detects that and latches the backfill off (one WARN, no per-tick patch churn) until the control plane restarts. e2e: new duckling_shard_backfill assertion (cnpg lane) waits for the spec field to match the pinned shard; on an old XRD it probes field support with its own patch and SKIPs loudly instead of failing, so it goes live the moment charts #12918 deploys to mw-dev.
Operator-driven migration of an org's DuckLake catalog between metadata
stores, in three directions: cnpg shard -> cnpg shard, external(RDS) ->
cnpg, and cnpg -> external as the emergency escape hatch. Only the
Postgres catalog moves; S3 parquet never does. Full design:
docs/design/resharding.md.
Config store: reshard operation rows + a verbose operator-facing log
table (goose 000018), claim/heartbeat/epoch-fenced writes, cancel flag,
and the LOAD-BEARING connection barrier: the lease-grant transaction
refuses orgs whose warehouse is resharding, checked under the same
per-org advisory lock the ready->resharding CAS takes. The connect-time
57P03 gates (PG wire + Flight) are UX only - a lease could otherwise be
granted up to a queue-timeout after the gate ran.
Runner (provisioner, every CP replica; single winner per op via claim
CAS): block -> drain (leases+queue zero, then zero live workers - each
runs a catalog-writing DuckLakeCheckpointer; lingering hot-idle workers
retire via the standard CAS path, never raw pod deletes; parked Flight
sessions are destroyed locally per CP or they hold leases for the token
TTL) -> pause compaction (explicit spec patch, key-presence recorded and
restored exactly) -> flip -> copy -> verify -> cleanup -> finalize with
an end-of-op report (maintenance-mode duration, per-op tables/rows/bytes).
cnpg->ext inverts copy and flip: the type flip deletes the cnpg source
role/DB, so it IS the cleanup and only runs after verify; its target
password is ephemeral (request -> in-process stash -> runner memory,
never persisted). Rollback always patches the source shard VALUE back
(never removes the key); ext->cnpg rollback nulls cnpgShard (XRD CEL).
Cancel is honored between steps and inside every wait loop. External
stores are never modified.
Catalog copier: one REPEATABLE READ read-only snapshot, faithful DDL
from pg_catalog introspection, raw binary COPY passthrough, constraints
then non-constraint indexes (PK-backed excluded), per-table verify plus
an outside-tx source-stability recheck that catches concurrent writers
(e.g. an in-flight compaction job), fenced by a target-DB advisory lock.
Admin API (RoleGate/audited): POST /orgs/:id/reshard, GET
/orgs/:id/reshards, GET /reshards/:id{,/log?after_id=}, POST
/reshards/:id/cancel. Console: Reshard button on org detail -> target
form (shard select or external + once-only password) -> operation page
with header (org, from->to, status/step, started/finished, maintenance
duration, copy counters) + live incremental log with auto-scroll and a
Cancel button. Drift checker treats resharding as expected.
Tests: configstore postgres suite (claim/fencing/cancel/log pagination +
the grant-gate), runner unit tests (all three directions, rollbacks,
cancel, ephemeral-password takeover loss), admin handler tests
(validation, secrets never persisted), migration asserts to v18, and
four e2e assertions incl. the REAL ext->cnpg positive path off the
harness RDS and a real bogus-shard rollback
(DUCKGRES_RESHARD_FLIP_TIMEOUT=90s in the CI manifests).
Test Impact PlanDeterministic summary of how this PR changes tests, CI runners, and coverage-risk signals. Summary
Signals
Coverage risk: needs review Warnings
|
- Migration numbering: main's storage-usage migration deliberately took 000019, leaving 000018 for this branch — union the migration-test asserts (17, 18, 19; latest 19) and add 18 + the reshard tables to the v8-baseline downgrade block (main's DELETE list couldn't include 18). - Reshard grant-gate test: seed the org row before the warehouse row — fk_duckgres_orgs_warehouse rejects an orphan warehouse (caught by CI; this machine has no docker for the postgres-backed suite). - PASS line: union of main's compute+storage wording and the reshard assertions.
…check The reshard assertions reconnect immediately after the resharding->ready flip, but the connect-time gate reads the polled config snapshot — up to one poll interval of lag during which connects still get the 57P03 'reshard in progress, please retry shortly' rejection. Add that message to _pg_exec's transient-retry list (it says retry, so the harness should), and switch the mid-drain negative connect check to a one-shot psql so it observes the rejection instead of retrying through it.
contype is the internal "char" type (OID 18), which pgx cannot scan in binary format into a string — the ext->cnpg e2e run streamed the whole catalog and then failed exactly here (and rolled back cleanly).
…ip timeout The e2e's CI-wide DUCKGRES_RESHARD_FLIP_TIMEOUT=90s (added for the bogus-shard rollback assertion) also bounded the LEGITIMATE ext->cnpg cutover, which needs minutes: the endpoint converged in ~20s but Ready=True (provider-sql role/DB creation + cnpg SASL credential propagation) takes longer, so the real cutover rolled back at 90s. Make the cutover wait per-operation: new cutover_timeout_seconds on the op row/API/UI form (0 = 15m default; DUCKGRES_RESHARD_FLIP_TIMEOUT now only overrides that default). The e2e rollback assertion passes 90s on its own op, the ext->cnpg positive path passes 600s, and the CI env knob is gone from the manifests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves an org's DuckLake metadata catalog between metadata stores from the admin console, with a verbose live operation log. Three directions: cnpg shard → cnpg shard, external (RDS) → cnpg, and cnpg → external as the emergency escape hatch. Only the Postgres catalog moves — S3 parquet never does. Design doc:
docs/design/resharding.md; new LOAD-BEARING section inCLAUDE.md.Operator flow
Org detail → Reshard metadata store… → form (target shard select / external endpoint + once-only password, drain timeout, typed confirm) → operation page: header (org, from→to, status + current step e.g. "Waiting for all connections to drain…", started/finished, maintenance-mode duration, tables/rows/bytes copied), a Cancel button, and a live incremental log pane. An end-of-op report (also on failure/cancel) lands in the log: maintenance duration, copy counters, cleanup outcome.
The parts that had to be right
ready→reshardingCAS takes — the snapshot-polled connect-time 57P03 gates (PG wire + Flight) alone are unsound (a lease can be granted up to a queue-timeout after they ran).DuckLakeCheckpointerthat writes the catalog independent of sessions. Lingering hot-idle workers retire via the standard CAS retire path (never raw pod deletes). Parked reconnectable Flight sessions (which hold leases up to the 1h token TTL) are destroyed locally per CP once truly idle.cnpgShardchange re-points role/DB in place (source orphaned; dropped explicitly only after verify); a TYPE flip to external makes Crossplane DELETE the cnpg role/DB — so cnpg→ext runs copy-before-flip (the flip IS the cleanup). External stores are never modified.cnpgShard(XRD CEL). Cancel honored between steps and inside every wait loop.Tests
tests/configstore/reshard_postgres_test.go— claim CAS, takeover fencing, cancel, logafter_idpagination, and the grant-gate test (lease refused during resharding, other orgs unaffected, drain state sees leases+queue). Migration asserts bumped to v18.controlplane/provisioner/reshard_runner_test.go— happy paths for all three directions (incl. copy-before-flip ordering for cnpg→ext), drain-timeout / flip-timeout / verify-mismatch rollbacks, cancel-during-drain, ephemeral-password loss.controlplane/admin/reshard_test.go— validation matrix, 409s, cancel, password never persisted or echoed.harness.sh):reshard_validation,reshard_cancel_during_drain(held session keeps drain waiting; new connections blocked; drain-not-kill),reshard_bogus_shard_rollback(real flip → Synced=False → rollback,spec.cnpgShardpatched back, data intact;DUCKGRES_RESHARD_FLIP_TIMEOUT=90sin CI manifests), and the REAL ext→cnpg positive path: catalog moved off the harness RDS onto shard-001 with data verified intact.npm run build+ existing vitest suite green.Explicitly not covered (and why)
tests/integrationcatalog-copy test was superseded by the real e2e copy (RDS → cnpg shard exercises the copier against two real Postgres flavors, strictly stronger).🤖 Generated with Claude Code