UI-driven resharding: metadata-store migrations with live operation log (#914)

* Backfill pinned cnpg shard into Duckling spec (spec.metadataStore.cnpgShard)

Which cnpg shard a tenant's metadata lives on is currently only a
composition-pinned status stamp (status.metadataStore.assignedShard) —
not explicit anywhere in spec or the config store. charts PR #12918 adds
an optional, schema-validated spec.metadataStore.cnpgShard field whose
precedence is: spec override > pinned status > cnpg.activeShard.

The provisioner now backfills that field on ready cnpg-shard ducklings
with the CR's own pinned assignedShard — the same derived-output →
durable-input move as the existing bucket-name backfill, and a pure
no-op for the composition since the value equals the pin. Once every
duckling carries it, the tenant's shard is explicit spec: the
precondition for shard-migration cutovers, which patch this field to a
DIFFERENT shard (an explicitly operator-driven step, never done by the
backfill; an existing spec value is never overwritten).

On a cluster whose Duckling XRD predates the field, the API server
silently prunes the patch; the post-patch read-back detects that and
latches the backfill off (one WARN, no per-tick patch churn) until the
control plane restarts.

e2e: new duckling_shard_backfill assertion (cnpg lane) waits for the
spec field to match the pinned shard; on an old XRD it probes field
support with its own patch and SKIPs loudly instead of failing, so it
goes live the moment charts #12918 deploys to mw-dev.

* Add UI-driven resharding: metadata-store migrations with live op log

Operator-driven migration of an org's DuckLake catalog between metadata
stores, in three directions: cnpg shard -> cnpg shard, external(RDS) ->
cnpg, and cnpg -> external as the emergency escape hatch. Only the
Postgres catalog moves; S3 parquet never does. Full design:
docs/design/resharding.md.

Config store: reshard operation rows + a verbose operator-facing log
table (goose 000018), claim/heartbeat/epoch-fenced writes, cancel flag,
and the LOAD-BEARING connection barrier: the lease-grant transaction
refuses orgs whose warehouse is resharding, checked under the same
per-org advisory lock the ready->resharding CAS takes. The connect-time
57P03 gates (PG wire + Flight) are UX only - a lease could otherwise be
granted up to a queue-timeout after the gate ran.

Runner (provisioner, every CP replica; single winner per op via claim
CAS): block -> drain (leases+queue zero, then zero live workers - each
runs a catalog-writing DuckLakeCheckpointer; lingering hot-idle workers
retire via the standard CAS path, never raw pod deletes; parked Flight
sessions are destroyed locally per CP or they hold leases for the token
TTL) -> pause compaction (explicit spec patch, key-presence recorded and
restored exactly) -> flip -> copy -> verify -> cleanup -> finalize with
an end-of-op report (maintenance-mode duration, per-op tables/rows/bytes).
cnpg->ext inverts copy and flip: the type flip deletes the cnpg source
role/DB, so it IS the cleanup and only runs after verify; its target
password is ephemeral (request -> in-process stash -> runner memory,
never persisted). Rollback always patches the source shard VALUE back
(never removes the key); ext->cnpg rollback nulls cnpgShard (XRD CEL).
Cancel is honored between steps and inside every wait loop. External
stores are never modified.

Catalog copier: one REPEATABLE READ read-only snapshot, faithful DDL
from pg_catalog introspection, raw binary COPY passthrough, constraints
then non-constraint indexes (PK-backed excluded), per-table verify plus
an outside-tx source-stability recheck that catches concurrent writers
(e.g. an in-flight compaction job), fenced by a target-DB advisory lock.

Admin API (RoleGate/audited): POST /orgs/:id/reshard, GET
/orgs/:id/reshards, GET /reshards/:id{,/log?after_id=}, POST
/reshards/:id/cancel. Console: Reshard button on org detail -> target
form (shard select or external + once-only password) -> operation page
with header (org, from->to, status/step, started/finished, maintenance
duration, copy counters) + live incremental log with auto-scroll and a
Cancel button. Drift checker treats resharding as expected.

Tests: configstore postgres suite (claim/fencing/cancel/log pagination +
the grant-gate), runner unit tests (all three directions, rollbacks,
cancel, ephemeral-password takeover loss), admin handler tests
(validation, secrets never persisted), migration asserts to v18, and
four e2e assertions incl. the REAL ext->cnpg positive path off the
harness RDS and a real bogus-shard rollback
(DUCKGRES_RESHARD_FLIP_TIMEOUT=90s in the CI manifests).

* e2e: treat post-reshard connect gate as transient; one-shot negative check

The reshard assertions reconnect immediately after the resharding->ready
flip, but the connect-time gate reads the polled config snapshot — up to
one poll interval of lag during which connects still get the 57P03
'reshard in progress, please retry shortly' rejection. Add that message
to _pg_exec's transient-retry list (it says retry, so the harness
should), and switch the mid-drain negative connect check to a one-shot
psql so it observes the rejection instead of retrying through it.

* catalog copy: cast pg_constraint.contype to text

contype is the internal "char" type (OID 18), which pgx cannot scan in
binary format into a string — the ext->cnpg e2e run streamed the whole
catalog and then failed exactly here (and rolled back cleanly).

* Reshard: per-op cutover_timeout_seconds instead of a CI-wide short flip timeout

The e2e's CI-wide DUCKGRES_RESHARD_FLIP_TIMEOUT=90s (added for the
bogus-shard rollback assertion) also bounded the LEGITIMATE ext->cnpg
cutover, which needs minutes: the endpoint converged in ~20s but
Ready=True (provider-sql role/DB creation + cnpg SASL credential
propagation) takes longer, so the real cutover rolled back at 90s.

Make the cutover wait per-operation: new cutover_timeout_seconds on the
op row/API/UI form (0 = 15m default; DUCKGRES_RESHARD_FLIP_TIMEOUT now
only overrides that default). The e2e rollback assertion passes 90s on
its own op, the ext->cnpg positive path passes 600s, and the CI env
knob is gone from the manifests.

Author: benben

CLAUDE.md

@@ -528,6 +528,47 @@ touching this path:
   `compute_usage_pull_api` assertion (compute + storage) in
   `tests/e2e-mw-dev/harness.sh`.
 
+## Resharding (metadata-store migrations) — LOAD-BEARING CONTRACT
+
+Operator-driven moves of an org's DuckLake catalog between metadata stores
+(cnpg↔cnpg, ext→cnpg, cnpg→ext escape hatch), admin-console-driven with a
+verbose op log. Full design: `docs/design/resharding.md`. Pieces:
+`configstore/reshard.go` (+ migration `000018`),
+`provisioner/reshard_runner.go` + `catalog_copy.go`, `admin/reshard.go`, UI
+`ReshardForm.tsx`/`ReshardOperation.tsx`. Invariants:
+
+- **The sound connection barrier is the lease-GRANT check**, not the
+  connect-time 57P03 gates: the grant transaction refuses `resharding` orgs
+  under the same per-org advisory lock `SetWarehouseResharding` takes for the
+  `ready→resharding` CAS. Never rely on the snapshot-polled connect gate
+  alone — a lease can be granted up to a queue-timeout after it ran.
+- **Drain, never kill**: live queries always finish. Drain = leases==0 AND
+  queue==0 (one tx) AND zero live org workers (each runs a catalog-writing
+  `DuckLakeCheckpointer`). Lingering hot-idle workers are retired via the
+  standard CAS retire path only — never raw pod deletes. Parked reconnectable
+  Flight sessions are destroyed locally per CP (they'd hold leases ~1h).
+- **Flip semantics differ by direction**: a `cnpgShard` change re-points
+  role/DB in place (source ORPHANED — explicit cleanup after verify); a TYPE
+  flip to external makes Crossplane DELETE the cnpg role/DB → cnpg→ext runs
+  **copy-before-flip** (the flip IS the cleanup, only after verify).
+  **External stores are never modified/deleted.**
+- **Rollback patches the source shard VALUE back — never removes the key**
+  (precedence would fall through to the freshly-stamped bogus status pin);
+  ext→cnpg rollback must null `cnpgShard` (XRD CEL forbids it on external).
+- **The ext target password is ephemeral**: request → in-process stash →
+  runner memory; never in the op row, log, or audit. Takeover mid-copy fails
+  with a clear re-run message instead of proceeding without it.
+- **Runner fencing**: claim bumps `runner_epoch`; every runner write is
+  CAS-fenced on (runner, epoch); stale-heartbeat (>5m) ops are takeover-able;
+  the copy holds a target-DB advisory lock.
+- Touching any of this → update `tests/configstore/reshard_postgres_test.go`,
+  `provisioner/reshard_runner_test.go`, `admin/reshard_test.go`, the
+  migration asserts in `tests/configstore/migrations_postgres_test.go`, AND
+  the `reshard_*` assertions in `tests/e2e-mw-dev/harness.sh` (validation,
+  cancel-during-drain, bogus-shard-rollback, ext→cnpg positive path).
+  cnpg→ext positive path is unit-only (harness lacks the RDS password);
+  cnpg→cnpg positive path needs a second mw-dev shard (follow-up).
+
 ## TODO Reference
 
 `TODO.md` is a lightweight backlog for ideas that do not yet have a better

controlplane/admin/audit.go

@@ -201,10 +201,19 @@ func auditActionFor(method, path string) string {
 		case hasSeg(segs, "impersonate"):
 			// Impersonation records its own richer row; this is a fallback.
 			return "impersonate." + verb
+		case last == "reshard":
+			// POST /orgs/:id/reshard.
+			return "reshard.create"
 		case hasSeg(segs, "users"):
 			return "user." + verb
 		}
 		return "org." + verb
+	case "reshards":
+		// POST /reshards/:opid/cancel (reads aren't audited).
+		if last == "cancel" {
+			return "reshard.cancel"
+		}
+		return "reshard." + verb
 	}
 	return "config." + verb
 }

controlplane/admin/ducklings_drift.go

@@ -108,6 +108,11 @@ func (h *driftHandler) findDrift(c *gin.Context) {
 			CRReady:        ready,
 		}
 		switch {
+		case wh.State == configstore.ManagedWarehouseStateResharding:
+			// A reshard in flight legitimately holds the warehouse out of
+			// ready (and the CR may flap Ready during the cutover) — that is
+			// the operation working, not drift.
+			continue
 		case !present:
 			entry.Issue = "missing"
 			entry.Message = "warehouse exists but Duckling CR not found"