Direction: separate pack wiring from spin lifecycle

[dcmcand]

What this is

#641 and #549 both are attempts to answer "what is a spin", but they have different answers. The purpose of this issue is to take the learning from those two and propose a new definition of a spin.

Nothing here closes either PR. They stay open until we have talked this through, and if the discussion goes against this proposal then this issue is the one that gets closed.

What we know now

From @marcelovilla's Crossplane PoC (#641):

• Crossplane Spin PoC #641 demonstrated three layers of composition. However, only the run-time layer was not currently doable with our current tooling. Deriving hostnames from domain and injecting sibling namespaces is render-time templating, and so are the two NetworkPolicies. Reading z2jh's hub Secret for the dask-gateway API token is the only step that required observing cluster state.
• dask-gateway-pack's values.yaml says the pack is designed to install in the same namespace as data-science-pack, which makes apiTokenFromSecretName/apiTokenFromSecretKey and hub-URL inference work unaided. Installing them in separate namespaces breaks both as they exist today. Separate namespaces is a reasonable choice, and the PoC follows the pack's own cross-namespace instructions. The cost is that the token leaves the Secret it lives in and lands in the Application's valuesObject, which is not a Secret: plaintext in etcd, in kubectl get application -o yaml, and in the ArgoCD UI.
• The composition gates the whole dask-gateway Application on having read that token (01-compose.yaml.gotmpl:200). Crossplane garbage-collects composed resources the pipeline stops returning, and the Application carries prune: true plus the ArgoCD finalizer, so an empty or failed Secret read uninstalls the dask-gateway release. That is the general shape of the problem when desired state depends on observed state, and it is the reason for the render-time versus runtime split proposed under question 1.
• A central wiring layer ends up knowing pack internals: z2jh's Secret key path, DSP's OIDC Secret naming, the operator's client-id convention, dask-gateway's traefik service name, JupyterHub pod labels, ports 8786 and 8081. Nothing checks any of it at build or test time.

From @oldsj's spins design doc (#549):

• Packs need a relationship to cluster infrastructure. The llm-chat example turns on a GPU node pool, and an in-cluster composition engine with no cloud providers cannot produce one. docs(design): software pack spins #549 concludes from this that a spin should emit cluster.<provider>.node_groups. I disagree with that half and propose declare-and-validate instead, for the reasons under question 2; the underlying point that packs cannot ignore infrastructure stands.
• Provider-keyed logic belongs in data rather than NIC code. docs(design): software pack spins #549 makes this case with a versioned (provider, requirement, tier) table. Under declare-and-validate we do not need that table, but the principle carries over to ADR-0003's convention table, which is where cluster answers get matched to pack requirements.
• It names the cross-namespace gap from the other direction: "cross-namespace credential handoff is unsolved."

The two questions

Both efforts inherently answer 2 questions without being explicit about them:

1. How do packs wire to each other?
2. What is a spin, and who owns its lifecycle?

We should answer the first on its own. Its answer holds whatever we decide about the second, and settling it makes the second much smaller.

Question 1: pack wiring

I propose that we create a specification for a software pack contract. At a high level, this contract will define the requirements for the pack (hardware, required values, etc), optional configuration, and outputs that other packs may consume.

We would still need to work out the details, but my current thought is that producer pack publishes a contract in its own namespace and a consumer in another namespace reads it.

Practically this leaves us with 2 workstreams before we can take this further.

A cross-namespace contract projection

The authorization model is a big part we need to work out. Both #641 and #549 call this out. This would be the first stream of work. Cross-namespace projection is a privilege bridge, and producer opt-in, consumer request with approval, and admin grant all have different failure modes.

nebari-operator is the natural home. It already provisions Keycloak clients and writes client Secrets into pack namespaces.

A pack contract with inputs and outputs

ADR-0003 reserved this slot. nebari-integration.yaml is inputs-only today (required_values), and the ADR was explicit that this was temporary: "No version field, no templating, no kind. Extend later when we know more."

What we should add:

• Outputs, the way a Terraform module declares them.
• Typed required values, where the type names the check. type: storageClassName resolves against StorageClasses, type: nodeSelector resolves against schedulable nodes, untyped stays presence-only as it is today. That gives us real validation without inventing a capability vocabulary that will rot.
• A version field. NIC, the operator, pack CI and spin CI will all pin this contract.
• The identity and network surface alongside values. A sibling cannot write a NetworkPolicy or an RBAC grant against a pack that publishes only strings; it needs namespace, service name and port, pod selector labels, service account.
• Binding time on every output:

Declared outputs resolve at render time. Runtime-materialized outputs travel through the projection.

Helm needs render-time values to decide what to template, so those stay frozen. Runtime values do not, and they cover most of the interesting wiring. That rule is what lets whatever renders a spin stay dumb and never observe the cluster.

There is a third category: conventions the operator owns rather than the pack. The OIDC client id is generated by nebari-operator, and the PoC guessed at the convention and got it wrong. Those should be operator outputs read through the same contract, so we can change a convention without breaking every pack that re-derived it.

Question 2: what a spin is

After we work out how packs wire together, then we can decide the best way to answer this question. Here is my current thinking, though it may change as we work through question 1.

A spin is an ordinary versioned Helm chart that renders ArgoCD Applications, distributed like any other pack, with each child pack's targetRevision exposed as a value. This is the app-of-apps pattern apps/root.yaml already uses. Operator overrides go in values/<spin>/overlays/*.yaml through the ADR-0014 seam. Render-time wiring comes from chart templates, runtime wiring through the projection.

Rendering Applications instead of embedding subcharts is deliberate. Helm resolves dependency versions from Chart.yaml at package time, so a true umbrella chart would force a spin re-release to bump one child. Rendering Applications puts those versions in values where an overlay can reach them, which keeps the property #549 wants: "once installed, each is an ordinary pack that operators upgrade or remove individually."

NIC does not emit node groups, and neither does a spin. Infrastructure stays declared in config.yaml and owned by the operator, always. A pack declares what it needs as a required value (rwx_storage_class_name, gpu_node_group_name), ADR-0003's convention table satisfies it on clusters that have an answer, and an unsatisfied requirement fails with the name of what is missing. Where NIC can be helpful it prints the cluster.<provider>.node_groups block to paste, without writing the file.

Three reasons to prefer this over provisioning. config.yaml has no overlay seam, so a second writer turns every spin upgrade into a manual merge. Nothing NIC created means nothing to orphan when a pack is removed. And it is the only version that works on existing and local, where NIC does not own the infrastructure at all; #549 concedes this, calling GPU support on existing "an operator-supplied precondition".

The cost is that installing a GPU spin becomes two steps rather than one. In exchange, validation checks the AMI, the taint and the GPU Operator independently, which catches a half-configured cluster that generation would have gotten right and a hand-rolled setup will not.

Enforcement has two halves because NIC is not in the day-2 loop. A chart can enforce presence itself with Helm's required, so a pack added by commit fails at render and ArgoCD surfaces it. Checking that the referent actually exists needs something cluster-aware, which points at nebari-operator again.

The line sits there because of our architectural principle: "GitOps for software. ArgoCD manages everything that runs on the cluster after the cluster itself is provisioned." Making NIC the ongoing owner of pack manifests puts it on the wrong side of that, and it runs against ADR-0014, whose payoff is that "an overlay takes effect on ArgoCD's next sync with no nic deploy run at all."

Day 2

Operation	Mechanism	NIC involved?
Upgrade the spin	Bump one targetRevision, commit	No
Bump one pack independently	Pin it in values/<spin>/overlays/, commit	No
Override any value	Overlay, commit	No
Rollback	git revert, ArgoCD syncs	No
Producer changes a runtime output	Projection follows, no commit needed	No
Remove a pack	Drop from overlay values, ArgoCD prunes	No
Change the domain	Spin value or overlay, re-renders consistently	No
Spin adds a new infra requirement	Operator edits config.yaml, nic deploy	Yes, for the cluster change only

A spin upgrade is a commit. If the new version declares a requirement the cluster does not satisfy, it fails with what is missing rather than silently provisioning, and the operator decides whether to add it.

What we give up

• No automatic spin upgrades. Bumping a tag will not change installed software, which matches docs(design): software pack spins #549's position that "every new release must be revalidated, then applied deliberately."
• Per-pack validation still does not exist. docs(design): software pack spins #549 calls it a prerequisite, and this proposal does not make it easier.
• Installing a spin that needs infrastructure the cluster lacks is two steps: add the node group to config.yaml and redeploy, then install the spin.

Open questions

1. Is "no automatic spin upgrade" right, or does someone need to bump a tag and get new software?
2. What is the authorization model for cross-namespace projection?
3. Does the pack contract live in a new software-pack-spec repo, and do the software-pack-template docs and examples move with it?
4. Do we validate that a required value's referent exists, or only that it is set, in the first cut? Presence alone catches typos; it does not stop a GPU node group name that points at nothing, and the pods then sit Pending.
5. What survives from the Crossplane direction that this drops? The composition ergonomics and the readiness sequencing through function-sequencer were both real, and this shape does not replace them.

Work this implies

• An ADR for the pack contract, plus an Update section on ADR-0003, whose current text defers exactly this.
• An ADR for the cross-namespace projection.
• An ADR for the spin shape, superseding this issue once we agree.
• A software-pack-spec repo holding the versioned schema and a conformance suite, with the template repo's docs and examples moving in as conformance fixtures.
• Extending ADR-0014's overlay seam to pack and spin Applications. ADR-0014 parks this as follow-up and notes that TestHelmApps_SeamInvariants walks the embedded template FS, so it cannot cover Applications that are not embedded.
• A status note on ADR-0012 closing the spins carve-out, if we agree on this direction.
• Findings on [SPIKE] Research on Packs-of-Packs for Spins exploration #551 and [SPIKE] Research on Spins tooling ergonomics #566 so those spikes can close. [SPIKE] Research on Packs-of-Packs for Spins exploration #551's actual hypothesis, whether packs can compose using only native primitives with no bespoke value-passing layer, has not been tested: only the Crossplane arm was built, so the comparison [SPIKE] Research on Packs-of-Packs for Spins exploration #551 asked for does not exist yet.

Definition of done

• Team discussion held with @marcelovilla and @oldsj present
• Direction agreed, amended, or rejected, with the outcome recorded here
• If agreed: the three ADRs above written and merged
• If agreed: ADR-0003 and ADR-0012 amended so they do not contradict the new records
• docs(design): software pack spins #549 and Crossplane Spin PoC #641 dispositioned by their authors
• Follow-up implementation issues filed with their own DoDs, including tests and docs
• Findings posted on [SPIKE] Research on Packs-of-Packs for Spins exploration #551 and [SPIKE] Research on Spins tooling ergonomics #566

Related

• Crossplane Spin PoC #641 Crossplane Spin PoC
• docs(design): software pack spins #549 software pack spins design doc
• [SPIKE] Research on Packs-of-Packs for Spins exploration #551 packs-of-packs spike
• [SPIKE] Research on Spins tooling ergonomics #566 spins tooling ergonomics spike
• Surface base/overlay collisions before a base.yaml update is silently overridden #669 base/overlay collision reporting, which this shape leans on
• ADR-0003 software pack codegen, ADR-0004 out-of-tree providers, ADR-0012 Crossplane, ADR-0014 Helm valueFiles overlay seam