Skip to content

Commit 0d7f54d

Browse files
mliptak0mliptak0
committed
HYPERFLEET-837 - feat: Add resource recreation design for adapters
1 parent 5a1b8ed commit 0d7f54d

1 file changed

Lines changed: 222 additions & 0 deletions

File tree

‎hyperfleet/components/adapter/framework/adapter-recreation-flow-design.md‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
---
2+
Status: Active
3+
Owner: HyperFleet Team
4+
Last Updated: 2026-06-03
5+
---
6+
7+
# Adapter Resource Recreation Flow Design
8+
9+
**Jira**: [HYPERFLEET-837](https://issues.redhat.com/browse/HYPERFLEET-837)
10+
11+
## What & Why
12+
13+
**What**: Design the recreation workflow for resources that cannot be updated in-place (e.g., Kubernetes Jobs with immutable fields). When certain conditions change, the adapter deletes the old resource and creates a new one.
14+
15+
**Why**: Some resources have immutable fields that cannot be patched. Without a clear recreation mechanism:
16+
- Config authors have no way to express "recreate this resource when X changes"
17+
- Recreation must be safe across Sentinel's ~10s retry loop (may receive same event multiple times)
18+
- Deletion may span multiple reconciliation loops due to finalizers or async behavior
19+
- Creation must never be attempted while deletion is still pending
20+
21+
**Related Documentation:**
22+
- [Adapter Framework Design](./adapter-frame-design.md) — Core executor architecture
23+
- [Adapter Status Contract](./adapter-status-contract.md) — Status reporting patterns
24+
25+
### Scope
26+
27+
- CEL expression evaluation for recreation triggers (`lifecycle.recreate.when`)
28+
- Stateless, transport-agnostic delete+apply flow
29+
- Handling async deletion (finalizers, eventual consistency)
30+
- Both Kubernetes and Maestro transports
31+
- Safe retry across Sentinel's polling loop
32+
33+
### Out of Scope
34+
35+
- Intermediate status conditions during recreation (future enhancement)
36+
- Stuck deletion timeout behavior and alerts
37+
- lifecycle.delete logic (covered separately)
38+
39+
---
40+
41+
## Design
42+
43+
### Flow & Lifecycle
44+
45+
When a resource is configured with `lifecycle.recreate.when`, the executor:
46+
47+
1. **Pre-discovers** the current resource state (if any) so CEL expressions can compare incoming parameters with the current resource to detect changes that require recreation.
48+
49+
2. **Evaluates** `lifecycle.recreate.when` (a CEL expression). If true, enters the recreation flow; otherwise, uses the normal apply path (update in-place or create).
50+
51+
3. **In the recreation flow**: deletes the old resource, waits for it to be fully gone, then creates a new one. This flow is safe to retry — multiple attempts converge to the same end state.
52+
53+
### Configuration Structure
54+
55+
Configure resource recreation by adding a `lifecycle.recreate` block with a `when` expression that detects when the resource should be deleted and recreated:
56+
57+
```yaml
58+
resources:
59+
- name: "job"
60+
transport:
61+
client: "kubernetes"
62+
manifest:
63+
apiVersion: batch/v1
64+
kind: Job
65+
metadata:
66+
name: "{{ .clusterId }}-job"
67+
annotations:
68+
hyperfleet.io/generation: "{{ .generation }}"
69+
# ... job spec with immutable fields
70+
71+
# Required: discovery allows the CEL expression to compare current vs. desired state
72+
discovery:
73+
by_name: "{{ .clusterId }}-job"
74+
75+
# Recreation configuration: when the expression is true, delete old resource and create new
76+
lifecycle:
77+
recreate:
78+
when:
79+
expression: |
80+
# Recreate when the generation has changed (indicating spec changed)
81+
resources.?job.hasValue()
82+
&& string(generation) != resources.?job.metadata.annotations["hyperfleet.io/generation"]
83+
```
84+
85+
**Requirements when `lifecycle.recreate` is configured:**
86+
- `discovery` block must exist — pre-discovery populates resource state so CEL expressions can compare current vs. desired state
87+
- `lifecycle.recreate.when.expression` must be set to a non-empty CEL string
88+
- The expression is validated at adapter startup
89+
- Recreation works identically for K8s and Maestro transports (no transport-specific branching)
90+
91+
### Recreation Flow
92+
93+
When `lifecycle.recreate.when` evaluates to true:
94+
95+
```
96+
1. Check if resource exists (pre-discovery result in context)
97+
if not found: skip delete, go straight to apply
98+
99+
2. Delete existing resource via TransportClient.DeleteResource()
100+
(fire-and-forget, returns immediately)
101+
102+
3. Post-delete discovery: check if resource is truly gone
103+
if 404/NotFound: resource is gone → proceed to apply
104+
if still present (has deletionTimestamp or other deletion state):
105+
→ store discovered object in context
106+
→ return early with Operation=Recreate, Reason="deletion pending"
107+
→ do NOT call ApplyResource
108+
if discovery error (non-404): assume still present, return early
109+
110+
4. ApplyResource() with rendered bytes (no RecreateOnChange flag needed)
111+
112+
5. Post-apply discovery + nested discoveries (same as normal apply flow)
113+
```
114+
115+
### Happy Path Sequence
116+
117+
```mermaid
118+
sequenceDiagram
119+
participant Event as Event<br/>generation=2
120+
participant Executor as ResourceExecutor
121+
participant Transport as K8s/Maestro
122+
participant Context as ExecutionContext
123+
124+
Event->>Executor: Process event
125+
Executor->>Transport: GetResource(Job)
126+
Transport-->>Context: resources["job"] = {generation=1}
127+
128+
Note over Executor: Evaluate lifecycle.recreate.when → true
129+
130+
Executor->>Transport: DeleteResource(Job)
131+
Transport-->>Executor: 200 OK
132+
133+
Executor->>Transport: GetResource(Job)
134+
Transport-->>Executor: 404 Not Found
135+
136+
Executor->>Transport: ApplyResource(generation=2)
137+
Transport-->>Executor: 201 Created
138+
139+
Executor->>Transport: GetResource(Job)
140+
Transport-->>Context: resources["job"] = {generation=2}
141+
142+
Executor-->>Event: Success: Recreate
143+
```
144+
145+
### Pending Deletion Sequence
146+
147+
This is the critical case: deletion takes multiple reconciliation loops due to finalizers or async behavior. Sentinel sends the same event repeatedly (~10s intervals). Recreation must be safe to retry.
148+
149+
```mermaid
150+
sequenceDiagram
151+
participant A1 as Attempt 1<br/>~0s
152+
participant A2 as Attempt 2<br/>~10s
153+
participant A3 as Attempt 3<br/>~20s
154+
participant Executor as ResourceExecutor
155+
participant Transport as K8s/Maestro
156+
157+
A1->>Executor: Event (generation=2)
158+
Executor->>Transport: GetResource → {generation=1, Active}
159+
Executor->>Transport: DeleteResource
160+
Transport-->>Executor: 200 OK
161+
162+
Executor->>Transport: GetResource → {deletionTimestamp, finalizers pending}
163+
Note over Executor: Still present → Return early<br/>Do NOT apply yet
164+
165+
A2->>Executor: Event (generation=2, same)
166+
Executor->>Transport: GetResource → {deletionTimestamp, finalizers pending}
167+
Executor->>Transport: DeleteResource (re-issue, idempotent)
168+
Transport-->>Executor: 200 OK
169+
170+
Executor->>Transport: GetResource → {still present}
171+
Note over Executor: Still present → Return early
172+
173+
A3->>Executor: Finalizers complete
174+
Executor->>Transport: GetResource → 404 Not Found
175+
Executor->>Transport: ApplyResource(generation=2)
176+
Transport-->>Executor: 201 Created
177+
Note over Executor: Success: Recreate
178+
```
179+
180+
**Key property**: Delete is idempotent. Re-issuing DELETE when the resource already has `deletionTimestamp` is safe. Sentinel's retry loop naturally handles pending deletions without special logic.
181+
182+
### CEL Evaluation Context
183+
184+
`lifecycle.recreate.when` expressions evaluate with the same context as other CEL expressions:
185+
186+
- **Extracted parameters**: `generation`, `clusterId`, `namespace`, etc. (from event/preconditions)
187+
- **Discovered resource state**: `resources.?job.metadata.annotations`, `resources.?job.spec`, etc.
188+
- **Adapter metadata**: `adapter.executionStatus`, `adapter.name`, etc.
189+
- **Optional chaining**: `resources.?job.hasValue()` → prevents crashes on missing resources
190+
191+
Example expressions:
192+
```
193+
# Recreate when generation annotation mismatches
194+
resources.?job.hasValue() && string(generation) != resources.?job.metadata.annotations["hyperfleet.io/generation"]
195+
196+
# Recreate when API spec field changes
197+
resources.?cluster.hasValue() && resources.?cluster.spec.region != currentRegion
198+
```
199+
200+
### Edge Cases & Handling
201+
202+
**Resource doesn't exist at pre-discovery**: If discovery returns 404, skip the delete step and go directly to apply (create new resource).
203+
204+
**Post-delete discovery fails (non-API error)**: Assume resource is still present due to transient error. Return early with "deletion pending" reason. Retry on next Sentinel event.
205+
206+
**CEL accesses non-existent resource**: Use optional chaining (`resources.?job.hasValue() && ...`) to avoid crashes. CEL short-circuits evaluation, so type errors are prevented.
207+
208+
**Deletion in progress when recreation is triggered**: The framework detects if deletion is still pending (finalizers, async behavior) and defers creation. Retries naturally occur on the next Sentinel event (~10s), when the deletion may be complete.
209+
210+
---
211+
212+
## Responsibilities & Trade-offs
213+
214+
**What the framework provides:** Declarative recreation via CEL expressions. Works identically for K8s and Maestro. Safe to retry (delete is idempotent).
215+
216+
**What config authors own:**
217+
1. Test recreation flow in staging environment before production
218+
2. Validate that stuck deletions are prevented through proper RBAC, finalizers, and cleanup logic in their domain
219+
220+
**Trade-off:** Deletion is fire-and-forget; actual deletion may span multiple reconciliation loops. The framework cannot recover from stuck deletions — prevention through proper design in the staging environment is the only viable approach.
221+
222+
**Why `lifecycle.recreate.when` replaces `RecreateOnChange`:** The previous transport-layer flag only worked for K8s (ignored by Maestro) and was not safe to retry across Sentinel's polling loop. The new CEL-based approach is transport-agnostic and stateless.

0 commit comments

Comments
 (0)