You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: website/content/en/preview/concepts/disruption.md
+50-7Lines changed: 50 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -360,20 +360,51 @@ In this scenario, Karpenter cannot voluntary disrupt the node because:
360
360
361
361
As seen in this example, the more PDBs there are affecting a Node, the more difficult it will be for Karpenter to find an opportunity to perform voluntary disruption actions.
362
362
363
-
Secondly, you can block Karpenter from voluntarily disrupting and draining pods by adding the `karpenter.sh/do-not-disrupt: "true"` annotation to the pod.
364
-
You can treat this annotation as a single-pod, permanently blocking PDB.
363
+
Secondly, you can block Karpenter from voluntarily disrupting and draining pods by adding the `karpenter.sh/do-not-disrupt` annotation to the pod.
| **Duration** | `karpenter.sh/do-not-disrupt: "30m"` | Provides time-based protection for the specified duration after the pod starts running |
370
+
371
+
#### Duration-Based Protection
372
+
373
+
When using the duration format, the annotation will be "active" and pods will be protected from disruption for the specified time period after they start running (based on `pod.status.startTime`).
374
+
Once the duration expires, the annotation becomes inactive and the pod becomes eligible for disruption like any other pod.
375
+
This is useful for workloads that need protection during startup or critical phases but can be safely disrupted later.
376
+
377
+
The duration value must be a valid Go `time.Duration` string. Supported formats include:
If an invalid duration is specified, the annotation will be ignored and an event will be emitted on the pod indicating that the duration format is invalid.
389
+
{{% /alert %}}
390
+
391
+
#### Behavior and Consequences
392
+
393
+
You can treat this annotation as a single-pod blocking PDB that is active either permanently (boolean format) or temporarily while the duration hasn't elapsed (duration format).
365
394
This has the following consequences:
366
-
- Nodes with `karpenter.sh/do-not-disrupt` pods will be excluded from [Consolidation]({{<ref "#consolidation" >}}), and conditionally excluded from [Drift]({{<ref "#drift" >}}).
395
+
- Nodes with active `karpenter.sh/do-not-disrupt` pods will be excluded from [Consolidation]({{<ref "#consolidation">}}), and conditionally excluded from [Drift]({{<ref "#drift">}}).
367
396
- If the Node's owning NodeClaim has a [`terminationGracePeriod`]({{<ref "#terminationgraceperiod" >}}) configured, it will still be eligible for disruption via drift.
368
-
- Like pods with a blocking PDB, pods with the`karpenter.sh/do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{<ref "#termination-controller">}}).
397
+
- Like pods with a blocking PDB, pods with an active `karpenter.sh/do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{<ref "#termination-controller">}}).
369
398
Karpenter will not be able to complete termination of the node until one of the following conditions is met:
370
-
- All pods with the `karpenter.sh/do-not-disrupt` annotation are removed.
399
+
- All pods with the `karpenter.sh/do-not-disrupt` annotation are removed, or their annotation becomes inactive (duration has elapsed).
371
400
- All pods with the `karpenter.sh/do-not-disrupt` annotation have entered a [terminal phase](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) (`Succeeded` or `Failed`).
372
401
- The owning NodeClaim's [`terminationGracePeriod`]({{<ref "#terminationgraceperiod">}}) has elapsed.
373
402
374
-
This is useful for pods that you want to run from start to finish without disruption.
375
-
Examples of pods that you might want to opt-out of disruption include an interactive game that you don't want to interrupt or a long batch job (such as you might have with machine learning) that would need to start over if it were interrupted.
403
+
#### Examples
376
404
405
+
This is useful for pods that you want to run from start to finish without disruption, or that need protection during critical startup phases.
406
+
407
+
**Permanent protection** - useful for interactive games or long-running batch jobs:
377
408
```yaml
378
409
apiVersion: apps/v1
379
410
kind: Deployment
@@ -384,6 +415,18 @@ spec:
384
415
karpenter.sh/do-not-disrupt: "true"
385
416
```
386
417
418
+
**Duration-based protection** - useful for workloads with critical startup phases:
419
+
```yaml
420
+
apiVersion: apps/v1
421
+
kind: Deployment
422
+
spec:
423
+
template:
424
+
metadata:
425
+
annotations:
426
+
# Protect for 30 minutes after pod starts running
427
+
karpenter.sh/do-not-disrupt: "30m"
428
+
```
429
+
387
430
{{% alert title="Note" color="primary" %}}
388
431
The `karpenter.sh/do-not-disrupt` annotation does **not** exclude nodes from the forceful disruption methods: [Expiration]({{<ref "#expiration">}}), [Interruption]({{<ref "#interruption">}}), [Node Repair](<ref "#node-repair">), and manual deletion (e.g. `kubectl delete node ...`).
389
432
While both interruption and node repair have implicit upper-bounds on termination time, expiration and manual termination do not.
Copy file name to clipboardExpand all lines: website/content/en/preview/troubleshooting.md
+8-2Lines changed: 8 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -482,9 +482,15 @@ Review what [disruptions are](https://kubernetes.io/docs/concepts/workloads/pods
482
482
483
483
#### `karpenter.sh/do-not-disrupt` Annotation
484
484
485
-
If a pod exists with the annotation `karpenter.sh/do-not-disrupt: true` on a node, and a request is made to delete the node, Karpenter will not drain any pods from that node or otherwise try to delete the node. Nodes that have pods with a `do-not-disrupt` annotation are not considered for consolidation, though their unused capacity is considered for the purposes of running pods from other nodes which can be consolidated.
485
+
If a pod exists with an active `karpenter.sh/do-not-disrupt` annotation on a node, and a request is made to delete the node, Karpenter will not drain any pods from that node or otherwise try to delete the node. The annotation is considered "active" when:
486
+
- Set to `"true"` (permanent protection)
487
+
- Set to a valid duration (e.g., `"30m"`) and the pod has been running for less than that duration
486
488
487
-
If you want to terminate a node with a `do-not-disrupt` pod, you can simply remove the annotation and the deprovisioning process will continue.
489
+
Nodes that have pods with an active `do-not-disrupt` annotation are not considered for consolidation, though their unused capacity is considered for the purposes of running pods from other nodes which can be consolidated.
490
+
491
+
If you want to terminate a node with a `do-not-disrupt` pod, you can either remove the annotation from the pod or wait for duration-based protection to expire naturally, and the deprovisioning process will continue.
492
+
493
+
For more details on how this annotation works, see [Pod-Level Controls]({{<ref "./disruption#pod-level-controls" >}}) in the Disruption documentation.
0 commit comments