Stakater Reloader: Accumulated STAKATER_* Environment Variables Must Be Cleaned Up

[PRYeswanthReddy]

Deployments using Stakater Reloader with the default env-vars reload strategy are accumulating orphaned STAKATER_* environment variables in the pod template spec. Each ConfigMap/Secret rotation adds new env vars but never removes the old ones, leading to unbounded growth. Switching to --reload-strategy=annotations prevents new accumulation but does not clean up existing vars — kubectl apply merges and never removes externally-injected fields.

Affected Deployment

Name:               appsimulation
Namespace:          del-weu-cdg
Replicas:           2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:       RollingUpdate
RollingUpdateStrategy: 25% max unavailable, 25% max surge

Pod Template:
  Labels:
    app=appsimulation
    azure.workload.identity/use=true
    tier=frontend
  Annotations:
    fluentbit.io/exclude_stderr: true
    fluentbit.io/exclude_stdout: true
  Service Account:  del-sample-appsimulation-sa

  Containers:
    appsimulation:
      Image:      <acr-registry>/appsimulation:<tag>
      Port:       8080/TCP (http)
      Limits:
        cpu:     200m
        memory:  400Mi
      Requests:
        cpu:     100m
        memory:  100Mi
      Liveness:   http-get http://:8080/health delay=30s timeout=10s period=10s #success=1 #failure=3
      Readiness:  http-get http://:8080/home delay=10s timeout=10s period=5s #success=1 #failure=3

      Environment Variables from:
        appsimulation-config-<build-version>  ConfigMap  Optional: false
        appsimulation-secret-<build-version>  Secret     Optional: false

      Environment (PROBLEM — 16 orphaned STAKATER_* vars):
        STAKATER_..._SECRET:     <sha-hash>    # ← from build rotation 1
        STAKATER_..._CONFIGMAP:  <sha-hash>    # ← from build rotation 1
        STAKATER_..._SECRET:     <sha-hash>    # ← from build rotation 2
        STAKATER_..._CONFIGMAP:  <sha-hash>    # ← from build rotation 2
        ...                                    # ← repeated for every past rotation
        STAKATER_..._UNDEFINEDAPPBUILDNR_SECRET:    <sha-hash>
        STAKATER_..._UNDEFINEDAPPBUILDNR_CONFIGMAP: <sha-hash>

      Environment (legitimate app vars):
        ASPNETCORE_ENVIRONMENT:  cdg
        APP_BUILDNUMBER:         <version>
        APP_NAME:                appsimulation

Conditions:
  Progressing:  True  (NewReplicaSetAvailable)
  Available:    True  (MinimumReplicasAvailable)

Problem

1. 16 orphaned STAKATER_* env vars are present in the pod template, two per historical ConfigMap/Secret rotation (8 rotations tracked).
2. Each var name encodes the full build version string, making them unique and never overwritten.
3. kubectl apply (used by the deployment pipeline) performs a 3-way merge — it will not remove fields that were added by an external controller (Reloader).
4. Switching Reloader to --reload-strategy=annotations stops new accumulation but does not remove existing vars.

we are deploying the kubernetes manifests using a custom script which internally uses kubectl apply -f command. kindly let us know how we can cleanup the stakater env data.