[BUG] Mass pod restarts when controller restarts or leader changes

[davekonopka]

Describe the bug
I'm using Reloader to restart ArgoCD Rollouts pods for configmap and secrets changes. Whenever the controller pod restarts it seems to think all secrets & configmaps have changed triggering reloads of all app pods in all namespaces.

To Reproduce

I have the following annotations on apps:

reloader.stakater.com/auto: "true"

I restart the leader controller pod the second pod takes over and then immediately recognizes changes on secrets and configmaps for all apps. It leads to rolling restarts for lots of apps.

Expected behavior
If secrets and configmaps have not changed I wouldn't expect controller restarts or leader changes to trigger mass app restarts.

Environment

• Operator Version: v1.2.0
• Helm chart Version: v1.2.0
• Kubernetes Version: v1.32.9

Additional context
Add any other context about the problem here.

Chart values:

reloader:
  isArgoRollouts: true
  reloadOnCreate: true
  reloadOnDelete: true
  syncAfterRestart: true
  reloadStrategy: annotations
  enableHA: true
  deployment:
    replicas: 2
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

[faizanahmad055]

@davekonopka Can you please disable this (syncAfterRestart: true) property and try again? We are looking into this, but currently, this is flaky and can cause the behaviour that you described.

[davekonopka]

Thanks for following up. I disabled syncAfterRestart and I'm no longer seeing bulk pod restarts when the controller leader pod restarts and a new leader takes over.

Is the downside of disabling this edge cases where secret changes during a controller leader change or pod reset might not trigger app pod restarts?

Where does reloader store secrets/configmap hashes for an app with Argo rollouts? I'm not seeing anything on the rollout or its pod specs. I was wondering if that could be related to why controller restarts trigger mass restarts.

[faizanahmad055]

In case of argo rollout and annotation strategy, it stores in the pod annotation of last restart but doesn't track the history. The downside of disabling syncAfterRestart is, during the time when reloader pod is not running and when it comes up, if there is any change happened in secret/configmap, reloader will not trigger any restart when it starts up. But that time is very short in leader election, max is 15 seconds I believe.

[davekonopka]

Thanks for your responsiveness. I appreciate your work on this project.

[Ghostbird]

Ah, we were having some instability because of this too. Good to know that I can for now disable syncAfterRestart. I see that the pods get annotated with: reloader.stakater.com/last-reloaded-from : {"type":"SECRET","name":"my-secret","namespace":"my-namespace","hash":"22213d9f654223581dee78c2283cc1ded28241de","containerRefs":["api-container"],"observedAt":1771575921}

It seems reloader does not check the hash in the annotation against the current secret hash. It should skip the reload if it's not necessary.