Fix replica escalation during rolling updates with local recommender

[piyushjindal-dd]

Summary

Fixes a feedback loop where the WPA escalates replica count on every reconciliation cycle during rolling updates when using a local recommender. The WPA goes from steady-state (e.g., 18 replicas) to maxReplicas (e.g., 40+) during routine deployments, even when the metric is stable and within watermark bounds.

Fixes #333

The Bug

During a rolling update with maxSurge >= 1, Kubernetes maintains Status.Replicas = Spec.Replicas + maxSurge. The WPA sends Status.Replicas as CurrentReplicas to the recommender, but adjustReplicaCount compares the recommender's response against Spec.Replicas. This creates a +1 escalation on every cycle:

1. Spec=18, Status=19 (surge pod)
2. WPA sends CurrentReplicas=19 to recommender
3. Metric is between watermarks → recommender returns "hold at 19"
4. adjustReplicaCount compares 19 (recommended) vs 18 (Spec) → upscale!
5. WPA sets Spec=19. K8s creates surge pod → Status=20
6. Next cycle: recommender says "hold at 20", 20 > 19 → upscale to 20
7. Repeat until maxReplicas

Observed Production Impact (authenticator, us1.prod.dog, 2026-03-16)

• Replicas escalated 18 → 40 in ~5 minutes during a routine deploy
• Old-version pods scaled 18 → 50 via K8s proportional scaling — all throwaway
• CPU was stable at ~15% (well within 10%–30% watermarks)
• ~320 CPU cores wasted for 1+ hour (downscale delay)

Dashboards:

• Running instances by version (old pods scaling up)
• WPA effective replicas (escalation)
• Bits AI investigation

The Fix

Send Spec.Replicas instead of Status.Replicas as CurrentReplicas to the recommender (replica_calculator.go:283).

-CurrentReplicas: target.Status.Replicas,
+CurrentReplicas: target.Spec.Replicas,

Status.Replicas includes transient surge pods that Kubernetes creates during rolling updates. Spec.Replicas is the intended replica count — and it is already what adjustReplicaCount uses as its baseline (line 319). Sending the same value to the recommender breaks the feedback loop.

When there is no rolling update in progress, Spec.Replicas == Status.Replicas, so this change has no effect in steady state. The recommender still receives the actual running pod count via CurrentReadyReplicas if it needs it.

Why not a post-hoc clamp?

An earlier approach added a clamp after adjustReplicaCount to suppress the false-upscale result. That approach fixed the symptom but left the recommender receiving stale/inflated input during every rollout. Sending Spec.Replicas is the correct fix at the source: the recommender gets accurate input and returns an accurate recommendation.

[piyushjindal-dd]

@codex review

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[piyushjindal-dd]

Another simpler option - but this would be a breaking change as the contract with the clients would change. Not sure if we that is okay. If so, we can do that instead.

[sblumenthal]

As mentioned on slack, I can't find any reference to this contract in the docs or git history... I think this was more likely an oversight, possibly because the original change introduced over 2 years ago didn't test with maxSurge configured

I think this approach would be cleaner, so I think we should proceed with this one

[piyushjindal-dd]

Thanks, reverted the pr to simpler approach.

[piyushjindal-dd]

@clamoriniere @sblumenthal @vboulineau since you are listed as maintainers for WPA can I get feedback from your on this. Currently due to this our deployments are bit messy.