[BUG] Operator reverts manual StatefulSet replica scaling back to 1

[olamide226]

Bug Description

When manually scaling a StatefulSet replicas (e.g., increasing from 1 to 3), the ToolHive operator automatically reverts the replica count back to 1. This behavior prevents horizontal scaling of MCP servers.

Steps to Reproduce

1. Deploy an MCP server via the ToolHive operator (creates a StatefulSet with 1 replica)
2. Manually scale the StatefulSet:

   kubectl scale statefulset <mcpserver-name> --replicas=3

3. Observe that the operator reverts the replicas back to 1

Expected Behavior

The operator should NOT automatically revert manual scaling changes. The manually set replica count should persist.

Actual Behavior

The operator overrides the manual scaling and resets replicas to 1.

Root Cause

The MCPServer CRD lacks a replicas field to persist the desired replica state. Without this field, the operator has no way to know whether the replica count was intentionally changed, causing it to revert to its default state.

Proposed Solution

1. Add a replicas field to the MCPServer CRD spec to allow users to declare the desired replica count
2. Update the operator to respect this field and not override manual scaling changes
3. The field should be optional with a default value of 1 for backward compatibility

Example:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: my-mcp-server
spec:
  replicas: 3  # New field
  image: my-image:latest
  # ... other fields

Additional Context

• For scaling purposes, typically only one proxy/runner is needed since it routes to a headless service and can load balance between pods in the StatefulSet
• This is especially relevant for stateless MCP servers or Streamable HTTP MCP servers where load balancing works well
• For stateful MCP servers, scaling considerations may be more complex

Environment

• ToolHive Operator version: v0.6.12
• Kubernetes version: v1.33.3+k3s1

---

Related discussion: The community has confirmed this is a bug and the operator should NOT revert manual scaling changes.

[ChrisJBurns]

Hey @olamide226 thanks for the issue. You're completely correct in that we don't support scaling at the moment via increasing replica count. This is just down to some of the trickiness behind the scenes with the different transport types. For example, in the Streamable HTTP case scaling is a bit easier and we just have to do some logic in the proxy to ensure it knows where to route requests and responses. But stdio is tricky because its very nature its a constant 1:1 connection. This means that you'd have to have a proxy for each MCP StatefulSet in order to work. Otherwise if you have the issue with responses being returned to the wrong users - unless of course you have a bunch of complexity in the proxyrunner. Either way its quite a big technical task architecturally at the moment which we're constantly discussing internally within the project - so stay tuned!

[olamide226]

@ChrisJBurns understood. The original reason for this BUG was because we wanted to scale down to ZERO to temporarily stop the server. how do you suggest we stop an MCPServer for the aim of starting it again(without deleting the resource)? maybe some spec.Suspend attribute added to the proxy runner?

[olamide226]

@ChrisJBurns @JAORMX After giving it some thought, given the current architecture (operator owns the proxy Deployment; proxyrunner owns the backend StatefulSet), the cleanest transport‑agnostic stop/start/restart is to make the operator the source of truth for backend lifecycle, not just the proxy.

Two viable paths:

1. Minimal change (works with existing architecture)

• Stop: add spec.suspend (or spec.desiredState: Stopped) to MCPServer. In reconcile, if suspended:
  • scale proxy Deployment to 0
  • delete the backend StatefulSet + headless service (same logic you already have in finalizer)
• Start: set spec.suspend=false; operator scales proxy Deployment back to 1, proxyrunner
  recreates the StatefulSet.
• Restart: keep the current restart annotation on the CR, but update the proxyrunner to force a backend restart by:
  • deleting the StatefulSet pods (or bumping a pod‑template annotation on the StatefulSet) in the proxyrunner’s apply step, and
  • still rolling the proxy so it re‑applies the updated StatefulSet.

This keeps transport agnostic and avoids scaling complexities.

2. Structural change (cleanest long‑term)

• Move backend StatefulSet creation into the operator (controller‑runtime), and make proxyrunner
  JUST a proxy process.
• Then spec.suspend, spec.restartAt, etc. are all enforced directly by the operator against the
  StatefulSet, and we're no longer fighting a second reconciler (proxyrunner) that re‑applies
  replicas=1.

Right now, manual scaling down(replica=0) gets “reset to 1” because proxyrunner’s k8s runtime applies the
StatefulSet with replicas=1 on startup. That will continue unless we introduce a true lifecycle
signal (like spec.suspend) that proxyrunner honors or until we move that workload management into the
operator territory.