Scale-down skips decommission when a stale scylla/decommissioned label persists on the member Service, destroying local data

[nkolev-mparticle]

What happened

A rack scale-down (4 → 3 members) shut a member down without decommissioning it. On local-storage clusters this is destructive: the PVC is deleted, local-static-provisioner reclaims the disk within ~10 seconds, and the member's data is gone while it remains in the ring as a dead node owning its full token range.

Result: ~21 TB orphaned, the affected token ranges dropped to RF-1, and every subsequent topology operation on the cluster was blocked (decommission … Cannot start: nodes={<dead-host-id>}) until we ran nodetool removenode and waited hours for re-replication.

Root cause

pkg/controller/scylladbdatacenter/sync_statefulsets.go records decommission intent only when the label is absent:

if len(lastSvc.Labels[naming.DecommissionedLabel]) == 0 {
    lastSvcCopy := lastSvc.DeepCopy()
    // Record the intent to decommission the member.
    // TODO: Move this into syncServices so it reconciles properly. This is edge triggered
    //  and nothing will reconcile the label if something goes wrong or the flow changes.
    lastSvcCopy.Labels[naming.DecommissionedLabel] = naming.LabelValueFalse
    ...
    return progressingConditions, nil
}

If the member Service already carries scylla/decommissioned with any non-empty value — including a stale "true" from an earlier lifecycle of that same ordinal — this block is skipped. Control falls through to UpdateScale, the StatefulSet shrinks, and the pod is deleted with no decommission ever issued. The // Wait if any decommissioning is in progress guard immediately above also passes vacuously, because it only waits on == LabelValueFalse.

The existing TODO anticipates that the label is unreconciled. What does not appear to be recognised is the consequence: a stale label silently converts a decommission into an ungraceful removal, and on local storage that is unrecoverable data loss.

Evidence

Operator log at the moment of scale-down — no intent recorded, straight to UpdateScale (line 641 in v1.18.1):

I0731 15:18:05.799620  1 scylladbdatacenter/sync_statefulsets.go:641] "Scaling StatefulSet" ...

The member's own log shows a clean shutdown rather than a decommission, and gossip agrees:

STATUS:shutdown,true          # not LEFT
LOAD:23063591767157

Local volume reclaimed ~70 seconds after the scale-down:

15:19:15  local-static-provisioner  Deleted pv "local-pv-XXXX" from cache
15:19:25  local-static-provisioner  Found new volume at host path "/mnt/fast-disks/…"  capacity 59997849387008, creating Local PV
15:19:25  local-static-provisioner  Created PV "local-pv-XXXX"

A second rack scaled down in the same apply behaved correctly — its Service had no pre-existing label, so intent was recorded (scylla/decommissioned=false) and the operator waited. That decommission then could not start, blocked by the dead node from the first rack:

node_ops - decommission[…]: Started decommission operation: node=<self>
WARN node_ops - decommission[…]: Cannot start: nodes={<dead-host-id>}

So the two racks in one apply diverged purely on whether a stale label was present.

Affected versions

The gate and the TODO are byte-identical in v1.18.1, v1.19.2, v1.20.2, v1.21.0 and master. Diffing the region from // Wait if any decommissioning is in progress through UpdateScale between v1.18.1 and master shows no change, and no validation of the label against actual ring state was added.

Observed on v1.18.1, ScyllaDB 2025.4.5, storageClassName backed by local-static-provisioner with persistentVolumeReclaimPolicy: Delete.

Suggested fixes

1. Do not trust the label as proof of completion. Before scaling down, verify against the cluster that the member has actually left — e.g. it is absent from the ring, or its gossip STATUS is LEFT. A label saying true while the node is still NORMAL/present in the ring should block the scale-down, not authorise it.
2. Reconcile the label as the TODO suggests, so it cannot survive as stale state across an ordinal's lifecycle.
3. Fail closed on ambiguity. If the label is true but the member is still a ring participant, that is a contradiction worth surfacing as Degraded rather than proceeding.

Even fix 1 alone would have prevented the data loss here.

Workaround for other operators

Before any scale-down, assert the target ordinal's Service does not already carry scylla/decommissioned=true:

kubectl -n <ns> get svc <cluster>-<dc>-<rack>-<ordinal> \
  -o jsonpath='{.metadata.labels.scylla/decommissioned}'

If it is true while the member is still in the ring, do not scale down.

[czeslavo]

@nkolev-mparticle Thank you for the detailed report.

I'd like to understand how the node actually got into this state with a stale label from the previous cycle - the decommissioned label shouldn't normally be left on a service after a successful decommissioning, as the service gets pruned afterwards.

Was there ever an earlier scale-down of this rack that was reverted in-flight, i.e., members lowered and then raised back before the decommissioning concludes (meaning - decommissioning operation completes on a node, decommissioned=true is stamped, and the controller prunes a service)? If so, were there any manual recovery actions executed to bootstrap the node after it was decommissioned?

Would you be able to reconstruct (e.g., from kube audit logs) what client set the label on the service and when, to ensure it was the operator, not a manual operation?