Research on joining nodes in older versions for Kubernetes

[fabriziopandini]

Kubeadm version skew policies defines following rules

1. kubeadm can be used with Kubernetes components (aka control plane) that are the same version as kubeadm or one version older.
2. kubeadm can be used with a kubelet version that is the same version as kubeadm or three versions older
3. If new nodes are joined to the cluster, the kubeadm binary used for kubeadm join must match the last version of kubeadm used to either create the cluster with kubeadm init or to upgrade the same node with kubeadm upgrade

In Cluster API, assuming that kubeadm binary is backed into images and it is always of the same version of the Kubernetes version existing in the image, 1 and 3 might become an issue when you try to join machines with an older Kubernetes versions than the control plane, because we are going to use a kubeadm version older than the control plane version e.g.:

• During an upgrade to 1.35, if you scale up a MD still at 1.34, you are going to use kubeadm 1.34 -> control plane 1.35 (violates skew)
• During an upgrade to 1.35,, if a remediation happen on MD still at 1.34, you are going to use kubeadm 1.34 -> control plane 1.35 (violates skew)

Those issue were mitigate by several factors.

• In most of the cases, violating this policy does not lead to issue, because kubeadm is pretty stable nowdays.
• In managed topology, cluster upgrade is a single operation, and thus you might assume that version skew between CP and workers only exists for a short period of time
• In managed topology, during upgrade the system is designed to minimize rollouts, so it is expected and acceptable to defer changes.
• Over time we introduced MachineSet preflight checks preventing scale up or machine remediation to start when this can potentially lead to issue

Those mitigation worked well in the past, but while they started to become not ideal with chained upgrades, now that we are starting looking into #12466, those mitigations are becoming an issue, because we are going to support having workers in older versions for a longer period, and we do expect that MD/MP with pinned K8s versions will be fully operational.

This issue is about investigating how to overcome current system constraints. A few initial points to think about:

• How can we make kubeadm join possible on machines with older k8s versions

  • Is upgrading kubeadm in flight a viable option?
    • How do we upgrade kubeadm in flight?
    • How do we prevent upgrading kubeadm in flight violating other kubeadm skew rules (e.g. 2 from the initial list)?
    • Do we need more than upgrade kubeadm in flight (e.g. provide a different version of the kubeadm join configuration)?
  • Are there alternatives to upgrading kubeadm in flight a viable option?
  • How are we introducing this feature in CAPI?
    • Will this be implemented in core CAPI or deferred to users like ~everything we do on machine initialisation?
    • How we make this extensible / configurable by providers (if required)
    • Will this work on all the existing old images or only on new images

• Assuming we have a solution for kubeadm join on machines with older k8s versions

  • How do we make it possible to disable existing preflight checks preventing this operation

• How do we balance the fact that we are enabling operations on workers with older K8s versions, with the fact that we still want to minimize the number rollouts during upgrades?

Note: there might also more subtle issue when version skew exists in the cluster, might be worth to take a look at kubernetes/enhancements#4020 and related work

[ivelichkovich]

How do we upgrade kubeadm in flight?

We are working on a POC for this

How do we prevent upgrading kubeadm in flight violating other kubeadm skew rules (e.g. 2 from the initial list)?

This should be handled by the existing skew preflight checks, basically if the machine is within supported kubelet version skew with control plane (which we already check) then the kubeadm skew will be safe and supported

Do we need more than upgrade kubeadm in flight (e.g. provide a different version of the kubeadm join configuration)?

Yes we will need to make sure the cluster version is used for the kubeadm join config api version, this will be part of the POC.

also linking kubeadm docs on supported skew for posterity

[ivelichkovich]

Will this be implemented in core CAPI or deferred to users like ~everything we do on machine initialisation?

IMO ensure the correct kubeadm should be part of core capi because its common functionality almost all users would need

How we make this extensible / configurable by providers (if required)

Curious to hear extensibility use cases but we'd be happy to look into it if they come up

Will this work on all the existing old images or only on new images

ideally it will be seamless and not require image changes so it would work with old and new.

[sbueringer]

IMO ensure the correct kubeadm should be part of core capi because its common functionality almost all users would need

I think in general it should not be core CAPI, at most it should be CABPK. CAPI today has no insight into the bootstrap process. And this issue seems very specific to kubeadm.

I think the challenge if we implement it in CABPK is making this work across various operating systems (various Linux distros, Windows, ...) and across various options where the kubeadm binary might be coming from.

In Cluster API, assuming that kubeadm binary is backed into images and it is always of the same version of the Kubernetes version existing in the image, 1 and 3 might become an issue when you try to join machines with an older Kubernetes versions than the control plane

Just wanted to note that 1 should not be an issue with KCP, as KCP will never try to join a Machine with an older Kubernetes version.

[AcidLeroy]

@sbueringer Quick clarification on control plane rolling upgrades (e.g. 1.34 → 1.35): as I understand it, we scale up first and create a new machine with spec.version 1.35 (from KCP). That machine’s bootstrap uses kubeadm 1.35 for kubeadm join (CABPK’s joinControlplane() uses scope.ConfigOwner.KubernetesVersion(), i.e. the new machine’s version). So we’re effectively doing kubeadm 1.35 join into a cluster that was initialized with kubeadm 1.34 and whose existing nodes are still 1.34. So isn't CAPI already violating the skew policy of kubeadm <-> kubeadm?

[sbueringer]

I took another look at the policies and I think you are right. kubeadm does not seem to support basically "upgrade-by-join".

I think what is special about KCP is that KCP implements enough of kubeadm upgrade before it triggers a join with a new version. That makes it different from how "regular" kubeadm users are using kubeadm.

xref:

cluster-api/controlplane/kubeadm/internal/controllers/update.go

Line 56 in 41c4d39

if err := workloadCluster.AllowClusterAdminPermissions(ctx, parsedVersion); err != nil {

++

[neolit123]

@AcidLeroy i'm replying to your question in the kubeadm tracker:

The documentation for the skew policy for kubeadm to itself states that the binaries must be the exact version. In practice, is this strictly required? When doing a rolling upgrade in cluster-api, I believe the kubeadm bootstrap provider relies on joining nodes to an existing cluster using the new version's kubeadm. i.e., if a cluster is at 1.34, and you want to upgrade it to 1.35, a new node comes online and essentially runs a kubeadm join to the existing 1.34 cluster. I'm wondering if this is a pattern seen elsewhere or if this is considered bad practice.

kubeadm is stable enough for this to not matter much, but the moment something changes, the user that joins a 1.35 node to a cluster created by 1.34 they will see an error. for example, 1.35 added some new RBAC which is not yet present in 1.34.
the obvious workaround would be to apply the RBAC before joining the 1.35 node.

another interesting point is what happens with the ClusterConfiguration and component configs that are stored as objects. normally kubeadm upgrade would mutate them, but kubeadm join would not, so the user must make manual changes.

that is why CAPI has the stepping stone before rolling upgrade which @sbueringer mentioned.

the kubeadm team has arguably no bandwidth to support the kubeadm skew with itself e.g. N-1, and we could have some guard rails on best effort to not have super braking changes around the rolling join scenarios, i guess. but the policy is quite accurate as it is.

[ivelichkovich]

So in the example of 1.35 having some new RBAC, would CABPK/KCP be expected to be aware of the changes in kubeadm 1.34 to 1.35 and apply that RBAC when an upgrade is started from 1.34 to 1.35?