[question][CA] Why do we need cordon node in Refresh function mcmMananger when CA cordon itsel

[name212]

What would you like to be added:
N/A
Why is this needed:
N/A

Because the repository does not have an issue with the question type, I will ask my question here.

We have this method in Refresh in mcmManager

autoscaler/cluster-autoscaler/cloudprovider/mcm/mcm_manager.go

Line 442 in 36b3488

func (m *McmManager) Refresh() error {

which goes through all NodeGroups (aka MachineDeployment) gets all nodes from the annotations for deletion and cordons these nodes.

autoscaler/cluster-autoscaler/cloudprovider/mcm/mcm_cloud_provider.go

Line 358 in 36b3488

err = ngImpl.mcmManager.cordonNodes(toBeDeletedNodeNames)

At the same time, the autoscaler itself cordons nodes here as well

autoscaler/cluster-autoscaler/utils/taints/taints.go

Line 169 in 36b3488

return AddTaints(node, client, []apiv1.Taint{taint}, cordonNode)

https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/core/scaledown/actuation/actuator.go#L382
https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/core/scaledown/actuation/actuator.go#L212
https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/core/scaledown/actuation/actuator.go#L123

[name212]

@aaronfern @rishabh-11 I'm sorry, I call you out as the authors this pr's
#341
#344

31815d2

[name212]

The problem we encountered. We use a patched mcm https://github.com/deckhouse/mcm of a fairly old version and together with autoscaler 1.30.2 nodes were stuck in the cordon state and were not deleted.

We also have such warnings now:

root@nmit-delete-12-03-master-0:~# kubectl -n d8-cloud-instance-manager logs deployments/cluster-autoscaler | grep -e "^E"
Defaulted container "cluster-autoscaler" out of: cluster-autoscaler, kube-rbac-proxy
root@nmit-delete-12-03-master-0:~# kubectl -n d8-cloud-instance-manager logs deployments/cluster-autoscaler | grep -e "^W"
Defaulted container "cluster-autoscaler" out of: cluster-autoscaler, kube-rbac-proxy
W0424 13:44:23.003878       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0424 13:44:23.032320       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0424 13:44:33.151189       1 clusterstate.go:477] AcceptableRanges have not been populated yet. Skip checking
W0424 13:44:33.151281       1 clusterstate.go:477] AcceptableRanges have not been populated yet. Skip checking
W0424 13:45:03.333540       1 priority.go:90] Priority expander: node group nmit-delete-12-03-worker-cde1d229 not found in priority expander configuration. The group won't be used.
W0424 13:45:03.333550       1 priority.go:90] Priority expander: no priorities info found for any of the expansion options. No options filtered.
W0424 13:45:13.408790       1 clusterstate.go:492] Failed to find readiness information for nmit-delete-12-03-worker-cde1d229
W0424 17:15:18.450357       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:15:28.463747       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:15:38.475189       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:15:48.482555       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:15:58.489768       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:16:08.515250       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:16:18.522934       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:16:28.528395       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:16:38.537696       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:16:48.549218       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]
W0424 17:16:58.558623       1 mcm_cloud_provider.go:350] NodeGroup.Refresh() of "nmit-delete-12-03-worker-cde1d229" could not find Machine objects for toBeDeletedMachineNames ["nmit-delete-12-03-worker-cde1d229-c7d86-f4bmx"]

[aaronfern]

Hello @name212
Apologies for the very delayed repsonse

autoscaler/cluster-autoscaler/utils/taints/taints.go

Line 169 in 36b3488

return AddTaints(node, client, []apiv1.Taint{taint}, cordonNode)

In the function above, if you go up the caller and look at the value for cordonNode, it comes from a.ctx.CordonNodeBeforeTerminate which comes from a flag --cordon-node-before-terminating whose default value is false. If that is set to true by default, then we can probably consider removing our cordon nodes logic. (There is an open issue for that: kubernetes#8176)

When using the mcm provider, we preferred doing the node cordoning and the nodegroup (machinedeployment) size adjustment in the same atomic function (see https://github.com/gardener/autoscaler/blob/machine-controller-manager-provider/cluster-autoscaler/cloudprovider/mcm/mcm_cloud_provider.go#L389)
We took this call as we saw some instances of CA issuing incorrect scale-in requests (see #342 for one of the issues). We also thought this was necessary since we didn't want unscheduled pods to be allocated to nodes that the CA marked for deletion. This can happen if CA crashes and restarts and cleans all taints.

---

Regarding the problem you encountered, is your forked mcm in sync with https://github.com/gardener/machine-controller-manager?
when #341 was filed, we also files a PR in MCM (gardener/machine-controller-manager#965) that would be needed. Perhaps you an try syncing your fork as I see in https://github.com/deckhouse/mcm that some changes are missing

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues.
This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Mark this issue as rotten with /lifecycle rotten
• Close this issue with /close

/lifecycle stale

@gardener-ci-robot Command /add is not available to you but only to a Maintainer, Member, Author.

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues.
This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 60d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 60d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as active with /lifecycle active
• Mark this issue as frozen with /lifecycle frozen
• Mark this issue as fresh with /remove-lifecycle stale
• Mark this issue as rotten with /lifecycle rotten
• Close this issue with /close

/lifecycle stale

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues.
This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 60d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 60d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as active with /lifecycle active
• Mark this issue as frozen with /lifecycle frozen
• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close

/lifecycle rotten

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues.
This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 60d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 60d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as active with /lifecycle active
• Mark this issue as frozen with /lifecycle frozen
• Mark this issue as fresh with /remove-lifecycle rotten

/close