KCP rollout with new config will stuck when there is an unhealthy APIServer node

[jessehu]

What steps did you take and what happened?

1. Create a 1 CP or 3 CP cluster successfully without turning on MHC.
2. Update the existing KCP with a wrong apiserver params (e.g. add a space between two names in clusterConfiguration.apiServer.extraArgs.tls-cipher-suites).
   • Then the new CP Node will become ready, but its APIServer can not start and the APIServerPodHealthy condition with False status is added on the CP Machine (shown in picture 1).
   • The KCP won't become ready forever as expected。
3. If update the KCP with a correct apiserver params (same as the params when creating the cluster), then the CP Machine with unhealthy APIServer is deleted, and the KCP will become ready as expected.
4. If update the KCP with a correct apiserver params (different from the params when creating the cluster), then KCP tries to delete the oldest ready CP Machine (not the one with unhealthy APIServer), and fails at KCP preflight check. So the KCP won't become ready forever (not as expected)。

What did you expect to happen?

In step 4, KCP controller should delete the Machine with unhealthy APIServer, then the KCP rollout can succeed.

Cluster API version

1.5.2

Kubernetes version

1.25.15

Anything else you would like to add?

The root cause is either reconcileUnhealthyMachines() for MHC or upgradeControlPlane(ctx, controlPlane, machinesNeedingRollout) / selectMachineForScaleDown() do not consider the Machine CR with false APIServerPodHealthy condition.

• https://github.com/kubernetes-sigs/cluster-api/blob/main/controlplane/kubeadm/internal/controllers/controller.go#L391-L407
• https://github.com/kubernetes-sigs/cluster-api/blob/main/controlplane/kubeadm/internal/controllers/scale.go#L226-L237

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[sbueringer]

/triage accepted