KO does not follow k8s version skew policy. Cannot upgrade CP when kubelet has an older minor release.

[PoudNL]

Currently we have a Kubeone provisioned cluster running Kubernetes 1.23.17. This cluster has 3 control-plane nodes and approx. 35 worker nodes. Normally we upgrade the CP nodes to the next minor release and rollover the worker nodes to the same version.
This last step takes a couple of days in our environment, unfortunately speeding up that process is currently not possible.

Because we are running such an old version of Kubernetes, we would like to upgrade to 1.24.x on the control-planes and immediately upgrade to 1.25.x without first upgrading the worker nodes. This is in line with the Kubernetes version skew policy, because CP components can have a version skew of max 2 minors with Kubernetes 1.23.17.

Kubeone documentation states that it follows the version skew policy of Kubernetes, but unfortunately the upgrade from 1.24.x -> 1.25.x with kubeone 1.6.x fails due too the version skew. Kubeadm will report a warning message to let the administrator know that it is not recommended. This warning message will result in Kubeone to fail the process.

[192.168.199.141] [upgrade] Running cluster health checks
[192.168.199.141] [upgrade/version] You have chosen to change the cluster version to "v1.25.16"
[192.168.199.141] [upgrade/versions] Cluster version: v1.24.17
[192.168.199.141] [upgrade/versions] kubeadm version: v1.25.16
[192.168.199.141] [upgrade/version] FATAL: the --version argument is invalid due to these errors:
[192.168.199.141]
[192.168.199.141]       - There are kubelets in this cluster that are too old that have these versions [v1.23.17]
[192.168.199.141]
[192.168.199.141] Can be bypassed if you pass the --force flag
[192.168.199.141] To see the stack trace of this error execute with --v=5 or higher
WARN[21:59:12 CET] Task failed, error was: runtime: running task on "192.168.199.141"
ssh: running kubeadm upgrade on control plane leader
ssh: popen
Process exited with status 1

How can I make Kubeone force the update? I found some documentation about using kubeone upgrade instead of kubeone apply , but this feature is deprecated. Also I don't see any lines in the code that would add the --force parameter to the kubeadm command when upgrading.

So does anyone know how to solve my journey and let Kubeone indeed follow the version skew policy of Kubernetes?

[embik]

This error message is returned from kubeadm attempting to run on the control plane node. I couldn't find any documentation on this quickly and don't know why it wouldn't allow this as per the skew policy, but if it indeed is a kubeadm safe-guard it's probably not a good idea to override it.

[PoudNL]

That's true. it is maybe not best practice but definitely compatible for a short period of time. Even since 1.25 it will be possible to have a version skew of 2 minor versions between CP services and kubelet.

[xmudrii]

I provided some more details in the issue #3010 (comment)