upgrade k8s to 1.24 (or later) on dev and prod

[mbjones]

Our infrastructure is currently running k8s v 1.23.3 on both dev and prod clusters. Due to changes in how serviceaccount tokens are handled, we need to upgrade to 1.24 or 1.25 on both environments. In addition, 1.24 finalized the move away from support for dockershim, we will need to move to a new container runtime at the same time that we do the upgrade. From my reading, it seems like containerd directly is a good choice, but we should consider others. As we already have containerd installed, it might just be a reconfiguration away from using dockershim.

Because we are currently not running in a high availability configuration, this upgrade will probably require downtime as we upgrade the control plane node. Once it is upgraded, the worker nodes should be able to be updated without further downtime.

• Decide whether to upgrade to 1.24 or to continue on to 1.25
• Decide on a container runtime
• Upgrade dev-k8s
  • reconfigure to use container runtime
  • control plane
  • worker nodes
  • test deployments for compatibility
• Upgrade dev-k8s
  • reconfigure to use new container runtime
  • control plane
  • worker nodes

[mbjones]

Related to upgrading the OS on these hosts, adding additional nodes to the clusters (#31), and reconfiguring for a HA control plane (#1).