Upgrade to Kubernetes 1.24

[lemeurherve]

Previous upgrade: #3053

• Upgrade kubectl within docker-helmfile

  • First upgrade the updatecli manifest : chore(updatecli) track kubectl 1.24.x line docker-helmfile#266
  • Then apply the updatecli PR that is produced by this change: Bump kubectl version to 1.24.10 docker-helmfile#267
  • Then Apply the Kubernetes-Management PR that will use this new version - Bump jenkins-infra helmfile docker image to 2.5.69 kubernetes-management#3611

• Send an email (ideally 1-2 day priori to the operations) on both jenkinsci-dev and jenkins-infra mailing lists (google groups) to let users known "when" the operations will take place, and "what" will be the expected impacts

  • https://groups.google.com/g/jenkinsci-dev/c/pmWN0Bhl5Sg

• Upgrade DOKS (doks and doks-public)

  • [-] Open datetime announcement on status.jenkins.io for doks-public
  • [-] Disable cluster in ci.jenkins.io configuration as code
  • Disable kubernetes management (disable job on infra.ci.jenkins.io)
  • Bump Kubernetes version in jenkins-infra/digitalocean
  • Ensure cluster is re-created by terraform in jenkins-infra/digitalocean updated in DigitalOcean management interface
  • Re-Enable kubernetes management (and update kubeconfigs in the infra.ci's SOPS credentials (private repo), if kubernetes management cannot connect) and run a job on the main branch
  • Add the credential of the service account jenkins-agents in ci.jenkins.io if needed (manual step) doks only not doks-public
  • [-] Close announcement

• Upgrade EKS (cik8s and eks-public)

  • Open datetime announcement on status.jenkins.io - Create 2023-02-22-maintenance-eks-k8s-1-24.md status#245

  • Disable cluster in ci.jenkins.io configuration as code - chore(ci.jenkins.io): disable cik8s before updating it to Kubernetes 1.24 jenkins-infra#2653

  • Disable kubernetes management (disable job on infra.ci.jenkins.io)

  • Upgrade cluster through Terraform (control plane + node groups) - chore: update cik8s cluster to Kubernetes 1.24 aws#342

    • add-ons manually updated (see add updatecli manifest to track EKS addons versions aws#278 for next k8s upgrades)

  • Ensure the upgraded cluster is still OK (check kubernetes-management and run a job on the main branch)
    ⚠️ Ensure that the security group eks-cluster-sg-<cluster name>-<random ID> that is managed directly by EKS doesn't contain the tag kubernetes.io/cluster/<cluster name>
    Even with create_cluster_primary_security_group_tags = false this tag is wrongly set by the terraform module and must be manually deleted afterwards for now.

    

  • Enable it again in ci.jenkins.io config as code

  • Close announcement

• Upgrade AKS (privatek8s, publick8s and prodpublick8s)

  • [-] Check previous AKS upgrades and create one for 1.24

  • Open datetime announcement on status.jenkins.io

  • Disable kubernetes management (disable job on infra.ci.jenkins.io)

  • Upgrade privatek8s first
    ⚠️ It will put infra.ci out of service so might need manual steps to re-boostrap the kubernetes-management job

    • Version retrieved with az aks get-upgrades --resource-group prod-privatek8s --name privatek8s-emerging-ram --output table

    • Control plane upgraded with terraform

    • Nodes upgraded via the Azure Portal (for now)

      Details

      

  • Upgrade publick8s

  • Upgrade prodpublick8s

  • Re-Enable kubernetes management (and update kubeconfigs in the infra.ci's SOPS credentials (private repo), if kubernetes management cannot connect) and run a job on the main branch

  • Send an email on both jenkinsci-dev and jenkins-infra mailing lists to announce the end of maintenance

  • Close announcement

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% Upgrade to Kubernetes 1.22 #2930
2. 92% Upgrade to Kubernetes 1.21 #2866
3. 77% [INFRA-3118] Upgrade to Kubernetes 1.20 #2664

[dduportal]

We had an issue while upgrading cik8s on EKS with the following error message during the addons upgrade:

error updating EKS Add-On (<redacted>:aws-ebs-csi-driver): AccessDeniedException: User is not authorized to perform this action

Thanks to @lemeurherve 's researches, we found terraform-aws-modules/terraform-aws-eks#1725 which confirms that the Terraform EKS module is (was?) not exhaustive for the IAM eks required permissions.
We covered https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonelastickubernetesservice.html#amazonelastickubernetesservice-actions-as-permissions exhaustively to apply the missing permissions (mainly: eks:UpdateAddon 😅 but not only).

Applied the new permissions in the (private) repository jenkins-infra/terraform-state (ref. https://github.com/jenkins-infra/terraform-states/commit/14c134bdb778e7a6124ed398618c1a9f48cbf101) and triggered a new (successful) build.

[lemeurherve]

Upgrade of all our clusters to Kubernetes 1.24 successful 🎉