Upgrade to Kubernetes 1.22

[smerle33]

Previous upgrade: #2866

• Upgrade kubectl within docker-helmfile

• Upgrade DOKS

  • Ensure cluster is re-created by terraform in jenkins-infra/digitalocean
    • Initial Kubernetes 1.22 upgrade digitalocean/variables.tf kubernetes upgrade from 1.21 to 1.22 digitalocean#43
  • Add admin's kubeconfig in the infra.ci's SOPS credentials: (private repo)
  • Ensure that cluster is still off and not to be used by ci.jenkins.io while no more credit to use are available
  • Add the credential of the service account jenkins-agents in ci.jenkins.io (manual step)

• Upgrade EKS

  • Open datetime announcement on status.jenkins.io (next time)
  • Disable cluster in ci.jenkins.io's config as code
    • need to remove the whole block (0 as max-capacity is not stopping agent to be spawn) https://github.com/jenkins-infra/jenkins-infra/blob/production/hieradata/clients/azure.ci.jenkins.io.yaml#L41-L130
  • Upgrade cluster through Terraform (control plane + node groups)
  • Upgrade add-ons (weird: their version are not upgraded by the EKS module. Need to be specified as terraform variable + updatecli-zed from the tables at https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html, https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html and https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html)
    • if done by UI, select the checkbox "replace existing configuration"
    • and for vpc-cni select the inheritance from node.
  • Ensure that upgraded cluster is still ok (check kubernetes-management)
  • Enable it again in ci.jenkins.io config as code

• Upgrade AKS

  • Check previous AKS upgrades and create one for 1.22
  • Open datetime announcement on status.jenkins.io
  • Upgrade temp-privatek8s first. ⚠️ it will put infra.ci out of service so might need manual steps to re-boostrap the kubernetes-management job
  • Upgrade prodpublick8s
  • Close announcement

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% Upgrade to Kubernetes 1.21 #2866
2. 77% [INFRA-3118] Upgrade to Kubernetes 1.20 #2664

[dduportal]

• Initial kubectl 1.22 PR: [updatecli] [kubectl] Update version to 1.22.x docker-helmfile#105, replaced by [updatecli] [kubectl] Update version to to 1.22.x  docker-helmfile#116
  • Was blocked until today by [updatecli] Bump jenkins-infra helmfile docker image kubernetes-management#2403 but it's now UNBLOCKED.
  • Ping @smerle33 , ready for you to continue on kubectl upgrade in [updatecli] [kubectl] Update version to to 1.22.x  docker-helmfile#116

[dduportal]

Ping @smerle33 : as jenkins-infra/digitalocean#43 is merged, it looks to me that DOKS is done?

[smerle33]

from ci.jenkins.io

[lemeurherve]

from ci.jenkins.io

... for the Digital Ocean cluster.

[dduportal]

👏

[dduportal]

Status discussed during today's weekly meeting:

• AKS clusterS planned thursday 30th of June starting at 07:00UTC (communications sent)
• Once all clusters are upgraded, you'll have to write up the issue "Upgrade to Kubernetes 1.23" issue with ALL the things that:
  • Were forgotten in the "todo list"
  • Went wrong
  • you detected to be on the 1.23 changelog

Once these 2 tasks are ok, we can close the issue :)

[daniel-beck]

Noting that LDAP and get.jenkins.io are currently unavailable due to problems.

[lemeurherve]

We’ve identified the problem and working on it, will post an update as soon as it's resolved.

[lemeurherve]

LDAP restored, next: get.jenkins.io (same kind of problem)

[lemeurherve]

get.jenkins.io restored too

[lemeurherve]

Upgrade to Kubernetes 1.22 completed, sorry for the longer outage than expected, preparing a short post-mortem.

[dduportal]

One last step: create the issue in jenkins-infra/helpdesk for 1.23 from the current one + add the improvements/changes in it, so it should be a no brainer next time :)

[lemeurherve]

Issue created.