Upgrade to Kubernetes 1.27

[smerle33]

Service(s)

Other

Summary

Previous upgrade (1.26): #3683

Depreciation timelines for 1.26 (justifying the upgrade to 1.27):

• Digital Ocean: 28 February 2024 - https://docs.digitalocean.com/products/kubernetes/details/supported-releases/
• Azure AKS: Mar 2024 - https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar
• AWS EKS: June 11, 2024 - https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar

---

Task list:

• Upgrade kubectl within packer-images AllInOne

  • First upgrade the updatecli manifest chore(updatecli) track kubernetes 1.27.x baseline packer-images#1048
  • Then apply the updatecli PR that is produced by this change: Bump kubectl version to 1.27.10 packer-images#1050
  • Then Apply the Kubernetes-Management PR that will use this new version - https://github.com/jenkins-infra/packer-images/releases/tag/1.53.2 and Bump agent templates for infra.ci.jenkins.io (packer-image 1.53.2) kubernetes-management#4978

• ⚠️ Upgrade kubectl within puppet https://github.com/jenkins-infra/jenkins-infra/blob/bd80ebe3a412702b05165f83a8fc3ba92c4bc891/hieradata/clients/agent.trusted.ci.jenkins.io.yaml#L15 and https://github.com/jenkins-infra/jenkins-infra/blob/bd80ebe3a412702b05165f83a8fc3ba92c4bc891/hieradata/vagrant/common.yaml#L388 OR THE UPDATECLI manifest that update those. https://github.com/jenkins-infra/packer-images/blob/6c90ff53c1b616979d02d61e834da3a9f588ae74/updatecli/updatecli.d/kubectl.yaml#L29 don't forget to deploy the image and then use the PR produced by updatecli on kubernetes-management and jenkins-infra repositories to upgrade the image for the AllInOne ⚠️

• Upgrade DOKS (doks and doks-public) - see below - Upgrade to Kubernetes 1.27 #3948 (comment)

• Upgrade AWS EKS (cik8s and eks-public) - see below - Upgrade to Kubernetes 1.27 #3948 (comment)

• Upgrade AKS (privatek8s and publick8s) - see below - Upgrade to Kubernetes 1.27 #3948 (comment) AND Upgrade to Kubernetes 1.27 #3948 (comment)

Reproduction steps

No response

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% Upgrade to Kubernetes 1.25 #3582
2. 92% Upgrade to Kubernetes 1.24 #3387
3. 92% Upgrade to Kubernetes 1.23 #3053
4. 92% Upgrade to Kubernetes 1.22 #2930
5. 92% Upgrade to Kubernetes 1.21 #2866
6. 77% [INFRA-3118] Upgrade to Kubernetes 1.20 #2664

[smerle33]

Digital Ocean Clusters (as per 1.26 here #3683 (comment)):

• Changelog for Kube 1.27 to be checked with a focus on kubernetes/kubernetes@master/CHANGELOG/CHANGELOG-1.26.md#urgent-upgrade-notes
• DigitalOcean have their own changelog for their managed components: https://docs.digitalocean.com/products/kubernetes/details/changelog/#1-27-x
• Preparing upgrade of the cluster, the 2 at the same time (doks-public and doks) no need to disabling or announce for now as it is already unused ... but need to be check is we re activate DO.
  • Disable kubernetes-management of both clusters (PR) - feat(doks, doks-public) disable clusters management kubernetes-management#4982
  • Bump Kube version to 1.27 for doks-public
    • Terraform PR - feat(doks-public) bump Kubernetes version from 1.26.to 1.27 digitalocean digitalocean#179 [ │ Error: Unable to upgrade cluster version: POST https://api.digitalocean.com/v2/kubernetes/clusters/<redacted>/upgrade: 422 (request "<redacted>") **invalid upgrade path**]
    • Sanity check with a local helmfile diff (need to have no change) see comment below for DigitalOcean Token Upgrade to Kubernetes 1.27 #3948 (comment)
  • Bump Kube version to 1.27 for doks
    • Terraform PR - feat(doks): upgrade kubernetes from 1.26 to 1.27 digitalocean#186
      Upgrade triggered manually through the DO portal
    • Sanity check with a local helmfile diff (need to have no change))
  • Terraform jenkins-infra/digitalocean project need to have a successful build without changes on the main branch after the upgrades
  • Re-Enable kubernetes-management of both clusters (PR) - feat(doks/doks-public) re-enable digital ocean on kubernetes-management kubernetes-management#4992

Post Mortem

The upgrade as-code failed with the error Error: Unable to upgrade cluster version: POST https://api.digitalocean.com/v2/kubernetes/clusters/<redacted>/upgrade: 422 (request "<redacted>") **invalid upgrade path**]

The Digital Ocean UI explained a problem to upgrade to the latest patch version (handled by digital ocean directly on Sundays) : #3948 (comment)

We add to correct the webhook timeout to proceed the patch upgrade (through Digital Ocean UI) and then remove a "hack" we used (using a data source to configure the Kubernetes provider and decouple it from the cluster resource itself).

The Upgrade as-code went then smoothly.

As such, the upgrade procedure is amended to the following workflow:

• Disable Kubernetes-Management for Digital Ocean clusters
• Keep the terraform PRs (one or 2 clusters at a time, both can be done)
• Check the plans
• Merge the PRs and check the output
• Enable Kubernetes-Management for Digital Ocean clusters

[smerle33]

For Digital Ocean, to use terraform plan on local computer, we need to provide a token (no doctl auth)
export DIGITALOCEAN_ACCESS_TOKEN="$(security find-generic-password -a doctl-token-infra -w)";

[dduportal]

Update for DigitalOcean (on behalf of @smerle33 ):

• The upgrade through Terraform was not possible during previous upgrades due to a bug with (Terraform 1.1 + DigitalOcean terraform provider + Kubernetes provider) in the same project. Now that we have Terraform 1.6+, the "hack" we used (using a data source to configure the Kubernetes provider and decouple it from the cluster resource itself) can be removed. See feat(doks-public) bump Kubernetes version from 1.26.to 1.27 digitalocean digitalocean#179

• The upgrade of doks-public from 1.26.12-do.0 to 1.27.10-do.0 looked good on feat(doks-public) bump Kubernetes version from 1.26.to 1.27 digitalocean digitalocean#179 but failed with the following error:

  Error: Unable to upgrade cluster version: POST https://api.digitalocean.com/v2/kubernetes/clusters/<redacted>/upgrade: 422 (request "<redacted>") invalid upgrade path]
  

• Looking at the DigitalOcean console showed there was a patch upgrade to 1.26.13-do0. This updates should have been applied last Sunday automatically (as per https://github.com/jenkins-infra/digitalocean/blob/41f59716187b14fcc14ba4d4ff14eb8015db2c6f/doks-public-cluster.tf#L21-L24) but it did not.

  • Trying to apply the patch manually from the DigitalOcean cloud console failed with the following errors (along with the 30-ish usual warnings related to hostpath and resource requests/limits due to datadog and certmanager):

    

    Validating webhook with a TimeoutSeconds value smaller than 1 second or greater than 29 seconds will block upgrades.
    Resources:
        validating webhook configuration: cert-manager-webhook
    

    Mutating webhook with a TimeoutSeconds value smaller than 1 second or greater than 29 seconds will block upgrades.
    Resources:
        mutating webhook configuration: cert-manager-webhook
    

• Short term actions: we decided to start by upgrading to the latest patch so we can retry the upgrade 1.26.x to 1.27.x again with terraform.

  • Manually editing both mutating and validationg webhooks of cert-manager: their default timeout is 30 seconds: we changed it to 25 (note: both doks-public and doks cluster are removed from kubernetes-management temporarily so no risk of being overriden for now)
  • Ran again the patch upgrade process in the UI for doks-public: no more error.
  • Applied with success the 1.26.13-do0 patch to doks-public
  • Reverted feat(doks-public) bump Kubernetes version from 1.26.to 1.27 digitalocean digitalocean#179: the build of the DigitalOcean on the main branch is now green.

[dduportal]

Next steps:

• Delaying upgrade to 1.27.x to next monday (19 Feb. 2024) as @smerle33 is not available today anymore
• Bump doks to latest 1.26.x (includes manual change to its cert-manager webhooks if any) => @dduportal today
• Check validation in the DO UI console for the 1.27.x upgrade (just in case other errors appears) => @dduportal today
• Remove the "data source" in Digital Ocean's terraform (same as feat(doks-public) bump Kubernetes version from 1.26.to 1.27 digitalocean digitalocean#179 and feat(doks) bump Kubernetes version from 1.26 to 1.27 digitalocean digitalocean#180 but without the 1.27.x bump) => @dduportal today
• Adapt cert-manager setup on DO to have a 25s timeout for the mutating/validating webhooks (@smerle33 will do this on monday while upgrading to 1.27.x)

[smerle33]

doks-public seems ok for upgrade:

[smerle33]

need to create a updatecli manifest to track kubectl on trusted agent and add it to the "process" for upgrading with the kubectl cli for packer-images.

[smerle33]

to get last addon versions :

aws eks describe-addon-versions --profile prod  --kubernetes-version 1.27 --addon-name coredns --region us-east-2
aws eks describe-addon-versions --profile prod  --kubernetes-version 1.27 --addon-name kube-proxy --region us-east-2
aws eks describe-addon-versions --profile prod  --kubernetes-version 1.27 --addon-name vpc-cni --region us-east-2
aws eks describe-addon-versions --profile prod  --kubernetes-version 1.27 --addon-name aws-ebs-csi-driver --region us-east-2

coredns ; v1.10.1-eksbuild.7
kube-proxy: v1.27.10-eksbuild.2
vpc-cni: v1.16.4-eksbuild.2
aws-ebs-csi-driver: v1.28.0-eksbuild.1

[smerle33]

AZURE

Update: AKS Upgrade plan privatek8s

• Check changelog:

Current changelog notable elements for Kubernetes 1.27:

privatek8s

• Announcement: date/time UTC - 18 March 2024 - 15h30 UTC
  • Open datetime announcement on status.jenkins.io - open/kub/1.27/privatek8s status#487
  • jenkins-infra mailing - https://groups.google.com/g/jenkins-infra/c/5Exw280pRYE
  • IRC #jenkins-infra - https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$IdAyQ2Mu8uGCEInl4tToccrnI3orN2m0vnls7OPfUwc?via=g4v.dev&via=gitter.im&via=matrix.org
• Disable kubernetes management for privatek8s cluster (remove it from Jenkinsfile_k8s) - chore: remove privatek8s cluster from the pipeline kubernetes-management#5053
• Terraform PR to upgrade cluster to 1.27 but NO merge - feat(privatek8s): bump kubernetes to 1.27.9 for privatek8s azure#643
• Disable terraform azure job (⚠️ As the upgrade will put infra.ci out of service so need to be run manually)
• Make sure local (on your machine) repository is up to date and you are logged-in: run a terraform plan
• Merge Terraform PR even without checks (as infra.ci restart will re-enable job)
• Perform upgrade from local machine
• Check that helmfile diff still works and shows no change and no warning
• Wait for infra.ci to be up again
• Enable kubernetes management for privatek8s cluster and check it runs successfully -
• Check terraform job and eventually trigger it for success with no changes
• Close announcements

[smerle33]

AZURE

Update: AKS Upgrade plan publick8s

publick8s

• Announcement: date/time UTC
  • Open datetime announcement on status.jenkins.io - <open publick8s kub 1.27 migration status#489>
  • jenkins-dev and jenkins-infra mailing lists - https://groups.google.com/g/jenkinsci-dev/c/AzxFjO2T8qU
  • IRC #jenkins-infra - https://status.jenkins.io/issues/2024-19-03-maintenance-az-publick8s-1-27/
• Disable kubernetes management for publick8s cluster (remove it from Jenkinsfile_k8s) - feat(publick8s) disable clusters management kubernetes-management#5068
• Terraform PR to upgrade cluster to 1.27 - bump kubernetes to 1.27.9 for publick8s azure#646
• Merge Terraform PR and let the upgrade happen
• Check terraform job and eventually trigger it for success with no changes
• Check that helmfile diff still works and shows no change and no warning
• Enable kubernetes management for publick8s cluster and check it runs successfully -
• Check for services healthiness (pagerduty, datadog, manual control, logging in on ci.jenkins.io, etc.)
• Close announcements

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

we took the opportunity to deploy nginx-ingress jenkins-infra/kubernetes-management#5032 and falco jenkins-infra/kubernetes-management#4997 upgrade.
This process should be added to the preparation for each upgrade of kubernetes, to take advantage of the downtime for important upgrades like those

[smerle33]