Enable nodelocal dnscache on prow build clusters

[chaodaiG]

No description provided.

[ameukam]

@chaodaiG How of curiosity, what's the reason for using this feature ?

EDIT: Saw the discussion kubernetes/test-infra#20716

[ameukam]

/assign @spiffxp @BenTheElder

[chaodaiG]

@chaodaiG How of curiosity, what's the reason for using this feature ?

It was because of this thread: kubernetes/test-infra#20716 (comment)

[BenTheElder]

/hold

[spiffxp]

/approve
/lgtm
This isn't actually updating k8s-infra-prow-build, but that's fine by me, I'll see what rollout looks like on k8s-infra-prow-build-trusted first

[spiffxp]

nit: this is unused right now

[spiffxp]

nit to self: I guess meant to keep the locals block for "vars that are going to be reused by multiple resources" vs. "all configurable things go up here", but I didn't comment as such (or do so consistently, e.g. bigquery_location doesn't below up here by such convention)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, chaodaiG, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[spiffxp]

spiffxp@spiffxp-macbookpro:prow-build-trusted (dns-cache %)$ terraform plan
#...
  # module.prow_build_cluster.google_container_cluster.prod_cluster[0] will be updated in-place
#...

[spiffxp]

/hold cancel

[spiffxp]

Took ~26min to apply (seems long, maybe due to this being a regional cluster?)

spiffxp@spiffxp-macbookpro:prow-build-trusted (main)$ terraform apply
# ... 
module.prow_build_cluster.google_container_cluster.prod_cluster[0]: Modifications complete after 25m49s [id=projects/k8s-infra-prow-build-trusted/locations/us-central1/clusters/prow-build-trusted]

Meanwhile, from https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache#enabling

If you are using maintenance windows, your nodes will not be recreated until a maintenance window occurs. If you prefer not to wait, you can manually "upgrade" the node pool to the same version it is already using, by setting the --cluster-version flag to the same GKE version the control plane is already running. You must use the gcloud command if you use this workaround. See the caveats section of the maintenance window documentation for more information.

We are using maintenance windows

$ gcloud container clusters describe prow-build-trusted --project=k8s-infra-prow-build-trusted --region=us-central1 --format="value(maintenancePolicy)"
resourceVersion=44fb9d63;window={'dailyMaintenanceWindow': {'duration': 'PT4H0M0S', 'startTime': '11:00'}}

I'm probably going to force an upgrade to avoid waiting

[spiffxp]

Per https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache#verifying_that_is_enabled, check that it's enabled:

spiffxp@cloudshell:~ (k8s-infra-prow-build-trusted)$ k --context=gke_k8s-infra-prow-build-trusted_us-central1_prow-build-trusted get pods -n kube-system -o wide | grep node-local-dns
spiffxp@cloudshell:~ (k8s-infra-prow-build-trusted)$

So, not running. Confirming that it's been configured for the cluster:

addonsConfig:
  dnsCacheConfig:
    enabled: true
  horizontalPodAutoscaling: {}
  httpLoadBalancing: {}
  kubernetesDashboard:
    disabled: true
  networkPolicyConfig: {}

So yeah, time to force nodes to upgrade

[spiffxp]

Using gcloud to do this per https://cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions#caveats_for_maintenance_windows

$ gcloud container node-pools list --project=k8s-infra-prow-build-trusted --cluster=prow-build-trusted --region=us-central1
NAME                                      MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
trusted-pool1-20200430235251092500000001  n1-standard-8  200           1.16.15-gke.6000
$ gcloud container clusters upgrade \
  --project=k8s-infra-prow-build-trusted \
  --region=us-central1 \
  prow-build-trusted \
  --node-pool=trusted-pool1-20200430235251092500000001 \
  --cluster-version=1.16.15-gke.6000
Upgrading prow-build-trusted... Updating trusted-pool1-20200430235251092500000001, done with 0 out of 3 nodes (0.0%): 1 being processed...

Just based on how long this has taken thus far, I suspect k8s-infra-prow-build won't get this completely until end of day (it has many more nodes to upgrade). May batch up with an upgrade to v1.17, we should have already gotten there as part of stable release channel?

[chaodaiG]

Just based on how long this has taken thus far, I suspect k8s-infra-prow-build won't get this completely until end of day (it has many more nodes to upgrade). May batch up with an upgrade to v1.17, we should have already gotten there as part of stable release channel?

That sounds about right to me. And yes I have seen v1.17 in stable channel, so batching up makes sense to me

[spiffxp]

Deployed to k8s-infra-prow-build-trusted. Need to see some job traffic to verify things are still working as expected

$ k --context=gke_k8s-infra-prow-build-trusted_us-central1_prow-build-trusted get pods -n kube-system -o wide | grep node-local-dns
node-local-dns-6z877                                             1/1     Running   0          43m   10.128.0.33   gke-prow-build-trust-trusted-pool1-20-caef7902-a7r0   <none>           <none>
node-local-dns-7dcjp                                             1/1     Running   0          38m   10.128.0.34   gke-prow-build-trust-trusted-pool1-20-5e50b276-1wj9   <none>           <none>
node-local-dns-gk2lj                                             1/1     Running   0          46m   10.128.0.32   gke-prow-build-trust-trusted-pool1-20-fd38017e-4yal   <none>           <none>

Timestamps if anyone needs to correlate disruptive behavior during this time.

$ gcloud container operations list --project=k8s-infra-prow-build-trusted --filter="startTime>='2021-02-19'" --format="table(startTime,endTime,operationType,name,status)"
START_TIME                      END_TIME                        TYPE            NAME                              STATUS
2021-02-19T14:28:32.358854864Z  2021-02-19T14:28:32.602803386Z  UPDATE_CLUSTER  operation-1613744912358-e2104c61  DONE
2021-02-19T14:28:42.728906016Z  2021-02-19T14:54:08.93231119Z   UPDATE_CLUSTER  operation-1613744922728-74f4f253  DONE
2021-02-19T15:13:14.422720553Z  2021-02-19T15:25:56.940972811Z  UPGRADE_NODES   operation-1613747594422-88695ac3  DONE

[spiffxp]

being able to filter by cluster on deck would be nice, in the meantime

curl https://prow.k8s.io/prowjobs.js?omit=pod_spec,decoration_config >prowjobs.js
<prowjobs.js jq \
'.items
  | map(
      select(
        .spec.cluster == "k8s-infra-prow-build-trusted" 
        and .status.pendingTime >= "2021-02-19T14"
      )
    ) 
  | sort_by(.status.pendingTime) 
  | map(
      .status | {time: .pendingTime, state, url}
    )'

[
  {
    "time": "2021-02-19T14:16:35Z",
    "state": "failure",
    "url": "https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-csi-driver-smb-push-images/1362768045499486208"
  },
  {
    "time": "2021-02-19T14:26:27Z",
    "state": "success",
    "url": "https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/kops-postsubmit-push-to-staging/1362770527264968704"
  },
  {
    "time": "2021-02-19T14:38:13Z",
    "state": "success",
    "url": "https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-k8sio-gcr-prod-backup/1362773489949347840"
  },
  {
    "time": "2021-02-19T15:08:13Z",
    "state": "success",
    "url": "https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-k8sio-image-promo/1362781036726980608"
  },
  {
    "time": "2021-02-19T15:10:26Z",
    "state": "success",
    "url": "https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-image-promo/1362781597782249472"
  },
  {
    "time": "2021-02-19T15:44:26Z",
    "state": "success",
    "url": "https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-csi-driver-host-path-push-images/1362790152912506880"
  },
  {
    "time": "2021-02-19T15:54:28Z",
    "state": "success",
    "url": "https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-cluster-api-push-images/1362792678915313664"
  }
]

[spiffxp]

I'm satisfied, will open followup PR to apply to k8s-infra-prow-build

[spiffxp]

Opened #1686

[BenTheElder]

Thank you both!