Kops - Don't use computeMetadata for retrieving public IP

[rifelpet]

Since workload identity was enabled on the e2e clusters, presubmit jobs have been failing because the computeMetadata endpoints are no longer available.

We were relying on it to get our public ip, falling back to ifconfig.co.
The fallback logic wasnt working though: the response body isn't empty, its Not Found.
I'm moving the logic from kops-e2e-runner.sh, which our periodic jobs have already been migrated away from, to kubetest and only using ifconfig.co. This way both our presubmit and periodic e2e jobs use the same logic for restricting access to the clusters. And it gets us one step closer to moving off of kops-e2e-runner.sh entirely :)

I'm not a big fan of relying on this third party service for such a simple use case, so if anyone has better ideas on how to get our public IP any feedback would be appreciated.

[rifelpet]

/assign @BenTheElder

[rifelpet]

/cc @fejta

[BenTheElder]

/approve
/lgtm
/hold

[BenTheElder]

Perhaps log something about this?

[BenTheElder]

Er about using this value to set admin access (commented on the wrong line)

[rifelpet]

done 👍

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [BenTheElder]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

/lgtm
/unhold

[fejta]

Alternatively add a service account to this job

[rifelpet]

It does have a service account preset defined here and this log seems to indicate it is being used. Is that not sufficient?

[fejta]

What does adding a preset-service-account: "true" label to the pod do here?

By service account I mean the pod needs a kubernetes service account. That will look something like this:

test-infra/config/jobs/kubernetes/test-infra/test-infra-trusted.yaml

Line 88 in ac36d0b

serviceAccountName: deployer

We declare the service accounts that exist for jobs in these two locations, depending on the cluster they run in:

test-infra/prow/cluster/build_serviceaccounts.yaml

Lines 7 to 9 in ac36d0b

	# TODO(fejta): better define these rules and enforce them in presubmit.
	iam.gke.io/gcp-service-account: k8s-infra-gcr-promoter@k8s-gcr-audit-test-prod.iam.gserviceaccount.com
	name: k8s-gcr-audit-test-prod

test-infra/prow/cluster/trusted_serviceaccounts.yaml

Lines 29 to 31 in ac36d0b

	annotations:
	iam.gke.io/gcp-service-account: [email protected]
	name: deployer

The annotation controls which GCP service account metadata server (and thus the k8s service account) authenticates as

If the pod does not define a service account (default behavior) identity for its containers then GKE does not provide those container with a metadata server.

[fejta]

The log is showing that the job has explicit credentials in a secret file passed to the job, which it uses for authentication (the secret contents of that file authenticates the user, the metadata server is not involved here).

[rifelpet]

ah ok, so this service account is separate from the serviceAccountName of the pod. That makes sense.

So we would be expected to create a new ServiceAccount resource in build_serviceaccounts.yaml correct? Which gcp-service-account would I want to use? Clearly this doesn't actually require GCP permissions, just access to the metadata server itself.

[fejta]

Use [email protected], which is what the secrets/service-account file authenticates as. Maybe add a kops service account to that build_serviceaccounts.yaml file?

If you want to do this (which would be fantastic) then send me the PR. (Once it goes it you should be able to remove the preset-service-account: true label from your pods and not have to deal with the secret.json file any more too)