[Failing Test] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]

[amwat]

Which jobs are failing:

ci-kubernetes-gce-conformance-latest-kubetest2

Which test(s) are failing:

Kubernetes e2e suite.[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]

Since when has it been failing:

Fairly new job, but it's the only Conformance test which is failing consistently.

Testgrid link:

https://testgrid.k8s.io/conformance-all#Conformance%20-%20GCE%20-%20master%20-%20kubetest2

Reason for failure:

e.g. https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-kubetest2/1365061586389045248

Feb 25 23:28:50.753: INFO: Pod logs: 
2021/02/25 23:28:19 OK: Got token
2021/02/25 23:28:19 OK: got issuer https://kubernetes.io/kubetest2
2021/02/25 23:28:19 Full, not-validated claims: 
openidmetadata.claims{Claims:jwt.Claims{Issuer:"https://kubernetes.io/kubetest2", Subject:"system:serviceaccount:svcaccounts-5085:default", Audience:jwt.Audience{"oidc-discovery-test"}, Expiry:1614296299, NotBefore:1614295699, IssuedAt:1614295699, ID:""}, Kubernetes:openidmetadata.kubeClaims{Namespace:"svcaccounts-5085", ServiceAccount:openidmetadata.kubeName{Name:"default", UID:"de16732d-7e39-46b8-8d7d-44bde84f9a91"}}}
2021/02/25 23:28:19 Get "https://kubernetes.io/kubetest2/.well-known/openid-configuration": x509: certificate signed by unknown authority

Anything else we need to know:

This is a new job as part of migration to kubetest2 effort: kubernetes/enhancements#2464

Comparing it to: https://testgrid.k8s.io/conformance-all#Conformance%20-%20GCE%20-%20master

e.g

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest/1365104623890731008

I0226 02:22:17.970] Feb 26 02:22:17.970: INFO: Pod logs: 
I0226 02:22:17.970] 2021/02/26 02:21:46 OK: Got token
I0226 02:22:17.971] 2021/02/26 02:21:46 OK: got issuer https://kubernetes.default.svc.cluster.local
I0226 02:22:17.971] 2021/02/26 02:21:46 Full, not-validated claims: 
I0226 02:22:17.971] openidmetadata.claims{Claims:jwt.Claims{Issuer:"https://kubernetes.default.svc.cluster.local", Subject:"system:serviceaccount:svcaccounts-7975:default", Audience:jwt.Audience{"oidc-discovery-test"}, Expiry:1614306706, NotBefore:1614306106, IssuedAt:1614306106, ID:""}, Kubernetes:openidmetadata.kubeClaims{Namespace:"svcaccounts-7975", ServiceAccount:openidmetadata.kubeName{Name:"default", UID:"eec5ddc6-9991-411c-ab25-22e83658511e"}}}
I0226 02:22:17.972] 2021/02/26 02:21:46 OK: Constructed OIDC provider for issuer https://kubernetes.default.svc.cluster.local
I0226 02:22:17.972] 2021/02/26 02:21:46 OK: Validated signature on JWT
I0226 02:22:17.972] 2021/02/26 02:21:46 OK: Got valid claims from token!
I0226 02:22:17.972] 2021/02/26 02:21:46 Full, validated claims: 
I0226 02:22:17.973] &openidmetadata.claims{Claims:jwt.Claims{Issuer:"https://kubernetes.default.svc.cluster.local", Subject:"system:serviceaccount:svcaccounts-7975:default", Audience:jwt.Audience{"oidc-discovery-test"}, Expiry:1614306706, NotBefore:1614306106, IssuedAt:1614306106, ID:""}, Kubernetes:openidmetadata.kubeClaims{Namespace:"svcaccounts-7975", ServiceAccount:openidmetadata.kubeName{Name:"default", UID:"eec5ddc6-9991-411c-ab25-22e83658511e"}}}
I0226 02:22:17.973] 
I0226 02:22:17.973] Feb 26 02:22:17.970: INFO: completed pod

The issuer seems to differ https://kubernetes.io/kubetest2 vs https://kubernetes.default.svc.cluster.local
would be good to know why/where it's coming from, especially to know if there's a discrepancy/inherent assumptions in how the cluster needs to be created.

This doesn't seem to be related to another issue regarding the same test: #99470

/cc @BenTheElder @spiffxp

@kubernetes/sig-auth-bugs

[BenTheElder]

The JWT issuer comes from the cluster configuration. I suspect kube-up is not invoked quite the same between the two. --service-account-issuer= on the apiserver

[amwat]

👍 yup, I think that's the diff.

I0225 22:32:58.776940 9 flags.go:59] FLAG: --service-account-issuer="https://kubernetes.io/kubetest2"

in the kube-apiserver.log

https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-kubetest2/1365061586389045248/artifacts/3eb373ce-77b6-11eb-a92b-f2148a2022dd/cluster-logs/kubetest2-master/

as opposed to

I0226 01:06:51.825524 9 flags.go:59] FLAG: --service-account-issuer="https://kubernetes.default.svc.cluster.local"

https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest/1365104623890731008/artifacts/bootstrap-e2e-master/

which seems to be coming from:

kubernetes/cluster/gce/gci/configure-kubeapiserver.sh

Line 145 in 1ddc4af

params+=" --service-account-issuer=${SERVICEACCOUNT_ISSUER}"

which differs in the config-default.sh (used by kubetest2)

kubernetes/cluster/gce/config-default.sh

Line 514 in 1ddc4af

export SERVICEACCOUNT_ISSUER="https://kubernetes.io/${CLUSTER_NAME}"

vs config-test.sh (used by kubetest)

kubernetes/cluster/gce/config-test.sh

Line 566 in 1ddc4af

export SERVICEACCOUNT_ISSUER='https://kubernetes.default.svc.cluster.local'

I can try to add a fix for this in kubetest2 but would still like confirmation on whether this difference between the default vs test configuration is WAI?

[BenTheElder]

I'm not sure if we require support for alternative issuers, the other issuer is the "usual" one in our jobs for in-cluster issuer.
cc @mtaufen

[amwat]

ref: #88048

[liggitt]

cc @mtaufen

the test config is more correct... it's a resolveable host that serves discovery data for the issuer

[spiffxp]

/milestone v1.21