Feature Reqest: Don't bind the admin.conf user to system:masters

[raesene]

edit by neolit123

tasks for 1.29

• k/e tracking issue and KEP:
  kubeadm: generate a client certificate with "system:masters" in a separate file enhancements#4214
  kubeadm: add KEP 4214 for separate super-user kubeconfig enhancements#4218
  keps/../kubeadm: 4214: revisit test plan and risks/mitigations enhancements#4302
• k/k changes:
  kubeadm: add support for separate super-admin.conf kubeconfig file kubernetes#121305
  kubeadm: poll additional CRB create calls for kubeadm:cluster-admins kubernetes#121609
  kubeadm: do not poll in TestEnsureAdminClusterRoleBindingImpl kubernetes#121648
  test/e2e_kubeadm: add test for the kubeadm:cluster-admins CRB kubernetes#121674
  kubeadm: ensure the kubelet and kube-apiserver wait checks go first kubernetes#121743
  kubeadm: change SystemPrivilegedGroup in apiserve-kubelet-client.crt kubernetes#121837
  kubeadm: support updating certificate organization during 'kubeadm certs renew' kubernetes#121841
• k/website changes:
  kubeadm: introduce documentation changes for super-admin.conf  website#43540
  certificates.md: add note about system:masters in apiserver cert website#43870
• e2e test changes:
  kinder: add workflow for testing super-admin.conf #2949
  kinder: fix missing quotes in super-admin-tasks.yaml #2951
  kinder: fix unwanted sudo in super-admin-tasks.yaml #2952
  kubeadm: add new test job for super-admin.conf test-infra#31161
  kinder: add tests for /etc/kubernetes/pki/apiserver-kubelet-client.crt #2960

tasks for 1.30

• kubeadm: remove code to be removed in 1.30 kubernetes#122786
  • cleanup of TODOs in kubeadm: add support for separate super-admin.conf kubeconfig file kubernetes#121305
  • cleanup of TODOs in kubeadm: support updating certificate organization during 'kubeadm certs renew' kubernetes#121841
• update e2e:
  • kinder: update super-admin workflow for 1.30 #3012
  • kinder: apply yet another fix for super-admin workflow #3015

---

original issue description

At the moment, when a new Kubeadm cluster is created, the default credential that is provided for cluster operators to use has a subject group of system:masters. e.g.

Subject: O = system:masters, CN = kubernetes-admin

This is a special group for which cluster-admin rights are hardcoded into the Kubernetes API server source code (https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/rbac/escalation_check.go#L38).

As a result, this credential cannot be revoked, if a cluster operator loses control of it, or if one of their admins moves team or leaves the organization, the only way to revoke access completely is to rotate the cluster certificate authority keys.

Whilst Kubernetes in general doesn't currently support certificate revocation (kubernetes/kubernetes#18982) it could be possible to reduce the risk posed by a lost admin.conf file, by having a specific clusterrolebinding for the admin.conf user to the cluster-admin clusterrole.

This would allow cluster operators to largely revoke the rights assigned to that credential, by removing the binding. It would still have system:authenticated rights, but these are likely (in most clusters) to be significantly lower than cluster-admin.

[neolit123]

the admin.conf is like a root password...it must not be shared.
given i've seen a few cases where users share the file, i'm going to create a PR to make this very clear in https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/.

it makes perfect sense for a deployer on cluster creation to sign break-glass credentials and that is what the admin.conf is.

As a result, this credential cannot be revoked, if a cluster operator loses control of it, or if one of their admins moves team or leaves the organization, the only way to revoke access completely is to rotate the cluster certificate authority keys.

sharing the admin.conf in such a scenario is not ideal.
kubeadm gives you the kubeadm kubeconfig user command that can be used to create a kubeconfig for any user / group.

/kind documentation

[raesene]

With the lifetime on the certificate being a year, it seems likely that in many cases users who had valid access to the credential when the cluster was created, may no longer need access during that period.

I'd agree that admin.conf should be a break-glass credential, but it may benefit cluster operators to provide a mechanism for that credential to be revoked, given its lifetime.

[neolit123]

With the lifetime on the certificate being a year, it seems likely that in many cases users who had valid access to the credential when the cluster was created, may no longer need access during that period.

you can also rotate the cluster CA, which will invalidate the system:master credentials.
https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/

I'd agree that admin.conf should be a break-glass credential, but it may benefit cluster operators to provide a mechanism for that credential to be revoked, given its lifetime.

that is a kubernetes problem, since revocation is not supported in core.
kubernetes/kubernetes#18982

something that we can improve with respect to kubeadm admin credentials is to sign two files:

• admin.conf: doesn't use system:masters but still has super powers
• superadmin.conf: uses system:masters and is the break-glass credential
  but this change needs a KEP / proposal and might not land anytime soon. i don't think i will have time for it in 1.22.

[neolit123]

/assign
assigning myself to make it very clear what admin.conf is in https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

[neolit123]

@raesene please add LGTM to this PR:
kubernetes/website#27141

[raesene]

done :)

[mlbiam]

Would an off-line CA that could issue break-glass admin certs be more secure? you still have a "root" credential in the off-line CA that you would need to protect, but could issue shorter lived/time boxed admin.conf certs that could be controlled through other means.

[arianvp]

Would perhaps be useful to also put the suggestion of kubeadm kubeconfig user command in the output of kubeadm init as well so people really do not download it

[neolit123]

Would an off-line CA that could issue break-glass admin certs be more secure? you still have a "root" credential in the off-line CA that you would need to protect, but could issue shorter lived/time boxed admin.conf certs that could be controlled through other means.

by off-line do you mean a CA that the cluster components and kubeadm are not aware of, such as a root CA of the user org?
if yes, i don't think signing the system:masters credentials with it would work.
it must be signed with the intermediate CA that is passed to the cluster and if the credential leaks the same intermediate CA must be rotated (since, no revocation in k8s).

Would perhaps be useful to also put the suggestion of kubeadm kubeconfig user command in the output of kubeadm init as well so people really do not download it

there is a warning for that on this page:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Warning: Kubeadm signs the certificate in the admin.conf to have Subject: O = system:masters, CN = kubernetes-admin. system:masters is a break-glass, super user group that bypasses the authorization layer (e.g. RBAC). Do not share the admin.conf file with anyone and instead grant users custom permissions by generating them a kubeconfig file using the kubeadm kubeconfig user command.

but a message for that in the output of "init" seems fine for 1.23. PRs welcome.
we are currently in code freeze for 1.22.

[arianvp]

Excellent. I'll draft a PR.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[neolit123]

/lifecycle frozen
/unassign

[pacoxu]

New TestGrid alerts at sig-cluster-lifecycle-kubeadm#kubeadm-kinder-super-admin-latest since commit/dbd3f3564.

--------------------------------------------------------------------------------
task-09-post-upgrade-primary-cp
Run commands after kubeadm upgrade is called on the primary CP node

command : /bin/bash -c set -x
CMD="docker exec kinder-super-admin-control-plane-1"

# Both admin.conf and super-admin.conf must exist
${CMD} test -f /etc/kubernetes/admin.conf || exit 1
${CMD} test -f /etc/kubernetes/super-admin.conf || exit 1

# Check certificate subject for .conf files
${CMD} grep 'client-certificate-data' /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = kubeadm:cluster-admins, CN = kubernetes-admin" || exit 1
${CMD} grep 'client-certificate-data' /etc/kubernetes/super-admin.conf | awk '{print $2}' | base64 -d | openssl x509 -subject -noout | grep "subject=O = system:masters, CN = kubernetes-super-admin" || exit 1

# Check certificate subject for apiserver-kubelet-client.crt
${CMD} openssl x509 -subject -noout -in /etc/kubernetes/pki/apiserver-kubelet-client.crt | grep "subject=O = kubeadm:cluster-admins, CN = kube-apiserver-kubelet-client" || exit 1

# Check if the admin.conf user still has RBAC permissions
${CMD} kubectl -n kube-system --kubeconfig /etc/kubernetes/admin.conf get cm kubeadm-config || exit 1

# Ensure exit status of 0
exit 0

timeout : 5m0s
force   : false
--------------------------------------------------------------------------------

+ CMD='docker exec kinder-super-admin-control-plane-1'
+ docker exec kinder-super-admin-control-plane-1 test -f /etc/kubernetes/admin.conf
+ docker exec kinder-super-admin-control-plane-1 test -f /etc/kubernetes/super-admin.conf
+ exit 1

We may update the CI after we removed some logics in kubernetes/kubernetes#122786.
Do I understand it correctly? @HirazawaUi @neolit123

[neolit123]

We may update the CI after we removed some logics in kubernetes/kubernetes#122786.
Do I understand it correctly? @HirazawaUi @neolit123

yes, we need to update the test a little.
i will send a PR.

[neolit123]

We may update the CI after we removed some logics in kubernetes/kubernetes#122786.
Do I understand it correctly? @HirazawaUi @neolit123

yes, we need to update the test a little. i will send a PR.

PR
#3012

[neolit123]

this is done