Support User Namespaces in pods

[derekwaynecarr]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add support for user namespaces in pods
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/README.md
• Discussion Link: This was discussed in several sig-node meetings, most notable Nov 2 2021 and in this PR, that feedback was incorporated already by this KEP is taking: keps/127: Support User Namespaces #2101.
• Primary contact (assignee): @rata @giuseppe
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.25
  • Beta release target (x.y): 1.30
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • v1.24 127: Add KEP for user namespaces support #3065
    • v1.25 KEP-127: Mark as implementable (target phase I for 1.25) #3275
    • v1.27 KEP-127: Support userns in stateless pods with idmap mounts #3811
    • v1.28 [KEP-127] Add Pod Security Standards to User Namespaces KEP #4044
    • v1.28 KEP-127: add support for stateful pods #4084
    • v1.28 KEP-127: Fix user namespaces feature gate name #4107
    • v1.28 KEP-127: Update feature gate name and update last-milestone for 1.29 #4147
  • Code (k/k) update PR(s):
    • v1.27 KEP-127: user namespace support for stateless pods kubernetes#116377
    • v1.28 apis: drop check for volumes with user namespaces kubernetes#118691
    • v1.29 KEP-127: Update PSS based on feature gate kubernetes#118760
  • Docs (k/website) update PR(s):
    • v1.27 Document user namespace changes in v1.27 (KEP-127) website#39860
    • v1.28 Doc update for userns in 1.28 website#41908 (v1.28)
    • v1.29 [KEP-127] Add documentation about user namespaces and PSS website#43803 (v1.29)
• Beta
  • KEP (k/enhancements) update PR(s): KEP-127: graduate to Beta for 1.30 #4439
  • Code (k/k) update PR(s):
    • KEP-127: require userns support from the CRI runtime before using it kubernetes#123216
    • KEP-127: kubelet: honor kubelet user mappings kubernetes#123593
    • KEP-127: Add UserNamespacesPodSecurityStandards e2e test kubernetes#121606
  • Docs (k/website) update(s):
    • Feature blog 1.30: user namespaces website#45354
    • User namespaces doc changes for 1.30 website#45178

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[derekwaynecarr]

This work is being done by @pweil- and is reviewed by @derekwaynecarr, it is sponsored by @kubernetes/sig-node

[mdshuai]

@derekwaynecarr Could you help create a user story card for this feature?

[idvoretskyi]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

[pweil-]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

Yes, this feature is experimental only so it would be considered alpha.

[idvoretskyi]

@derekwaynecarr @pweil- can you confirm that this item targets beta in 1.6?

[adelton]

@derekwaynecarr, the proposal kubernetes/kubernetes#34569 was closed by bot due to inactivity.

@pweil-, in kubernetes/kubernetes#34569 (comment) you've proposed the approach pweil-/kubernetes@16f29eb which changes the group of /var/lib/kubelet/pods to the remapped root group. Do I understand it correctly that this is currently not tracked in any pull request?

[adelton]

@pweil-, I also wonder if similar to docker's /var/lib/docker/<uid>.<gid> approach when --userns-remap is used, it might make sense to use /var/lib/kubelet/pods-<uid>.<gid> and just chown/chgroup everything in those subdirectories to the remapped <uid>.<gid>. Why did you opt for just the chgrp and not the full chown?

[pweil-]

@adelton in the end, I think having this be transparent to Kubernetes is the right approach. Whether that be something like shiftfs or implementation in the CRI (moby/moby#28593). You are correct that my existing proposal is not currently tracked in an open PR anymore.

The reasoning behind using the chgrp was to follow our fsgroup strategy where we just ensure group access instead of uid access.

[adelton]

Thanks @pweil-.

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

I have now filed kubernetes/kubernetes#55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

[pweil-]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question 😄

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

Yes

I have now filed kubernetes/kubernetes#55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

👍 subscribed

[adelton]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question

Ideally, the pod would specify how many distinct uids/gids it would require / list of uids it wants to see inside of the containers, and docker or different container runtime would setup the user namespace accordingly. But unless docker also changes ownership of the volumes mounted to the containers, Kubernetes will have to do that as part of the setup.

[adelton]

@pwel-, what is the best way to get some review and comments on kubernetes/kubernetes#55707, to get it closer to mergeable state?

[0xmichalis]

@pweil- ^

[pweil-]

@adelton I would try to engage the sig-node folks either at their Tuesday meeting or on slack: https://github.com/kubernetes/community/tree/master/sig-node

[adelton]

@derekwaynecarr, could you please bring kubernetes/kubernetes#55707 to sig-node's radar?

[idvoretskyi]

@pweil- @derekwaynecarr any progress on this feature is expected?

[salehsedghpour]

Hello @mrunalp, @derekwaynecarr and @vikaschoudhary16 , 1.30 Enhancements team here! Is this enhancement targeting 1.30? If it is, can you follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board? Thanks!

[rata]

@salehsedghpour yes, it is targeting 1.30. What step from there is missing?

[salehsedghpour]

@rata, lead-opted-in label is missing.

[rata]

@salehsedghpour ohh, I missed that 🙈 ! @SergeyKanzhelev @mrunalp can you please add the lead-opted-in label?

[mrunalp]

/label lead-opted-in

[tjons]

Hello @giuseppe 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on Friday, February 9th, 2024 at 02:00 UTC.

This enhancement is targeting for stage beta for 1.30 (correct me, if otherwise)

Here’s where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.30. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria.
• KEP has submitted a production readiness review request for approval and has a reviewer assigned.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[tjons]

With all the requirements fulfilled this enhancement is now marked as tracked for the upcoming enhancements freeze 🚀

[Checksumz]

Hi @derekwaynecarr ,

👋 from the v1.30 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement!

We encourage blogs for features including, but not limited to: breaking changes, features and changes important to our users, and features that have been in progress for a long time and are graduating.

To opt in, you need to open a Feature Blog placeholder PR against the website repository.
The placeholder PR deadline is 27th February, 2024.
Here's the 1.30 Release Calendar

[sftim]

Blog team editor here: please do consider opting in. Say hi in Slack (#sig-docs-blog) if you have any questions. This is beta a big thing to announce.

[drewhagen]

Hello @derekwaynecarr 👋, 1.30 Docs Lead here.

Does this enhancement work planned for 1.30 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.30 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday February 22nd 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[rata]

@Checksumz thanks! I'm interested in writing a blog post for the 1.30 release. Let me confirm during next week if I'll do it. I want to check some stuff with other project releases.

@drewhagen yes, there are doc changes on this release. I've just opened a placeholder PR: kubernetes/website#45178. Thanks for the remainder!

[Checksumz]

@Checksumz thanks! I'm interested in writing a blog post for the 1.30 release. Let me confirm during next week if I'll do it. I want to check some stuff with other project releases.

@drewhagen yes, there are doc changes on this release. I've just opened a placeholder PR: kubernetes/website#45178. Thanks for the remainder!

thanks for your response @rata

[tjons]

Hey again @derekwaynecarr 👋 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 6th March 2024 .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PR is open and needs to be merged before code freeze:

• KEP-127: require userns support from the CRI runtime before using it kubernetes#123216

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

[Checksumz]

@Checksumz thanks! I'm interested in writing a blog post for the 1.30 release. Let me confirm during next week if I'll do it. I want to check some stuff with other project releases.

@drewhagen yes, there are doc changes on this release. I've just opened a placeholder PR: kubernetes/website#45178. Thanks for the remainder!

@rata Thank you for your interest in the 1.30 Feature Blog please can you open a placeholder PR by February 26th ?

[rata]

@Checksumz Here it is (against dev-1.30): kubernetes/website#45354

Thanks!

[tjons]

Hello @derekwaynecarr 👋, Enhancements team here.

With all the implementation(code related) PRs merged as per the issue description:

• KEP-127: require userns support from the CRI runtime before using it kubernetes#123216

This enhancement is now marked as tracked for code freeze for the 1.30 Code Freeze!

[sreeram-venkitesh]

Hello 👋, 1.31 Enhancements Lead here.

If you wish to progress this enhancement in v1.31, please have the SIG lead opt-in your enhancement by adding the lead-opted-in label and set the milestone to v1.31 before the Production Readiness Review Freeze.

/remove-label lead-opted-in

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[remram44]

/remove-lifecycle stale

[kannon92]

What are the plans for 1.32?

[rata]

No plans for 1.32 so far, just trying to get final releases of runc and so to use this. We will be back afterwards :)