Kubernetes | Docker: Add support for rootless images

[pat-s]

fix #1077
fix #3552
fix #3164

• Docs

Background

The blocking issue for non-root images in k8s are the volume mount permissions as noted by @6543 in #1077 (comment).

Volumes are by default owned by root:root. This can be changed for the group using securityContext.fsGroup. However, only the mount point of the volume has 775 permissions, i.e. the option for the group to write.

WP creates <mount point>/$CI_WORKSPACE internally by setting workingDir in

woodpecker/pipeline/backend/kubernetes/pod.go

Line 185 in a544836

WorkingDir: step.WorkingDir,

.
This directory gets created with root:<fsgroup> but with 755, i.e. the group can't write there, even with being set correctly.

Solution

Setting workingDir to workspaceBase, i.e. /woodpecker, and creating <workspaceBase>/src/<repo slug> via the container user solves this. The subsequently created dir is then owned by the container user and all operations work.
To achieve this, workspaceBase has been added to the Step struct and set as workingDir during pod creation.

Sidework

The hard-coded $HOME is an additional issue here. Removing it and letting the container define $HOME is the most straightforward solution. 99% of all containers have $HOME defined and I don't see any downside from removing the hard-coded setting, especially as it can't be overwritten within environment: even.

SecurityContext

Most non-root images run with UID=1000 and GID=1000. Hence, I argue that we should set fsGroup: 1000 as the default (coupled with fsGroupChangePolicy: "OnRootMismatch") and document that users need to change that backend option in case their image user has a different value. This is coupled with fsGroupChangePolicy, which I added as a new SecurityContext param, but only if fsGroup is also set.

Setting runAsUser is not helpful as it would conflict with rootful images. Though, because non-root images are making use of the group write permissions, setting fsGroup should be enough.

Acknowledgements

I learned a lot about workspaceBase and workspacePath in #2510, thanks @zc-devs and @dominic-p for the discussion!

Reprex

when:
  event:
    - push

skip_clone: true

steps:
  test:
    image: renovate/renovate
    commands:
      - whoami
      - pwd
      - echo $HOME
      - ls -la /woodpecker
      - echo $CI_WORKSPACE
      - ls -la $CI_WORKSPACE
      - mkdir /woodpecker/test
      - ls -la /woodpecker/test

yields

+ whoami
ubuntu
+ pwd
/woodpecker/src/<redacted>/$CI_REPO
+ echo $HOME
/home/ubuntu
+ ls -la /woodpecker
total 24
drwxrwsr-x. 4 root   ubuntu  4096 Oct  2 08:31 .
drwxr-xr-x. 1 root   root      26 Oct  2 08:31 ..
drwxrws---. 2 root   ubuntu 16384 Oct  2 08:31 lost+found
drwxr-sr-x. 3 ubuntu ubuntu  4096 Oct  2 08:31 src
+ echo $CI_WORKSPACE
/woodpecker/src/<redacted>/$CI_REPO
+ ls -la $CI_WORKSPACE
total 8
drwxr-sr-x. 2 ubuntu ubuntu 4096 Oct  2 08:31 .
drwxr-sr-x. 3 ubuntu ubuntu 4096 Oct  2 08:31 ..
+ mkdir /woodpecker/test
+ ls -la /woodpecker/test
total 8
drwxr-sr-x. 2 ubuntu ubuntu 4096 Oct  2 08:31 .
drwxrwsr-x. 5 root   ubuntu 4096 Oct  2 08:31 ..

Using "clone" plugin

+ whoami
ubuntu
+ pwd
/woodpecker/src/<redacted>/$CI_REPO
+ echo $HOME
/home/ubuntu
+ ls -la /woodpecker
total 24
drwxrwsr-x. 4 root ubuntu  4096 Oct  2 13:42 .
drwxr-xr-x. 1 root root      26 Oct  2 13:42 ..
drwxrws---. 2 root ubuntu 16384 Oct  2 13:42 lost+found
drwxrwsr-x. 3 root ubuntu  4096 Oct  2 13:42 src
+ echo $CI_WORKSPACE
/woodpecker/src/<redacted>/$CI_REPO
+ ls -la $CI_WORKSPACE
total 24
drwxrwsr-x. 4 root ubuntu 4096 Oct  2 13:42 .
drwxrwsr-x. 3 root ubuntu 4096 Oct  2 13:42 ..
drwxrwsr-x. 9 root ubuntu 4096 Oct  2 13:42 .git
drwxrwsr-x. 2 root ubuntu 4096 Oct  2 13:42 .woodpecker
-rw-rw-r--. 1 root ubuntu   60 Oct  2 13:42 Dockerfile
-rw-rw-r--. 1 root ubuntu  854 Oct  2 13:42 build.yaml
+ mkdir /woodpecker/test
+ ls -la /woodpecker/test
total 8
drwxr-sr-x. 2 ubuntu ubuntu 4096 Oct  2 13:42 .
drwxrwsr-x. 5 root   ubuntu 4096 Oct  2 13:42 ..

[6543]

well instead of removing it, we should add a check if it was not set and then set it accordingly

and HOME should not be WORKDIR in normal cases ... (as tools often create/alter (config-)files in HOME)

[pat-s]

well instead of removing it, we should add a check if it was not set and then set it accordingly

What is "accordingly"? I don't see why we would need to do that - why not let the image decide it?

and HOME should not be WORKDIR in normal cases

It isn't anyways? WORKDIR is /woodpecker/src/<repo slug>? (= $CI_WORKSPACE)

[woodpecker-bot]

Deploying preview to https://woodpecker-ci-woodpecker-pr-4151.surge.sh

[6543]

That will work for command steps, not sure what plugins will do ...

[6543]

Also we should extend the git plugin to allow to chown the content ... via setting

[pat-s]

See my updated OP WRT to the "clone" plugin example.

With the "clone" plugin, the perms are 775 on $CI_WORKSPACE. Hence, all subsequent operations should work.

[zc-devs]

Most non-root images run with UID=1000 and GID=1000

Considering the Bitnami's images alone, the statement is too strong (already).

[pat-s]

We won't be able to cover all GIDS, yeah, but I think 1000 is the best possible default? And better than having none.

[zc-devs]

We won't be able to cover all GIDs, but I think 1000 is the best possible default? And better than having none.

⬇️

We won't be able to cover all UIDs, but I think 1000 is the best possible default? And better than having none.

??? :)

I gave you an example of widely used images (IMO, the second ones after official), which will likely be broken by the default settings from this PR. There is a lot other images, that use group other than 1000, I believe. In case of Bitnami, curious person can start from their 9485 issue.

---

Let me share my thoughts.

We have the workspace. So, the base is the volume mount point, some sub-directories can exist there.

One of them is sources. I think, clone plugin is responsible for creating this sub-directory and set appropriate permissions.

The workspace defines the ... working directory shared by all workflow steps.

So, prerequisites are:

1. Source directory needs to be shared by all steps
2. We do not know what user/group step would run under

The third set of permissions is generally referred to as "others". So, clone plugin should set permissions like

chmod -R 777 /woodpecker/src/
# or
chmod -R 666 /woodpecker/src/

Edit 1: ⬆️ I assumed that working directory = sources directory.
Edit 2: OK, sources could not be writable, but then working directory should be something else and "some thing" should set write permissions ⬇️

In regards volume mount point...

The workspace defines the volume shared by all workflow steps

The logic and prerequisites are the same: something should grant w (probably x) permissions to others on volume root. Obviously, fsGroup doesn't work here. This "some thing" should be the first step in a workflow. So, it could be some "volume initialization" step. Or this responsibility could be delegated to clone plugin (temporarily?).

---

I'm sorry for giving non-asked feedback, BTW.

[pat-s]

??? :)

UIDs are not of interest. The mount point needs to have group w to initialize CI_WORKSPACE with user w.

I gave you an example of widely used images (IMO, the second ones after official), which will likely be broken by the default settings from this PR.

Why would they? AFAIU you can't run non-root bitnami images right now in k8s and execute arbitrary steps due to the apparent limitations. Even when adapting securityContext etc. With this PR, you can (for the first time)? Stating that they're broken with this PR infers that they would be working right now?

There is a lot other images, that use group other than 1000, I believe.

🤷️ We can only set one default value and we won't be able to cover all non-root variants. A dynamic setting is also not possible AFAICS. Setting a default here is better than none? Do you have a better suggestion?

So, it could be some "volume initialization" step. Or this responsibility could be delegated to clone plugin (temporarily?).

Why? I don't see that we would need that. And I also would prefer to avoid adding an init container or similar.
Is the current logic fully clear?

1. We ensure group w on the volume mount point (=workingDir)
2. Create the actual workingDir (src/) within the container.
3. Have a functional workingDir with user w and group w

In my tests so far, I couldn't find a use case yet for which it doesn't work or for which it would break something else. Still needs more testing of course, but I am optimistic so far, that this is a good solution.

[codecov]

Codecov Report

Attention: Patch coverage is 55.55556% with 4 lines in your changes missing coverage. Please review.

Project coverage is 26.48%. Comparing base (8b4d440) to head (d394432).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pipeline/backend/kubernetes/pod.go	57.14%	2 Missing and 1 partial ⚠️
pipeline/backend/kubernetes/kubernetes.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4151   +/-   ##
=======================================
  Coverage   26.48%   26.48%           
=======================================
  Files         375      375           
  Lines       27396    27402    +6     
=======================================
+ Hits         7255     7257    +2     
- Misses      19477    19480    +3     
- Partials      664      665    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zc-devs]

Stating that they're broken with this PR infers that they would be working right now?

I do not know and lazy to test :) But point was ⬇️

Setting a default here is better than none? Do you have a better suggestion?

So, you've introduced some options that allow to run any non-root image, right? OK, that is good 👍
But why do you want to set default? Why 1000, why not 1001, 100, 101, 65532?
I think, there should not be the default value, the same as for runAsUser and runAsGroup. Or it should be optional (agent?) setting.

I couldn't find a use case yet for which it doesn't work

1. Plugins, which rely on current behavior, will probably be broken.
   WorkingDir: step.WorkSpaceBase,

workspace:
  base: /go
  path: cache # who creates this directory? it was container runtime previously.

skip_clone: true

steps:
  build:
    image: go-build-plugin
    settings:
      some: value

3. Add support for nonroot OCI images #1077 (comment)

steps:
  build:
    image: alpine-build-plugin
    settings:
      apkbuild-dir: /woodpecker/src/repo/app/ # what owner/permissions was set there by clone plugin?
    backend_options:
      kubernetes:
        securityContext:
          runAsUser: buildozer
          runAsGroup: abuild

Would plugin has access to desired directory?

With the "clone" plugin, the perms are 775 on $CI_WORKSPACE

Seems, yes. But would abuild like the owner of "workdir"? ¯_(ツ)_/¯ There was some reason to fix the owner, but I do not remember now. Maybe it is not related here.
However, there is no write permissions in workspace for users other, than clone step runs under.

[pat-s]

But why do you want to set default? Why 1000, why not 1001, 100, 101, 65532?

In some scenarios implicit defaults exists that are honored by a majority. IMO this applies here.
Using 1000 would mean we cover a large fraction of non-root images by default, without having to set a custom securityContext for every pipeline. This is better than having to do it for all images. Yes, nobody can give a exact number WRT to the share but again, why go with 0 if you can have more without issues?

#4151 (comment), which rely on current behavior, will probably be broken.

Yes, the PR is breaking. Even though I'd say not many users overwrite workspace.

However, there is no write permissions in workspace for users other, than clone step runs.

Do you mean by non-root plugins that use a different non-root user?
This would apply yes, however right now all plugins are rootful AFAIK (something to address).
We could set 777 to workaround that. Given that the pod PVs are only used once during the steps, I don't see why this would be an issue.

[zc-devs]

Do you mean by non-root plugins that use a different non-root user?

Yes, but apparently, it applies to general steps also:

steps:
  test:
    image: alpine
    commands:
    - echo "hello" > touch /woodpecker/src/repo/temp.txt
    backend_options:
      kubernetes:
        securityContext:
          runAsUser: 101 # other than `clone` user
          runAsGroup: 101 # other than `clone` group

We could set 777 to workaround that

So, fsGroup stuff isn't needed then? :)

something should grant w (probably x) permissions to others on volume root. This "some thing" should be the first step in a workflow. This responsibility could be delegated to clone plugin (temporarily?)

"on volume root" && "workspace" (aka source directory) in case of clone plugin.

[pat-s]

So, fsGroup stuff isn't needed then? :)

It is (for skip_clone: true), because the first mkdir is needed with working group w on /woodpecker. We could chain a chmod 775 after this, so that $CI_WORKSPACE has 775 instead of 755.
Fun fact: the value of fsGroup is not important it seems, as the container user will automatically be member of the group referenced by fsGroup (I justed tested this with a nonsense value). The only important part is though: mkdir needs to work to be able to create $CI_WORKSPACE.

Scenario: skip_clone: true:

1. Use non-root image with GID 1000 -> $CI_WORKSPACE will be 755 owned by UID:GID of the first image.
2. For a second step, the existing volume would be mounted. Assuming a different non-root UID/GID now, no permissions on $CI_WORKSPACE would exist. So I guess forcing a chmod 775 right from the first creation of $CI_WORKSPACE makes sense.

-> Just tested this and it seems that using any value for fsGroup is sufficient. In step2, the group owner is adjusted and the image user is automatically part of that group, ensuring group w on $CI_WORKSPACE, recursively.

---

So based on all findings so far, defining fsGroup with an arbitrary value and ensuring workingDir = mount point seems to solve all non-root execution issues?

[pat-s]

I now verified that NOT setting fsGroup won't work when using multiple non-root steps. An arbitrary value for fsGroup (I went with 1000 for the default now) works, even if using different images with different UID/GIDs of non-root users.

[zc-devs]

We could set 777 = something should grant w permissions to others

then you are talking about user/group permissions

775, 755

There are rx permissions for others. I'm lost...

An arbitrary value for fsGroup works, even if using different images with different UID/GIDs of non-root users.

I'm convinced, but have other questions:

1. Does Docker support fsGroup or something similar? OK, PR aims only Kubernetes, but still need to test Docker and Local.
2. Performance. Each time (step), when fsGroup differ from previous, Kubernetes will change group of each file/dir. I doubt it would big hit to performance, but is it necessary at all, if we can just chmod -R o+w /woodpecker in clone step (first aka initialization) once?
3. Does it work on Windows?

[pat-s]

There are rx permissions for others. I'm lost...

Not sure what you mean, but we need w - the diff between 5 and 7 is missing w.

but still need to test Docker and Local.

WRT to the workspaceBase changes, yes. Everything else is k8s-specific.

if we can just chmod -R o+w /woodpecker in clone step (first aka initialization) once?

The solution aims to be generic and cover skip_clone: true explicitly. Ideally, plugin-git would also run rootless and then fsGroup would be a hard requirement.

The PR does a few things which are all needed together to make non-root images work:

• Remove the hard-coded $HOME (most important for the "docker" backend)
• Ensure correct permissions on $CI_WORKSPACE for skip_clone: true and skip_clone: false
• Ensure that all subsequent steps have write access to $CI_WORKSPACE, thanks to fsGroup

Does it work on Windows?

Most changes don't apply to it. The only ones that do, are related to $HOME again. It should I guess, but I can't and won't test it. Not even sure if anyone is using WP on Win? But this is a different discussion.

[zc-devs]

Not sure what you mean, but we need w - the diff between 5 and 7 is missing w

Yes

We could set 777 = something should grant w permissions to others

So, others is the last digit => last digit should be 6 or 7. chmod -R o+w /woodpecker in the first step.
I've explained already why others.

Most changes don't apply to it. The only ones that do, are related to $HOME again.

I was not about HOME in the question 3. Let me cite the docs:

The .spec.os.name field should be set to windows to indicate that the current Pod uses Windows containers.
If you set the .spec.os.name field to windows, you must not set the following fields in the .spec of that Pod:
spec.securityContext.fsGroup
spec.securityContext.fsGroupChangePolicy

and your code

// if unset, set fsGroup to 1000 by default to support non-root images

---

To recap, there are two solutions:

1. Each time changing group ownership of all files/dirs by using Kubernetes specific fsGroup, which doesn't work on Windows (therefore should not be default value)
2. Or granting necessary permissions (rwx, all?) to others once in the first step (roughly speaking), which is universal solution

---

I retain my first question (Does Docker support fsGroup or something similar?). I'm confused
Title: Kubernetes: Add support for rootless images
Lables: docker, kubernetes
Closes: 1077, 3552, 3264

The solution aims to be generic
fsGroup would be a hard requirement
Ensure that all subsequent steps have write access to $CI_WORKSPACE, thanks to fsGroup

What mechanism does set write access in Docker backend? Or in Docker non-root steps work without write access to $CI_WORKSPACE?

[pat-s]

Each time changing group ownership of all files/dirs by using Kubernetes specific fsGroup, which doesn't work on Windows (therefore should not be default value)

Why is Windows in the discussion here? The fsGroup part does not touch Windows-related code, does it?

Or granting necessary permissions (rwx, all?) to others once in the first step (roughly speaking), which is universal solution

I explained this before: this won't work for skip_clone: true, at least the first step needs fsGroup to create $CI_WORKSPACE (or must be a rootful image (e.g. the clone plugin)). The latter shouldn't be needed necessarily and hence, yes, fsGroup is required at least for the first step. And as only setting it there would be complex and codewise and also possibly confuse users, I think we should just set it for every pod.

Using rootless images should ideally just work and not require any additional flags to enable it.

(Does Docker support fsGroup or something similar?).

Docker does not have the same volume restrictions k8s has. Running rootless in docker is much easier and should already work after the removal of hard-coded $HOME. Hence, the backend/docker label.

What mechanism does set write access in Docker backend? Or in Docker non-root steps work without write access to $CI_WORKSPACE?

Yes, docker is different (and simpler but also less secure). But this is a different topic now, I am out of words 🙃️

[zc-devs]

HOME

it can't be overwritten within environment
instead of removing it, we should add a check if it was not set

I would extract Home overriding into separate PR:
If it was set in environment, then override default. Overwise, leave /root default (for now).

[pat-s]

@woodpecker-ci/maintainers any comments/review?

[lafriks]

Have to test it yet but my mine concern is permissions also not only for workspace folder but files used in later steps also, how will it work for mixed root/non-root steps

[pat-s]

Root can always write.
For rootless, fsGroup takes care setting (group) permissions for every step. See previous discussions in this PR.