Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788

tsde · 2021-12-27T16:48:05Z

RKE version:

1.3.3 (using terraform RKE provider v1.3.0)

Docker version: (docker version,docker info preferred)

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 17
  Running: 9
  Paused: 0
  Stopped: 8
 Images: 13
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: cde01e96ed658bc5050abe1bb601b4b4510ba7a2
 runc version: e4bccdbd64361ac5ea8ba90bb8845add78f957a6
 init version: 
 Security Options:
  seccomp
   Profile: default
  selinux
  cgroupns
 Kernel Version: 5.10.84-flatcar
 Operating System: Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.807GiB
 Name: worker-01
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.0
VERSION_ID=3033.2.0
BUILD_ID=2021-12-10-1820
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"

5.10.84-flatcar

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Master/Worker nodes provisionned by terraform using the RKE provider v1.3.0. Nodes are vSphere virtual machines based on the Flatcar OVA.

cluster.yml file:
As I'm using the terraform provider, here's the tf rke_cluster declaration

resource "rke_cluster" "main" {
  kubernetes_version = "v1.22.4-rancher1-1"
  cluster_name       = "test-cluster"
  authentication {
    strategy = "x509"
    sans     = "<...redacted...>"
  }
  dynamic "nodes" {
    for_each = flatten([local.rke_cluster_master_nodes, local.rke_cluster_worker_nodes])
    content {
      address           = nodes.value["address"]
      ssh_key           = nodes.value["id_rsa"]
      labels            = nodes.value["labels"]
      role              = nodes.value["roles"]
      hostname_override = nodes.value["name"]
      user              = nodes.value["user"]
    }
  }
  dns {
    provider = "coredns"
  }
  ingress {
    provider     = "none"
  }
  network {
    plugin  = "calico"
    options = {
        "calico_cloud_provider" : "none",
        "calico_flex_volume_plugin_dir" : "/var/lib/kubelet/volumeplugins"
    }
  }
  services {
    kube_api {
      audit_log {
        enabled = true
      }
      secrets_encryption_config {
        enabled = true
      }
    }
  }
  upgrade_strategy {
    drain                        = false
    max_unavailable_worker       = 1
    max_unavailable_controlplane = 1
  }
}

Steps to Reproduce:

Try to update a kubernetes cluster from 1.21 (or possibly earlier versions) to 1.22 when using Flatcar OS 3033.2.0. I imagine that a fresh 1.22 installation would lead to the same result.

Results:

The following error occurs:

Failed running cluster err:[[selinux] Host [10.130.0.241] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]

As shown in docker info above, SELinux is enabled on dockerd, triggering this specific step from RKE. Starting from 1.22, a dedicated custom SELinux policy must be installed on SELinux-enabled nodes. As I'm using Flatcar Linux, it's not possible to deploy this RPM as-is.

I'm quite a newbie when it comes to SELinux and I don't see how I can easily work around this as disabling SELinux on the docker daemon is not an option for me. Is there any plan on RKE side to better integrate this with Flatcar Linux ? I may be missing a simple way to circumvent this so don't hesitate to tell me ^^

Thanks

The text was updated successfully, but these errors were encountered:

xeor · 2022-01-10T22:04:48Z

Did you find a solution to this? I think rke is installing this package automatic on rpm distroes, but flatcar is not one of those.
How is this done on other non-rpm distroes?

Should this be lifted to https://github.com/rancher/rke2-selinux, and a request for adding flatcar? Anyone know how this works together?

tsde · 2022-01-11T15:04:19Z

@xeor i didn't find a solution yet. I naively tried to register the SE module extracted from the rpm directly in a flatcar instance and, obviously, it failed as /usr is read-only on Flatcar.

I'm also considering opening an issue on Flatcar side and see what's their opinion about this and what would be the best approach. Maybe opening a request for a new package to be included in Flatcar, but don't know if it meets Flatcar requirements for new packages ?

In the meantime, it would be interesting to have a feedback from the rancher/rke team about this ^^

And yes, I imagine the same situation occurs with rke2-linux rpm package.

stale · 2022-03-13T19:25:54Z

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

bitfisher · 2022-03-31T08:42:01Z

Any progress so far?

tsde · 2022-04-04T17:17:43Z

@bitfisher I opened an issue on Flatcar side too (see flatcar/Flatcar#598). The current workaround is to have selinux disabled for the docker service. This is of course not ideal but work seems currently on-going on Flatcar side to smooth out SELinux related stuff flatcar/Flatcar#673

mohsenmottaghi · 2022-05-24T18:31:51Z

We have the same issue in our production cluster, and we are stuck in v1.21.X. but I tried to disable SELinux in a test cluster by adding a systemd dropin and it works without issue.

systemd:
  units:
    - name: "docker.service"
      enabled: true
      dropins:
        - name: "01-selinux.conf"
          contents: |
            [Service]
            Environment=DOCKER_SELINUX=--selinux-enabled=false

But I don't want to disable SELinux in my production clusters.

bitfisher · 2022-07-13T08:18:14Z

Any updates?
Disabling SELinux in production clusters isn't really an option!

bitfisher · 2022-08-05T10:51:24Z

Is there any other option than disabling SELinux?

github-actions · 2022-10-05T03:22:57Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

bitfisher · 2022-10-06T07:26:55Z

Unstale please

bitfisher · 2022-11-03T14:14:06Z

Any Updates?

Is there any other option than disabling SELinux?

github-actions · 2023-01-03T02:26:55Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

bitfisher · 2023-01-03T08:12:44Z

This is still an issue, so unstale please!

A response from anyone at Rancher would be highly appreciated!

Anyone managed updating k8s to >= 1.22 without disabling SELinux?

Any guiadance from Rancher regarding this issue?

github-actions · 2023-03-05T02:51:17Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

bitfisher · 2023-03-06T06:55:18Z

Unfortunately this is still an issue, so unstale please!

A response from anyone at Rancher would be highly appreciated!

Anyone managed updating k8s to >= 1.22 without disabling SELinux?

Any guidance from Rancher regarding this issue?

mikekuzak · 2023-03-06T20:10:39Z

The thing is that Rancher/SUSE will pull out their support matrix and say Flatcar is not supported officially .. and they don't provide so far any support for any other immutable container OS. (sic!)

bitfisher · 2023-03-07T08:02:34Z

That's bad news :(

I haven't seen any comment from Rancher/SUSE that Flatcar-Support will be dropped. Even in their docs it's still mentioned as supported. (weird)

So we will either have to switch to another OS or drop RKE :(

Any other options you see?

lazyfrosch · 2023-03-07T08:09:13Z

As @mikekuzak is not associated with SUSE or Rancher, this statement is not helping in any way...

Flatcar has never been officially supported, and I guess they are busy with other issues. So Flatcar will never be a priority.

Remember, this is open source, means you could contribute a SELinux solution. Though I would do that for RKE2.

But this is also not an official statement, give them some slack, Rancher is a nice project.

framctr · 2023-03-22T09:09:04Z

Same issue with Fedora CoreOS 37 on OpenStack:

Failed running cluster err:[[selinux] Host [192.168.3.104] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]

Installing rancher-selinux from GitHub does not solve the issue.

The only solution for now is to use a 1.21 kubernetes-rancher version (e.g., v1.21.14-rancher1-1).

mikekuzak · 2023-10-01T10:27:32Z

unstale

github-actions · 2023-12-01T02:21:29Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

tailtwo · 2023-12-04T15:54:35Z

Unstale again please. There still isn't a way to upgrade past 1.21 on Fedora CoreOS.

mikekuzak · 2023-12-04T16:12:54Z

disable SELinux on docker ...

…

On Mon, 4 Dec 2023, 15:54 tailtwo, ***@***.***> wrote: Unstale again please. There still isn't a way to upgrade past 1.21 on CoreOS. — Reply to this email directly, view it on GitHub <#2788 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEOPPA427OUWQ7NAPUSZW63YHXW4PAVCNFSM5K2UKAB2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBTHA4TGOJSHE2A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

github-actions · 2024-02-04T02:07:49Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

mohsenmottaghi · 2024-02-05T19:50:01Z

Unstale again please

github-actions · 2024-05-27T02:17:47Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

mikekuzak · 2024-05-27T06:48:23Z

unstale

…

On Mon, 27 May 2024, 05:18 github-actions[bot], ***@***.***> wrote: This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions. — Reply to this email directly, view it on GitHub <#2788 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEOPPA5MS73P3IVPRD3ZSPLZEKJWFAVCNFSM5K2UKAB2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJTGI2TENZQG43Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

github-actions · 2024-07-27T02:20:55Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

mohsenmottaghi · 2024-07-27T03:41:45Z

Unstale

github-actions · 2024-09-26T02:38:21Z

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

mikekuzak · 2024-10-10T06:51:39Z

unstale

kinarashah · 2024-10-10T15:58:04Z

Are policies being installed by https://github.com/rancher/rancher-selinux/releases? There's a requirement for k8s >= 1.22 clusters to install the rancher-selinux RPM package.

Going through the history, I see the following in the original design discussion:

If this package is not installed, rke up should fail during the pre-check, which should show something like:

[selinux] Host [x] does not recognize SELinux label [rke_container_t]. This is required for Kubernetes version [v1.22.0-rancher0]. Please install rancher-selinux RPM package and try again".

Looks like rke_container_t was added to our rancher-selinux repo as part of the original implementation, PR for reference: https://github.com/rancher/rancher-selinux/pull/6/files

Given that this change was scoped for Kubernetes versions 1.22 and higher, the version alignment looks correct so I am wondering what is missing here that needs to be fixed.

mikekuzak · 2024-10-10T18:18:46Z

Thanks for the explanation, is this also the case with RKE2/K3s? As RKE is slowly being phased out .

…

On Thu, 10 Oct 2024, 16:58 Kinara Shah, ***@***.***> wrote: Are policies being installed by https://github.com/rancher/rancher-selinux/releases? There's a requirement for k8s >= 1.22 clusters to install the rancher-selinux RPM package. Going through the history, I see the following in the original design discussion: If this package is not installed, rke up should fail during the pre-check, which should show something like: [selinux] Host [x] does not recognize SELinux label [rke_container_t]. This is required for Kubernetes version [v1.22.0-rancher0]. Please install rancher-selinux RPM package and try again". Looks like rke_container_t was added to our rancher-selinux repo as part of the original implementation, PR for reference: https://github.com/rancher/rancher-selinux/pull/6/files Given that this change was scoped for Kubernetes versions 1.22 and higher, the version alignment looks correct so I am wondering what is missing here that needs to be fixed. — Reply to this email directly, view it on GitHub <#2788 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEOPPA3OQFG3EQEILJMU6R3Z22P2FAVCNFSM5K2UKAB2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TENBQGU2TAMRTHE3Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

kinarashah · 2024-10-10T19:12:41Z

@mikekuzak For installing with RKE2, you can check https://github.com/rancher/rke2-selinux and https://docs.rke2.io/security/selinux and open an issue at https://github.com/rancher/rke2/issues/ if you run into problems.

tsde mentioned this issue Jan 18, 2022

Can't install Kubernetes >=1.22 with RKE due to missing SELinux custom policies flatcar/Flatcar#598

Open

stale bot added the status/stale label Mar 13, 2022

stale bot removed the status/stale label Mar 31, 2022

github-actions bot added the status/stale label Oct 5, 2022

github-actions bot removed the status/stale label Oct 7, 2022

jiaqiluo added team/area2 [zube]: To Triage labels Oct 7, 2022

github-actions bot added the status/stale label Jan 3, 2023

github-actions bot removed the status/stale label Jan 4, 2023

github-actions bot added the status/stale label Mar 5, 2023

github-actions bot removed the status/stale label Mar 7, 2023

github-actions bot added the status/stale label Oct 1, 2023

github-actions bot removed the status/stale label Oct 2, 2023

github-actions bot added the status/stale label Dec 1, 2023

github-actions bot removed the status/stale label Dec 5, 2023

github-actions bot added the status/stale label Feb 4, 2024

github-actions bot removed the status/stale label Feb 6, 2024

jiaqiluo removed the [zube]: To Triage label Mar 27, 2024

github-actions bot added the status/stale label May 27, 2024

github-actions bot removed the status/stale label May 28, 2024

github-actions bot added the status/stale label Jul 27, 2024

github-actions bot removed the status/stale label Jul 28, 2024

github-actions bot added the status/stale label Sep 26, 2024

github-actions bot closed this as completed Oct 10, 2024

kinarashah reopened this Oct 10, 2024

github-actions bot removed the status/stale label Oct 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788

Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788

tsde commented Dec 27, 2021

xeor commented Jan 10, 2022

tsde commented Jan 11, 2022 •

edited

Loading

stale bot commented Mar 13, 2022

bitfisher commented Mar 31, 2022

tsde commented Apr 4, 2022

mohsenmottaghi commented May 24, 2022

bitfisher commented Jul 13, 2022

bitfisher commented Aug 5, 2022

github-actions bot commented Oct 5, 2022

bitfisher commented Oct 6, 2022

bitfisher commented Nov 3, 2022

github-actions bot commented Jan 3, 2023

bitfisher commented Jan 3, 2023

github-actions bot commented Mar 5, 2023

bitfisher commented Mar 6, 2023

mikekuzak commented Mar 6, 2023

bitfisher commented Mar 7, 2023

lazyfrosch commented Mar 7, 2023

framctr commented Mar 22, 2023 •

edited

Loading

mikekuzak commented Oct 1, 2023

github-actions bot commented Dec 1, 2023

tailtwo commented Dec 4, 2023 •

edited

Loading

mikekuzak commented Dec 4, 2023 via email

github-actions bot commented Feb 4, 2024

mohsenmottaghi commented Feb 5, 2024

github-actions bot commented May 27, 2024

mikekuzak commented May 27, 2024 via email

github-actions bot commented Jul 27, 2024

mohsenmottaghi commented Jul 27, 2024

github-actions bot commented Sep 26, 2024

mikekuzak commented Oct 10, 2024

kinarashah commented Oct 10, 2024

mikekuzak commented Oct 10, 2024 via email

kinarashah commented Oct 10, 2024

Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788

Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788

Comments

tsde commented Dec 27, 2021

xeor commented Jan 10, 2022

tsde commented Jan 11, 2022 • edited Loading

stale bot commented Mar 13, 2022

bitfisher commented Mar 31, 2022

tsde commented Apr 4, 2022

mohsenmottaghi commented May 24, 2022

bitfisher commented Jul 13, 2022

bitfisher commented Aug 5, 2022

github-actions bot commented Oct 5, 2022

bitfisher commented Oct 6, 2022

bitfisher commented Nov 3, 2022

github-actions bot commented Jan 3, 2023

bitfisher commented Jan 3, 2023

github-actions bot commented Mar 5, 2023

bitfisher commented Mar 6, 2023

mikekuzak commented Mar 6, 2023

bitfisher commented Mar 7, 2023

lazyfrosch commented Mar 7, 2023

framctr commented Mar 22, 2023 • edited Loading

mikekuzak commented Oct 1, 2023

github-actions bot commented Dec 1, 2023

tailtwo commented Dec 4, 2023 • edited Loading

mikekuzak commented Dec 4, 2023 via email

github-actions bot commented Feb 4, 2024

mohsenmottaghi commented Feb 5, 2024

github-actions bot commented May 27, 2024

mikekuzak commented May 27, 2024 via email

github-actions bot commented Jul 27, 2024

mohsenmottaghi commented Jul 27, 2024

github-actions bot commented Sep 26, 2024

mikekuzak commented Oct 10, 2024

kinarashah commented Oct 10, 2024

mikekuzak commented Oct 10, 2024 via email

kinarashah commented Oct 10, 2024

tsde commented Jan 11, 2022 •

edited

Loading

framctr commented Mar 22, 2023 •

edited

Loading

tailtwo commented Dec 4, 2023 •

edited

Loading