Initial integration tests for cgroupv2

[kolyshkin]

This refactors integration test helpers a bit, and adds some initial tests for cgroupv2.

Split into logical small(er) commits for the sake of easier reviewing, so better to review it commit by commit.

This is far from complete, but is it mergeable and already helped to find a surprising number of bugs:

• cgroupv2: use "max" for negative values #2272 cgroupv2: use "max" for negative values (and a followup: cgroupv2: only treat -1 as "max" #2300)
• cgroupv2: don't try to set kmem for systemd case #2270 cgroupv2: don't try to set kmem for systemd case
• cgroupv2: fix setting MemorySwap #2288 cgroupv2: fix setting MemorySwap
• cgroupv2: fix fs2 driver initialization #2299 cgroupv2: fix fs2 driver initialization
• cgroupv2 fs driver does not work with default cgroups path #2298 cgroupv2 fs driver does not work with default cgroups path (should be fixed by cgroupv2: fix fs2 driver default path #2305)
• [SELinux] cgroupv2: runc run --systemd-cgroup do not put container in proper cgroup #2310 cgroupv2: runc run --systemd-cgroup do not put container in proper cgroup (should be fixed by [WIP] cgroupv2: fix systemd driver not putting pid into cgroup #2311)
• runc --systemd-cgroup run: need to check that StartTransientUnit succeeds #2313 runc --systemd-cgroup run: need to check that StartTransientUnit succeeds

[AkihiroSuda]

Thanks a lot for working on this, do you plan to work on runc events as well?

[AkihiroSuda]

Can we execute this on Travis?

[kolyshkin]

Can we execute this on Travis?

Probably not, Travis only provides Ubuntu 18.04 and I think it's kernel is too old.

We need to find some other CI platform, say one that provides Fedora 31. Maybe Cirrus-CI could be used, I have yet to research it.

[AkihiroSuda]

We have been already running Fedora 31 KVM on Travis for unittest

runc/.travis.yml

Line 36 in e4363b0

- sudo ssh default sudo podman run --privileged --cgroupns=private -v /lib/modules:/lib/modules:ro test make localunittest

[kolyshkin]

We have been already running Fedora 31 KVM on Travis for unittest

Right! I forgot about vargant/kvm. Will try to enable localintegration then.

[kolyshkin]

Not sure if it's possible to run a podman container with systemd in it.

[AkihiroSuda]

No need to use Podman/Docker for systemd test

runc/.travis.yml

Line 22 in e4363b0

- sudo PATH="$PATH" make localintegration RUNC_USE_SYSTEMD=1

[kolyshkin]

Description updated to refer to the up-to-date relevant PRs and issues.

do you plan to work on runc events as well?

@AkihiroSuda not at the moment, at least not until we fix cgroup v2.

[AkihiroSuda]

I asked it because we have runc events tests 😄, but can be skipped at this moment. (I'm even not sure whether there is real consumer of runc events... None of Moby/Podman/containerd/CRI-O uses runc events.)

https://github.com/opencontainers/runc/blob/master/tests/integration/events.bats

[kolyshkin]

Removed the patch that caused test breakage

[kolyshkin]

OK the tests are running (currently with systemd only) and passing!

Let me see if I can move bats installation into a script.

[AkihiroSuda]

• bats should be installed in Vagrantfile
• integration -> localintegration

[kolyshkin]

bats should be installed in Vagrantfile

I did that initially, but by copy-pasting the installation steps from the Dockerfile. I'd rather have it in a script executed from both places. Is there a way to source the script to Vagrantfile (like I do for Dockerfile)?

[AkihiroSuda]

https://www.vagrantup.com/docs/provisioning/shell.html#external-script

[AkihiroSuda]

I guess, it should be also possible to invoke scripts under /vagrant from the inline script, but haven't confirmed.

[kolyshkin]

Seems like I found a way. Let me test locally and then I'll push it.

[kolyshkin]

OK we need runc binary and bats on the vagrant host. We build runc inside a debian container, it might now work on a fedora host.

Alternatively, we build it on host with whatever golang version Fedora 31 provides (currently 1.13.9 which is good).

[kolyshkin]

So, version 1 is to copy bats and runc from the container to the host and run the test (we can't use make since it wants to rebuild runc):

diff --git a/.travis.yml b/.travis.yml
index 3e12b820..44773938 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,6 +34,9 @@ matrix:
         - sudo ssh default sudo podman build -t test /vagrant
         # Mounting /lib/modules into the container is necessary as CRIU wants to load (via iptables) additional modules
         - sudo ssh default sudo podman run --privileged --cgroupns=private -v /lib/modules:/lib/modules:ro test make localunittest
+        # Copy bats and compiled runc binary from test image to vagrant host
+        - sudo ssh default 'sudo podman run --rm test tar cf - /usr/local/bin/bats /usr/local/libexec/bats* | sudo tar xf - -C / && sudo podman run --rm test tar cf - runc | sudo tar xf - -C /vagrant'
+        - sudo ssh default 'cd /vagrant && sudo RUNC_USE_SYSTEMD=yes bats -t tests/integration'
   allow_failures:
     - go: tip

version 2 (install git, bats, golang, make on the host, compile and run test as usual):

diff --git a/.travis.yml b/.travis.yml
index 44773938..aaca6307 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,6 +34,7 @@ matrix:
         - sudo ssh default sudo podman build -t test /vagrant
         # Mounting /lib/modules into the container is necessary as CRIU wants to load (via iptables) additional modules
         - sudo ssh default sudo podman run --privileged --cgroupns=private -v /lib/modules:/lib/modules:ro test make localunittest
+        - sudo ssh default 'cd /vagrant && sudo RUNC_USE_SYSTEMD=yes make localintegration'
   allow_failures:
     - go: tip
 
diff --git a/Dockerfile b/Dockerfile
index 94be151e..e7d203a7 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,5 +1,4 @@
 ARG GO_VERSION=1.13
-ARG BATS_VERSION=03608115df2071fff4eaaff1605768c275e5f81f
 ARG CRIU_VERSION=v3.13
 
 FROM golang:${GO_VERSION}-buster
@@ -49,13 +48,8 @@ RUN dpkg --add-architecture armel \
 RUN useradd -u1000 -m -d/home/rootless -s/bin/bash rootless
 
 # install bats
-ARG BATS_VERSION
-RUN cd /tmp \
-    && git clone https://github.com/sstephenson/bats.git \
-    && cd bats \
-    && git reset --hard "${BATS_VERSION}" \
-    && ./install.sh /usr/local \
-    && rm -rf /tmp/bats
+COPY script/install-bats.sh /tmp
+RUN /tmp/install-bats.sh && rm /tmp/install-bats.sh
 
 # install criu
 ARG CRIU_VERSION
diff --git a/Vagrantfile b/Vagrantfile
index 165b0780..c8ec0ad0 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -13,6 +13,7 @@ Vagrant.configure("2") do |config|
     v.cpus = 2
   end
   config.vm.provision "shell", inline: <<-SHELL
-    dnf install -y podman
+    dnf install -y podman git make golang-go
   SHELL
+  config.vm.provision "shell", path: "script/install-bats.sh"
diff --git a/script/install-bats.sh b/script/install-bats.sh
new file mode 100644
index 00000000..d63f4f0a
--- /dev/null
+++ b/script/install-bats.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+set -u -e -o pipefail
+: "${BATS_VERSION:="03608115df2071fff4eaaff1605768c275e5f81f"}"
+
+cd /tmp \
+    && git clone https://github.com/sstephenson/bats.git \
+    && cd bats \
+    && git reset --hard "${BATS_VERSION}" \
+    && ./install.sh /usr/local \
+    && rm -rf /tmp/bats

@AkihiroSuda which one do you like better? (I kinda don't like either)

[kolyshkin]

OK, due to some options set by vagrant ssh-config, command ssh default do not create a pty on the remote side, so stdin is a pipe and runc exec with "terminal": true set in config.json tries to treat stdin (fd 0) as a terminal, and it fails.

Fix: add -t to ssh options to enforce a pty.

[kolyshkin]

My tests are now passing. Got unrelated failures in podman:

STEP 12: COPY tests/integration/multi-arch.bash tests/integration/
--> b6acd7d5fd6
STEP 13: ENV ROOTFS /busybox
--> c11d4c496d4
STEP 14: RUN mkdir -p "${ROOTFS}"
error opening file `/sys/fs/cgroup//system.slice/buildah-buildah794169410-22047.scope/cgroup.freeze`: No such file or directory
kill container: No such process
error running container: error reading container state: exit status 1
Error: error building at STEP "RUN mkdir -p "${ROOTFS}"": error while running runtime: exit status 1
travis_time:end:0874bc6c:start=1586682807783553791,finish=1586683034446188144,duration=226662634353,event=script
�[0K�[31;1mThe command "sudo ssh default sudo podman build -t test /vagrant" exited with 125.�[0m
travis_time:start:0a243466
�[0K$ sudo ssh default sudo podman run --privileged --cgroupns=private -v /lib/modules:/lib/modules:ro test make localunittest
Trying to pull docker.io/library/test...
  denied: requested access to the resource is denied
Trying to pull registry.fedoraproject.org/test...
  manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/test...
  name unknown: Repo not found
Trying to pull registry.centos.org/test...
  manifest unknown: manifest unknown
Trying to pull quay.io/test...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
Error: unable to pull test: 5 errors occurred:
	* Error initializing source docker://test:latest: Error reading manifest latest in docker.io/library/test: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

	* Error initializing source docker://registry.fedoraproject.org/test:latest: Error reading manifest latest in registry.fedoraproject.org/test: manifest unknown: manifest unknown
	* Error initializing source docker://registry.access.redhat.com/test:latest: Error reading manifest latest in registry.access.redhat.com/test: name unknown: Repo not found
	* Error initializing source docker://registry.centos.org/test:latest: Error reading manifest latest in registry.centos.org/test: manifest unknown: manifest unknown
	* Error initializing source docker://quay.io/test:latest: Error reading manifest latest in quay.io/test: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"


travis_time:end:0a243466:start=1586683034452475855,finish=1586683036696948228,duration=2244472373,event=script
�[0K�[31;1mThe command "sudo ssh default sudo podman run --privileged --cgroupns=private -v /lib/modules:/lib/modules:ro test make localunittest" exited with 125.�[0m

[kolyshkin]

OK, looks like everything works except runc events (which is not modified for cgroupv2 memory.events) and, for some strange reason, runc ps (which works unless --systemd-cgroup is set).

[AkihiroSuda]

Travis didn't notify the result, but it is passing 👍

LGTM, thanks!

[kolyshkin]

for some strange reason, runc ps (which works unless --systemd-cgroup is set)

This is pretty shocking: cgroup v2 driver do not place pids in the cgroup (#2310), and the only test to show it is runc ps.

This seems to be easy to fix but I want to do it on top of #2299 fix is trivial, see #2311

[kolyshkin]

@mrunalp @crosbymichael PTAL

[kolyshkin]

• fixed ps failures by adding dnf update container-selinux to Vagrantfile :)
• rebased

[kolyshkin]

OK, vagrant on Travis is installing container-selinux-2.117.0-1.gitbfde70a.fc31 while on my vagrant box I have container-selinux-2.124.0-3.fc31.noarch. Alas, we need to run dnf update.

[AkihiroSuda]

LGTM

[mrunalp]

LGTM