Enabling kmem accounting can break applications on CentOS7

[TvdW]

After upgrading from Docker (CE) 17.05.0 to 17.12.0, these kernel messages started showing up on my machines:

[43073.004575] SLUB: Unable to allocate memory on node -1 (gfp=0x8020)
[43073.022211]   cache: ip6_dst_cache(0:18479a49f76b66461d8fedd17a6eba407f08cc8960bd0246d39df8eb21f20b4f), object size: 448, buffer size: 448, default order: 2, min order: 0
[43073.061420]   node 0: slabs: 74, objs: 2610, free: 0
[43073.077875]   node 1: slabs: 88, objs: 3141, free: 0

I was able to reproduce this on Docker 17.06.0, and eventually traced it to the commit introduced by #1350 which enables kmem accounting for all containers. But as Docker helpfully suggests, kmemcg are experimental before linux 4.0:

$ sudo docker update --kernel-memory 1000g 18479a49f76b
You specified a kernel memory limit on a kernel older than 4.0. Kernel memory limits are experimental on older kernels, it won't work as expected and can cause your system to be unstable.

After a few more tests I was able to reproduce the issue on 17.05.0, by passing --kernel-memory 1000g to docker run. The kernel log slowly fills up (10 messages per hour?) with SLUB warnings, and containers seem far less stable than normal (i.e. they crash).

Steps to reproduce

1. CentOS 7, latest kernel (3.10.0-693.17.1.el7.x86_64)
2. Docker 17.06.0+ or runc with equivalent settings
3. kmem limit unset, mem limit 40G, with 80G free memory left for page caches
4. An application that very heavily uses the local disk to cause caches to build up (in my case Apache Cassandra)

Eventually, these messages will start popping up in the kernel logs, and in rare cases it leads to an application getting killed/crashed.

---

With all of the above said, I'm 99% sure this is a kernel bug related to running an ancient kernel, and runc's patch would at best be a workaround. If only I had a way to get redhat's attention so they can fix it 😃

[hqhq]

@TvdW The root cause is there are kernel memory limit bugs in 3.10, if you don't want to use kernel memory limit because it's not stable on your kernel, the best solution would be to disable kernel memory limit on your kernel.

I can't think of a way to workaround this on runc side without causing issues like #1083 and #1347 , unless we add some ugly logic like do different things for different kernel versions, I'm afraid that won't be an option.

[cizixs]

@hqhq Is there any document on how to disable kernel memory limit?

We're using CentOS 7, and run into a similar problem. It seems CentOS enables kernel memory limit by default even though it's a experimental feature(not sure why), and RunC uses this feature without checking.

[pmoust]

@hqhq cgroup.memory=nokmem is not available on RHEL/CentOS 3.10 (kmem accounting is active in the legacy cgroup hierarchy)

[hqhq]

@cizixs You need to disable CONFIG_MEMCG_KMEM in kernel config and recompile the kernel, or you can help to add an option in runc and other tools on top of runc to disable kmem accounting as I suggested in kubernetes/kubernetes#61937 (comment) .

[jieyu]

I got a bit confused by this ticket. I tested with docker 17.09.0-ce and CentOS 7:

[jie@core-dev memory]$ docker info | grep Version
Server Version: 17.09.0-ce
Kernel Version: 3.10.0-693.5.2.el7.x86_64
[jie@core-dev memory]$ find /sys/fs/cgroup/memory -name memory.limit_in_bytes -type f -print -exec cat {} \; | grep -A 1 docker                                                                                                                                                                                               
/sys/fs/cgroup/memory/docker/c02dfa0697e198453e1f352a4f794dbcd8bda0dbc851fc27031b5395201a5b6e/memory.limit_in_bytes
1073741824
/sys/fs/cgroup/memory/docker/memory.limit_in_bytes
9223372036854771712
[jie@core-dev memory]$ find /sys/fs/cgroup/memory -name memory.kmem.limit_in_bytes -type f -print -exec cat {} \; | grep -A 1 docker
/sys/fs/cgroup/memory/docker/c02dfa0697e198453e1f352a4f794dbcd8bda0dbc851fc27031b5395201a5b6e/memory.kmem.limit_in_bytes
9223372036854771712
/sys/fs/cgroup/memory/docker/memory.kmem.limit_in_bytes
9223372036854771712
[jie@core-dev memory]$ docker inspect c02dfa0697e1 | grep Memory
            "Memory": 1073741824,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 2147483648,
            "MemorySwappiness": null,

I don't see kmem accounting being turned on by default in docker 17.09.0-ce. Am I missing something?

[jieyu]

Ok, I think I get it now because runc will enable the accounting by setting it to 1 first and set it to -1 to just enable the accounting feature
https://github.com/opencontainers/runc/pull/1350/files#diff-b56f02f0dc51a436b542c6d80bdef7e8R68

[cyphar]

In theory this is fixed by #1921 -- though RHEL will have to rebuild runc with BUILDTAGS=nokmem...

[crosbymichael]

@cyphar or fix their kernel....

[cyphar]

This is fixed by #1921. But I just noticed there's actually a bug in it (it will ignore explicitly-set kernel memory limits).

[pmoust]

So is there a follow-up on #1921 @cyphar ?

[pmoust]

I did some digging and the follow-up item is #1938, mentioning it here for posterity.

[cyphar]

Yes, sorry I tagged #1921 and not this issue.

[ethercflow]

Please look at this: https://pingcap.com/blog/try-to-fix-two-linux-kernel-bugs-while-testing-tidb-operator-in-k8s/

[zionwu]

There is several wrong information in the artical https://pingcap.com/blog/try-to-fix-two-linux-kernel-bugs-while-testing-tidb-operator-in-k8s :

1. runc of Docker v18.09.1 does not disable kmem accounting.
   The approach to verify is running the command docker run -d --name test --kernel-memory 100M nginx:1.14.2. If kmem accounting is enabled, the command succeed and container is created. If kmem accounting is disabled, the command fails with the following message:

docker run -d --name test --kernel-memory 100M nginx:1.14.2
eb8dfe53ea903a9207bd356999b14c7ef3b57d9d35d33635c0e8b700387f60ce
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:396: setting cgroup config for procHooks process caused \\\"kernel memory accounting disabled in this runc build\\\"\"": unknown.

2. The buildtags BUILDTAGS="nokmem" is useless. The correct buildtags is GOFLAGS="-tags=nokmem". The complete command to compile kubelet is

./build/run.sh make kubelet KUBE_BUILD_PLATFORMS=linux/amd64 BUILDTAGS="nokmem"

3. A better approach to verify if kmem accounting is disabled in the host after reboot, is to run the following command:

find /sys/fs/cgroup/memory/kubepods/ -name memory.kmem.slabinfo | xargs cat > /tmp/mem.txt

If mem.txt is empty, kmem accounting is disabled.

[cyphar]

We didn't write the article you linked (and this issue is long since closed) so I don't know why you've posted in this issue, but to your points:

1. Yes, Docker doesn't disable kmem accounting in their builds of runc but issue trackers is not for the Docker project.
2. BUILDTAGS=nokmem does work if you're building runc directly using our project's Makefile. If you're building runc as part of another project's build system (such as Kubernetes) then you should consult their documentation.
3. You can also just check if /sys/fs/cgroup/memory contains any "memory.kmem" files (such as /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes).

EDIT: Ah, someone else linked to the article above.