Containers not exiting

[bryanlarsen]

1. Issue or feature description

Docker containers don't exit after their process has exited

2. Steps to reproduce the issue

1. Use nvidia-docker with Kubernetes 1.8.2 for about a month.
2. docker run -it --rm ubuntu:16.04 echo hello

"hello" gets successfully echoed, but the shell prompt does not reappear, the container is hung.

This error message appears in the docker logs:

Handler for POST /v1.32/containers/8024d87149987268cc55cb4928d22839c6d7091505b634d90479afa4a9a9b3f0/resize returned error: rpc error: code = Unknown desc = containerd: container not found

Restarting docker fixed the problem.

3. Information to attach (optional if deemed irrelevant)

• Kernel version from uname -a

Linux chaos 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Driver information from nvidia-smi -a

Driver Version : 384.90

• Docker version from docker version

17.09.0ce-0ubuntu

• NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'

ii  libnvidia-container-tools              1.0.0~alpha.2-1          amd64                    NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64             1.0.0~alpha.2-1          amd64                    NVIDIA container runtime library
ii  nvidia-384                             384.90-0ubuntu0.16.04.1  amd64                    NVIDIA binary driver - version 384.90
ii  nvidia-container-runtime               1.1.0+docker17.09.0-1    amd64                    NVIDIA container runtime
ii  nvidia-docker2                         2.0.1+docker17.09.0-1    all                      nvidia-docker CLI wrapper
ii  nvidia-opencl-icd-384                  384.90-0ubuntu0.16.04.1  amd64                    NVIDIA OpenCL ICD
un  nvidia-persistenced                    <none>                   <none>                   (no description available)
ii  nvidia-prime                           0.8.2                    amd64                    Tools to enable NVIDIA's Prime
ii  nvidia-settings                        361.42-0ubuntu1          amd64                    Tool for configuring the NVIDIA graphics driver

• NVIDIA container library version from nvidia-container-cli -V

version: 1.0.0
build date: 2017-10-30T23:47+00:00
build revision: ec15c7233bd2de821ad5127cb0de6b52d9d2083c
build compiler: gcc-5 5.4.0 20160609
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

• NVIDIA container library logs (see troubleshooting)

I've turned these on. If/when this reproduces, I'll attach. As mentioned, it took about a month for this to appear.

[flx42]

Very unlikely to be a nvidia-docker issue, containerd was probably not running (or uninstalled? being upgraded?) while it happened.

[flx42]

@bryanlarsen do you have another repro for this?

[flx42]

ping @bryanlarsen

[bryanlarsen]

It hasn't recurred since reporting it 2.5 weeks ago.

[flx42]

Closing for now, reopen if you ever get more details.