memcached:1.6.31-alpine is broken and directly stops with exit code 137 on Ubuntu 20.04 (docker.io 24.0.7-0ubuntu2~20.04.1)

[heidemn-faro]

Environment

EC2 instance, instance type m6i.large.

ubuntu@ip-x-x-x-x$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

ubuntu@ip-x-x-x-x:~$ docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.2

ubuntu@ip-x-x-x-x:~$ uname -a
Linux ip-172-31-47-70 5.15.0-1068-aws #74~20.04.1-Ubuntu SMP Tue Aug 6 19:32:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Not working

(Adding or removing the -ti options makes no difference.)

ubuntu@ip-x-x-x-x:~$ time docker run -ti memcached:1.6.31-alpine; echo $?

real    0m0.529s
user    0m0.010s
sys     0m0.014s
137

I tested the same on my local WSL2 machine (Ubuntu 20.04 with Docker version 26.1.0, build 9714adc), and there it works fine.

Other versions - working fine

It seems that only the Alpine version of 1.6.31 is broken.
I didn't check if they fully work, but at least they keep running instead of aborting with an error.

ubuntu@ip-x-x-x-x:~$ docker run -ti memcached:1.6.30-alpine
# Runs fine; killed with Ctrl-C.

ubuntu@ip-x-x-x-x:~$ docker run -ti memcached:1.6.31
# Runs fine; killed with Ctrl-C.

[heidemn-faro]

I tried to upgrade the docker.io package, but didn't help, same result.
Docker.io is the package from the official Ubuntu package repo.

Unpacking docker.io (24.0.7-0ubuntu2~20.04.1) over (20.10.21-0ubuntu1~20.04.2) ...

[heidemn-faro]

Interesting: After a sudo apt upgrade + reboot, it's working on the EC2 instance.

[heidemn-faro]

On one of the instances, I'm getting a flaky behavior:

$ docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.2

$ docker ps --all
CONTAINER ID   IMAGE                     COMMAND                  CREATED          STATUS                        PORTS     NAMES
e662724a7dd5   memcached:1.6.31-alpine   "docker-entrypoint.s…"   3 seconds ago    Exited (137) 3 seconds ago              charming_moser
3063083f31df   memcached:1.6.31-alpine   "docker-entrypoint.s…"   5 seconds ago    Exited (137) 5 seconds ago              stupefied_kowalevski
35d307cc45fa   memcached:1.6.31-alpine   "docker-entrypoint.s…"   15 seconds ago   Exited (137) 14 seconds ago             great_heyrovsky
e55c06298c52   memcached:1.6.31-alpine   "docker-entrypoint.s…"   23 seconds ago   Exited (137) 23 seconds ago             hardcore_lederberg
de540fd3b68e   memcached:1.6.31-alpine   "docker-entrypoint.s…"   2 minutes ago    Exited (0) 2 minutes ago                sad_wiles
b24b1370f58f   memcached:1.6.31-alpine   "docker-entrypoint.s…"   2 minutes ago    Exited (0) 2 minutes ago                distracted_golick

I've updated the same instance to docker.io (24.0.7-0ubuntu2~20.04.1) (no other packages upgraded) + did a reboot -> Works now. No it doesn't. Still flaky.

[heidemn-faro]

Maybe this helps: docker inspect 26e267333903

[
    {
        "Id": "26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b",
        "Created": "2024-09-11T11:48:53.694984569Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "memcached"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 137,
            "Error": "",
            "StartedAt": "2024-09-11T11:48:53.93512905Z",
            "FinishedAt": "2024-09-11T11:48:53.936962327Z"
        },
        "Image": "sha256:2aae0d824f4bc7874b3fd26fc5e61b2fe8c3fc9b58f82f3be24a6010c05ec1c1",
        "ResolvConfPath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/hostname",
        "HostsPath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/hosts",
        "LogPath": "/var/lib/docker/containers/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b/26e2673339033fc9365d45295f2be5551f92e0dddcacb151ed49666cda9c0b3b-json.log",
        "Name": "/quizzical_mayer",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                57,
                199
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498-init/diff:/var/lib/docker/overlay2/6b4156e1c73fc9bfe3435e1267bf456b9acc9d3808a59d0f69120505c849b2e9/diff:/var/lib/docker/overlay2/0291d95600090ab3469f937846edda895c5e992e2e30b6c5f62a008ba68f0bea/diff:/var/lib/docker/overlay2/39c2192c4a4ce4e2563fc62d50c6f5bc3d06052844f4c1732265bc6442d9e1a5/diff:/var/lib/docker/overlay2/accd7f99f8d4883549fcae51b44c5fdfe5c5c69a58590f2deace34feace95267/diff:/var/lib/docker/overlay2/f4c5a0c47c04223a936fec8d1ebabeda04c4a349b6c02fe45c7810288119e6f0/diff:/var/lib/docker/overlay2/14ff7ed0dee5f2bbb4ad1d59dd9d4571fcd85f873e2387158e1f95ca89689a23/diff",
                "MergedDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498/merged",
                "UpperDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498/diff",
                "WorkDir": "/var/lib/docker/overlay2/7445ac23f0f9d5d24e3f0adfa86ba149e9a066e67d21f8838681af1ae9fbd498/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "26e267333903",
            "Domainname": "",
            "User": "memcache",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "11211/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "MEMCACHED_VERSION=1.6.31",
                "MEMCACHED_URL=https://memcached.org/files/memcached-1.6.31.tar.gz",
                "MEMCACHED_SHA1=85e2cb9520beba71d7fc69f5717208a57facde28"
            ],
            "Cmd": [
                "memcached"
            ],
            "Image": "memcached:1.6.31-alpine",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "5ae80fd89db380d40f9a3e4ac0530634c549ae7ef8a8111470e248148dce38da",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/5ae80fd89db3",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "19d8795c83c1d65cdb2a041242daca1a3f36a9007528068ab860c23495558e62",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

[heidemn-faro]

Relevant diffs:

• 3598423
• memcached/memcached@1.6.30...1.6.31

[heidemn-faro]

Maybe this also helps:

$ docker run -v $PWD/log:/var/log/memcached.log -ti memcached:1.6.31-alpine -vv; echo $?; cat log
slab class   1: chunk size        96 perslab   10922
slab class   2: chunk size       120 perslab    8738
slab class   3: chunk size       152 perslab    6898
slab class   4: chunk size       192 perslab    5461
slab class   5: chunk size       240 perslab    4369
slab class   6: chunk size       304 perslab    3449
slab class   7: chunk size       384 perslab    2730
slab class   8: chunk size       480 perslab    2184
slab class   9: chunk size       600 perslab    1747
slab class  10: chunk size       752 perslab    1394
slab class  11: chunk size       944 perslab    1110
slab class  12: chunk size      1184 perslab     885
slab class  13: chunk size      1480 perslab     708
slab class  14: chunk size      1856 perslab     564
slab class  15: chunk size      2320 perslab     451
slab class  16: chunk size      2904 perslab     361
slab class  17: chunk size      3632 perslab     288
slab class  18: chunk size      4544 perslab     230
slab class  19: chunk size      5680 perslab     184
slab class  20: chunk size      7104 perslab     147
slab class  21: chunk size      8880 perslab     118
slab class  22: chunk size     11104 perslab      94
slab class  23: chunk size     13880 perslab      75
slab class  24: chunk size     17352 perslab      60
slab class  25: chunk size     21696 perslab      48
slab class  26: chunk size     27120 perslab      38
slab class  27: chunk size     33904 perslab      30
slab class  28: chunk size     42384 perslab      24
slab class  29: chunk size     52984 perslab      19
slab class  30: chunk size     66232 perslab      15
slab class  31: chunk size     82792 perslab      12
slab class  32: chunk size    103496 perslab      10
slab class  33: chunk size    129376 perslab       8
slab class  34: chunk size    161720 perslab       6
slab class  35: chunk size    202152 perslab       5
slab class  36: chunk size    252696 perslab       4
slab class  37: chunk size    315872 perslab       3
slab class  38: chunk size    394840 perslab       2
slab class  39: chunk size    524288 perslab       2
<26 server listening (auto-negotiate)
<27 server listening (auto-negotiate)
137

When starting memcached:1.6.30-alpine (the version which works correctly for me), the logs look the same - <27 server listening (auto-negotiate) is the last log entry for the startup.