Error Starting Kubernetes: limactl exited with code 1: Waiting for the guest agent to be running

[carlosdoordash]

Rancher Desktop Version 0.4.1
Mac OS Big Sur Version 11.5.2 (20G95)

When I launch Rancher Desktop, I wait a long time and then see "Error Starting Kubernetes: ... limactl exited with code 1". My lima.log has many Waiting logs like...

time="2021-09-13T15:07:38-07:00" level=info msg="[hostagent] Waiting for the essential requirement 4 of 4: \"the guest agent to be running\""

I looked at the other Kubernetes issues and they helped me to get to this point. Note that my first issue Waiting for the essential requirement 1 of 4: \"ssh\"" was resolved by upgrading to alpine-lima v0.1.4 to get the fix for the dot in my carlos.herrera username. I upgraded by replacing /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/alpline-lima-v0.1.0-std-3.13.5.iso and got to a fresh state with rm -rf ~/Library/State/rancher-desktop

background.log
k8s.log
lima.log

[jandubois]

Could you follow the instructions in #562 (comment) to create a connection to the VM while it is still waiting for the guest agent?

Please get a copy of the /var/log/lima-init.log file and attach it here (there is no need to get the user-data and authorized keys; your startup is going far enough that they won't be a problem).

[carlosdoordash]

socat stdio ~/Library/State/rancher-desktop/lima/rancher-desktop/serial.sock

root
root
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

lima-rancher-desktop:~# ^[[47;25Rcat /var/log/lima-init.log
cat /var/log/lima-init.log
+ ln -s /var/log/lima-init.log /var/log/cloud-init-output.log
+ LIMA_CIDATA_MNT=/mnt/lima-cidata
+ LIMA_CIDATA_DEV=/dev/disk/by-label/cidata
+ mkdir -p -m 700 /mnt/lima-cidata
+ mount -o 'ro,mode=0700,dmode=0700,overriderockperm,exec,uid=0' /dev/disk/by-label/cidata /mnt/lima-cidata
+ export LIMA_CIDATA_MNT
+ . /mnt/lima-cidata/lima.env
+ LIMA_CIDATA_USER=carlos.herrera
+ LIMA_CIDATA_UID=501
+ LIMA_CIDATA_MOUNTS=1
+ LIMA_CIDATA_MOUNTS_0_MOUNTPOINT=/Users/carlos.herrera/Library/Caches/rancher-desktop/k3s
+ LIMA_CIDATA_CONTAINERD_USER=
+ LIMA_CIDATA_CONTAINERD_SYSTEM=
+ LIMA_CIDATA_SLIRP_GATEWAY=192.168.5.2
+ awk '/^local-hostname:/ {print $2}' /mnt/lima-cidata/meta-data
+ LIMA_CIDATA_HOSTNAME=lima-rancher-desktop
+ hostname lima-rancher-desktop
+ LIMA_CIDATA_HOMEDIR=/home/carlos.herrera.linux
+ useradd --home-dir /home/carlos.herrera.linux --create-home --uid 501 carlos.herrera
+ echo 'carlos.herrera ALL=(ALL) NOPASSWD:ALL'
+ LIMA_CIDATA_SSHDIR=/home/carlos.herrera.linux/.ssh
+ mkdir -p -m 700 /home/carlos.herrera.linux/.ssh
+ awk '/ssh-authorized-keys/ {flag=1; next} /^ *$/ {flag=0} flag {sub(/^ +- /, ""); gsub("\"", ""); print $0}' /mnt/lima-cidata/user-data
+ id -g carlos.herrera
+ LIMA_CIDATA_GID=1000
+ chown -R 501:1000 /home/carlos.herrera.linux/.ssh
+ chmod 600 /home/carlos.herrera.linux/.ssh/authorized_keys
+ mkdir -p /var/lib/lima-init
+ IP_RENAME=/var/lib/lima-init/ip-rename
+ ip -o link
+ awk -f /usr/bin/lima-network.awk /var/lib/lima-init/ip-link /mnt/lima-cidata/network-config
+ chmod +x /var/lib/lima-init/ip-rename
+ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:55:55:27:5d:74 brd ff:ff:ff:ff:ff:ff
+ /var/lib/lima-init/ip-rename
+ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:55:55:27:5d:74 brd ff:ff:ff:ff:ff:ff
+ awk -f- /mnt/lima-cidata/network-config
+ awk -f- /mnt/lima-cidata/network-config
+ awk '/nameservers:/{flag=1; next} /^[^ ]/{flag=0} flag {gsub("^ +- +", ""); print}' /mnt/lima-cidata/user-data
+ tr '\n' ' '
+ DNS=
+ '[' -n  ]
+ exec /mnt/lima-cidata/boot.sh
LIMA| Executing /mnt/lima-cidata/boot/00-modprobe.sh
Loading kernel module "fuse"
Loading kernel module "tun"
Loading kernel module "tap"
Loading kernel module "bridge"
Loading kernel module "veth"
Loading kernel module "ip_tables"
Loading kernel module "ip6_tables"
Loading kernel module "iptable_nat"
Loading kernel module "ip6table_nat"
Loading kernel module "iptable_filter"
Loading kernel module "ip6table_filter"
Loading kernel module "nf_tables"
Loading kernel module "x_tables"
Loading kernel module "xt_MASQUERADE"
Loading kernel module "xt_addrtype"
Loading kernel module "xt_comment"
Loading kernel module "xt_conntrack"
Loading kernel module "xt_mark"
Loading kernel module "xt_multiport"
Loading kernel module "xt_nat"
Loading kernel module "xt_tcpudp"
Loading kernel module "overlay"
LIMA| Executing /mnt/lima-cidata/boot/01-alpine-ash-as-bash.sh
LIMA| Executing /mnt/lima-cidata/boot/03-etc-hosts.sh
+ sed -i /host.lima.internal/d /etc/hosts
+ echo -e '192.168.5.2\thost.lima.internal'
LIMA| Executing /mnt/lima-cidata/boot/05-persistent-data-volume.sh
+ test -f /etc/alpine-release
+ DATADIRS='/etc /home /tmp /usr/local /var/lib'
+ awk '$2 == "/" {print $3}' /proc/mounts
+ '[' tmpfs '==' tmpfs ]
+ mkdir -p /mnt/data
+ '[' -e /dev/disk/by-label/data-volume ]
+ mount -t ext4 /dev/disk/by-label/data-volume /mnt/data
+ '[' -d /mnt/data/etc ]
+ mkdir -p /etc
+ mount --bind /mnt/data/etc /etc
+ '[' -d /mnt/data/home ]
+ mkdir -p /home
+ mount --bind /mnt/data/home /home
+ '[' -d /mnt/data/tmp ]
+ mkdir -p /tmp
+ mount --bind /mnt/data/tmp /tmp
+ '[' -d /mnt/data/usr/local ]
+ mkdir -p /usr/local
+ mount --bind /mnt/data/usr/local /usr/local
+ '[' -d /mnt/data/var/lib ]
+ mkdir -p /var/lib
+ mount --bind /mnt/data/var/lib /var/lib
LIMA| Executing /mnt/lima-cidata/boot/10-alpine-prep.sh
+ test -f /etc/alpine-release
+ BRANCH=edge
+ awk '-F=' '$1=="VERSION_ID" {print $2}' /etc/os-release
+ VERSION_ID=3.13.6
+ BRANCH=v3.13
+ URL=https://dl-cdn.alpinelinux.org/alpine/v3.13/main
+ grep -q '^https://dl-cdn.alpinelinux.org/alpine/v3.13/main$' /etc/apk/repositories
+ URL=https://dl-cdn.alpinelinux.org/alpine/v3.13/community
+ grep -q '^https://dl-cdn.alpinelinux.org/alpine/v3.13/community$' /etc/apk/repositories
+ usermod -p '*' carlos.herrera
+ sed -i 's/AllowTcpForwarding no/AllowTcpForwarding yes/g' /etc/ssh/sshd_config
+ rc-service --ifstarted sshd reload
+ rc-service cgroups start
 * Mounting cgroup filesystem ... [ ok ]
+ rc-update add acpid
 * rc-update: acpid already installed in runlevel `default'; skipping
+ rc-service acpid start
 * Starting busybox acpid ... [ ok ]
LIMA| Executing /mnt/lima-cidata/boot/20-rootless-base.sh
+ command -v systemctl
+ exit 0
LIMA| Executing /mnt/lima-cidata/boot/25-guestagent-base.sh
+ seq 0 0
+ mountpointvar=LIMA_CIDATA_MOUNTS_0_MOUNTPOINT
+ eval echo '$LIMA_CIDATA_MOUNTS_0_MOUNTPOINT'
+ echo /Users/carlos.herrera/Library/Caches/rancher-desktop/k3s
+ mountpoint=/Users/carlos.herrera/Library/Caches/rancher-desktop/k3s
+ mkdir -p /Users/carlos.herrera/Library/Caches/rancher-desktop/k3s
+ chown carlos.herrera /Users/carlos.herrera/Library/Caches/rancher-desktop/k3s
chown: unknown user/group carlos:herrera
LIMA| WARNING: Failed to execute /mnt/lima-cidata/boot/25-guestagent-base.sh
LIMA| Executing /mnt/lima-cidata/boot/30-install-packages.sh
+ command -v apt-get
+ command -v dnf
+ command -v pacman
+ command -v zypper
+ command -v apk
+ '[' 1 -gt 0 ]
+ command -v sshfs
+ update_fuse_conf
+ '[' 1 -gt 0 ]
+ grep -q ^user_allow_other /etc/fuse.conf
LIMA| Executing /mnt/lima-cidata/boot/40-install-containerd.sh
+ '['  '!=' 1 ]
+ '['  '!=' 1 ]
+ exit 0
LIMA| Executing /mnt/lima-cidata/provision.system/00000000
+ errors=0
+ '[' -d /etc/rancher ]
+ ls -A /etc/rancher
+ '[' -n  ]
+ mkdir -p /mnt/data//etc/rancher /etc/rancher
+ mount --bind /mnt/data//etc/rancher /etc/rancher
+ mount --make-shared /
+ mount --make-shared /etc/rancher
+ mount --make-shared /tmp
+ mount --make-shared /var/lib
+ echo 'Done mounting Rancher persistent volumes.'
Done mounting Rancher persistent volumes.
+ exit 0
LIMA| Executing /mnt/lima-cidata/provision.system/00000001
 * WARNING: machine-id has already been started
LIMA| Exiting with code 1

[carlosdoordash]

Looks like another chown bug...

+ chown carlos.herrera /Users/carlos.herrera/Library/Caches/rancher-desktop/k3s
chown: unknown user/group carlos:herrera
LIMA| WARNING: Failed to execute /mnt/lima-cidata/boot/25-guestagent-base.sh

[jandubois]

Thanks you!

I don't understand why we didn't run into this when we discovered the first issue with chown, just in an earlier place.

I've created a PR against lima to fix this, but there is no easy way to upgrade the bundled lima in Rancher Desktop, so you'll have to wait for the next RD release to get the fix.

[carlosdoordash]

Thanks @jandubois for creating the lima PR: lima-vm/lima#241

[jandubois]

FYI: Since we are switching from adduser to useradd, and the latter doesn't allow dots in Linux usernames, we are going to map all those invalid names to a fallback name of "lima" in the future: lima-vm/lima#243

So the changes to the chown commands will not really be needed afterwards, but they are benign, so they are going to stay as well.