stuck on "Waiting for the essential requirement 1 of 4: "ssh" with non-Ubuntu and libslirp v4.6.1 #255

LATBauerdick · 2021-09-17T21:42:50Z

I get stuck on "Waiting for the essential requirement 1 of X: "ssh" even with libslirp v4.6.1 for non-Ubuntu guests only, both for alpine and for arch linux -- works fine with default.yaml and ubuntu.

The initial limactl start alpine works, but after a limactl stop alpine I get stuck with a large number of messages like

...
INFO[0080] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
...

and finally

...
INFO[0587] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0597] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0597] [hostagent] Waiting for the essential requirement 2 of 4: "sshfs binary to be installed" 
FATA[0601] did not receive an event with the "running" status

again, Ubuntu and default.yaml work just fine.

The text was updated successfully, but these errors were encountered:

jandubois · 2021-09-18T01:58:54Z

Which lima version did you use? A number of people ran into problems because they had user names that did not work properly on all Linux systems, e.g containing a dot firstName.lastName, email address: [email protected], all numeric: 12345678. In Lima 0.6.4 we are mapping all names that don't match the rather strict rules of useradd from the shadow package to a default name of lima to avoid these issues.

So if you have been using an earlier Lima version, please update and see if that fixes your problem. And if you created your own alpine.yaml, please update that as well to use the latest ISO image, as some issues have been fixed in there as well (e.g. having a uid on macOS > 25600).

LATBauerdick · 2021-09-18T04:24:30Z

I just installed lima from brew: limactl version 0.6.4

username is bauerdic, so that should be ok.

alpine.yaml is from the head.

jandubois · 2021-09-18T04:30:20Z

Could you follow the instructions from rancher-sandbox/rancher-desktop#562 (comment) (in spirit, the directory with the socket will be ~/.lima/alpine), to log into the Alpine VM while lima is still waiting for ssh to come up, and collect the output from sudo cat /var/log/lima-init.log and append it to this bug?

Whew, run-on-sentences-are-us! 😄

LATBauerdick · 2021-09-18T04:51:14Z

lima-alpine:~# ^[[53;16Rcat /mnt/lima-cidata/user-data
cat /mnt/lima-cidata/user-data
#cloud-config
# vim:syntax=yaml

growpart:
  mode: auto
  devices: ['/']

users:
  - name: "bauerdic"
    uid: "6170"
    homedir: "/home/bauerdic.linux"
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: true
    ssh-authorized-keys:
      - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKSfQOtfNmJm98lw8t7SRynbbgZhK+22Oy4syAAzvSLv [email protected]"
      - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJOXZjedCEONef8tQoqk8iZYODg0VoONlyfIz5tFfWXz [email protected]"

write_files:
 - content: |
      #!/bin/sh
      set -eux
      LIMA_CIDATA_MNT="/mnt/lima-cidata"
      LIMA_CIDATA_DEV="/dev/disk/by-label/cidata"
      mkdir -p -m 700 "${LIMA_CIDATA_MNT}"
      mount -o ro,mode=0700,dmode=0700,overriderockperm,exec,uid=0 "${LIMA_CIDATA_DEV}" "${LIMA_CIDATA_MNT}"
      export LIMA_CIDATA_MNT
      exec "${LIMA_CIDATA_MNT}"/boot.sh
   owner: root:root
   path: /var/lib/cloud/scripts/per-boot/00-lima.boot.sh
   permissions: '0755'
# This has no effect on systems using systemd-resolved, but is used
# on e.g. Alpine to set up /etc/resolv.conf on first boot.

manage_resolv_conf: true

resolv_conf:
  nameservers:
  - 191.68.0.142
  - 192.168.0.1
  - 192.168.5.3

jandubois · 2021-09-18T06:52:41Z

Sorry, I'm looking for the output of sudo cat /var/log/lima-init.log; can you attach that as well? Thanks!

LATBauerdick · 2021-09-18T14:01:34Z

sorry, here it is:

lima-alpine:~$ ^[[53;16Rsudo cat /var/log/lima-init.log
sudo cat /var/log/lima-init.log
+ ln -s /var/log/lima-init.log /var/log/cloud-init-output.log
+ LIMA_CIDATA_MNT=/mnt/lima-cidata
+ LIMA_CIDATA_DEV=/dev/disk/by-label/cidata
+ mkdir -p -m 700 /mnt/lima-cidata
+ mount -o 'ro,mode=0700,dmode=0700,overriderockperm,exec,uid=0' /dev/disk/by-label/cidata /mnt/lima-cidata
+ export LIMA_CIDATA_MNT
+ . /mnt/lima-cidata/lima.env
+ LIMA_CIDATA_USER=bauerdic
+ LIMA_CIDATA_UID=6170
+ LIMA_CIDATA_MOUNTS=2
+ LIMA_CIDATA_MOUNTS_0_MOUNTPOINT=/home/bauerdic
+ LIMA_CIDATA_MOUNTS_1_MOUNTPOINT=/tmp/lima
+ LIMA_CIDATA_CONTAINERD_USER=
+ LIMA_CIDATA_CONTAINERD_SYSTEM=
+ LIMA_CIDATA_SLIRP_GATEWAY=192.168.5.2
+ awk '/^local-hostname:/ {print $2}' /mnt/lima-cidata/meta-data
+ LIMA_CIDATA_HOSTNAME=lima-alpine
+ hostname lima-alpine
+ LIMA_CIDATA_HOMEDIR=/home/bauerdic.linux
+ useradd --home-dir /home/bauerdic.linux --create-home --uid 6170 bauerdic
+ echo 'bauerdic ALL=(ALL) NOPASSWD:ALL'
+ LIMA_CIDATA_SSHDIR=/home/bauerdic.linux/.ssh
+ mkdir -p -m 700 /home/bauerdic.linux/.ssh
+ awk '/ssh-authorized-keys/ {flag=1; next} /^ *$/ {flag=0} flag {sub(/^ +- /, ""); gsub("\"", ""); print $0}' /mnt/lima-cidata/user-data
+ id -g bauerdic
+ LIMA_CIDATA_GID=6170
+ chown -R 6170:6170 /home/bauerdic.linux/.ssh
+ chmod 600 /home/bauerdic.linux/.ssh/authorized_keys
+ mkdir -p /var/lib/lima-init
+ IP_RENAME=/var/lib/lima-init/ip-rename
+ ip -o link
+ awk -f /usr/bin/lima-network.awk /var/lib/lima-init/ip-link /mnt/lima-cidata/network-config
+ chmod +x /var/lib/lima-init/ip-rename
+ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:55:55:40:47:c7 brd ff:ff:ff:ff:ff:ff
+ /var/lib/lima-init/ip-rename
+ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:55:55:40:47:c7 brd ff:ff:ff:ff:ff:ff
+ awk -f- /mnt/lima-cidata/network-config
+ awk -f- /mnt/lima-cidata/network-config
+ awk '/nameservers:/{flag=1; next} /^[^ ]/{flag=0} flag {gsub("^ +- +", ""); print}' /mnt/lima-cidata/user-data
+ tr '\n' ' '
+ DNS='191.68.0.142 192.168.0.1 192.168.5.3 '
+ '[' -n '191.68.0.142 192.168.0.1 192.168.5.3 ' ]
+ sed -i '/export dns/a dns="191.68.0.142 192.168.0.1 192.168.5.3 "' /usr/share/udhcpc/default.script
+ exec /mnt/lima-cidata/boot.sh
LIMA| Executing /mnt/lima-cidata/boot/00-modprobe.sh
Loading kernel module "fuse"
Loading kernel module "tun"
Loading kernel module "tap"
Loading kernel module "bridge"
Loading kernel module "veth"
Loading kernel module "ip_tables"
Loading kernel module "ip6_tables"
Loading kernel module "iptable_nat"
Loading kernel module "ip6table_nat"
Loading kernel module "iptable_filter"
Loading kernel module "ip6table_filter"
Loading kernel module "nf_tables"
Loading kernel module "x_tables"
Loading kernel module "xt_MASQUERADE"
Loading kernel module "xt_addrtype"
Loading kernel module "xt_comment"
Loading kernel module "xt_conntrack"
Loading kernel module "xt_mark"
Loading kernel module "xt_multiport"
Loading kernel module "xt_nat"
Loading kernel module "xt_tcpudp"
Loading kernel module "overlay"
LIMA| Executing /mnt/lima-cidata/boot/01-alpine-ash-as-bash.sh
LIMA| Executing /mnt/lima-cidata/boot/03-etc-hosts.sh
+ sed -i /host.lima.internal/d /etc/hosts
+ echo -e '192.168.5.2\thost.lima.internal'
LIMA| Executing /mnt/lima-cidata/boot/05-persistent-data-volume.sh
+ test -f /etc/alpine-release
+ DATADIRS='/etc /home /tmp /usr/local /var/lib'
+ awk '$2 == "/" {print $3}' /proc/mounts
+ '[' tmpfs '==' tmpfs ]
+ mkdir -p /mnt/data
+ '[' -e /dev/disk/by-label/data-volume ]
+ mount -t ext4 /dev/disk/by-label/data-volume /mnt/data
+ '[' -d /mnt/data/etc ]
+ mkdir -p /etc
+ mount --bind /mnt/data/etc /etc
+ '[' -d /mnt/data/home ]
+ mkdir -p /home
+ mount --bind /mnt/data/home /home
+ '[' -d /mnt/data/tmp ]
+ mkdir -p /tmp
+ mount --bind /mnt/data/tmp /tmp
+ '[' -d /mnt/data/usr/local ]
+ mkdir -p /usr/local
+ mount --bind /mnt/data/usr/local /usr/local
+ '[' -d /mnt/data/var/lib ]
+ mkdir -p /var/lib
+ mount --bind /mnt/data/var/lib /var/lib
LIMA| Executing /mnt/lima-cidata/boot/10-alpine-prep.sh
+ test -f /etc/alpine-release
+ BRANCH=edge
+ awk '-F=' '$1=="VERSION_ID" {print $2}' /etc/os-release
+ VERSION_ID=3.13.6
+ BRANCH=v3.13
+ URL=https://dl-cdn.alpinelinux.org/alpine/v3.13/main
+ grep -q '^https://dl-cdn.alpinelinux.org/alpine/v3.13/main$' /etc/apk/repositories
+ URL=https://dl-cdn.alpinelinux.org/alpine/v3.13/community
+ grep -q '^https://dl-cdn.alpinelinux.org/alpine/v3.13/community$' /etc/apk/repositories
+ usermod -p '*' bauerdic
+ sed -i 's/AllowTcpForwarding no/AllowTcpForwarding yes/g' /etc/ssh/sshd_config
+ rc-service --ifstarted sshd reload
+ rc-service cgroups start
 * Mounting cgroup filesystem ... [ ok ]
+ rc-update add acpid
 * rc-update: acpid already installed in runlevel `default'; skipping
+ rc-service acpid start
 * Starting busybox acpid ... [ ok ]
LIMA| Executing /mnt/lima-cidata/boot/20-rootless-base.sh
+ command -v systemctl
+ exit 0
LIMA| Executing /mnt/lima-cidata/boot/25-guestagent-base.sh
+ seq 0 1
+ mountpointvar=LIMA_CIDATA_MOUNTS_0_MOUNTPOINT
+ eval echo '$LIMA_CIDATA_MOUNTS_0_MOUNTPOINT'
+ echo /home/bauerdic
+ mountpoint=/home/bauerdic
+ mkdir -p /home/bauerdic
+ id -g bauerdic
+ gid=6170
+ chown 6170:6170 /home/bauerdic
+ mountpointvar=LIMA_CIDATA_MOUNTS_1_MOUNTPOINT
+ eval echo '$LIMA_CIDATA_MOUNTS_1_MOUNTPOINT'
+ echo /tmp/lima
+ mountpoint=/tmp/lima
+ mkdir -p /tmp/lima
+ id -g bauerdic
+ gid=6170
+ chown 6170:6170 /tmp/lima
+ install -m 755 /mnt/lima-cidata/lima-guestagent /usr/local/bin/lima-guestagent
+ '[' -f /etc/alpine-release ]
+ mkdir -p /run/user/6170
+ id -g bauerdic
+ gid=6170
+ chown 6170:6170 /run/user/6170
+ chmod 700 /run/user/6170
+ cat
+ chmod 755 /etc/init.d/lima-guestagent
+ rc-update add lima-guestagent default
 * rc-update: lima-guestagent already installed in runlevel `default'; skipping
+ rc-service lima-guestagent start
 * Caching service dependencies ... [ ok ]
 * Starting lima-guestagent ... [ ok ]
LIMA| Executing /mnt/lima-cidata/boot/30-install-packages.sh
+ command -v apt-get
+ command -v dnf
+ command -v pacman
+ command -v zypper
+ command -v apk
+ '[' 2 -gt 0 ]
+ command -v sshfs
+ update_fuse_conf
+ '[' 2 -gt 0 ]
+ grep -q ^user_allow_other /etc/fuse.conf
LIMA| Executing /mnt/lima-cidata/boot/40-install-containerd.sh
+ '['  '!=' 1 ]
+ '['  '!=' 1 ]
+ exit 0
LIMA| Exiting with code 0

jandubois · 2021-09-18T18:44:44Z

Your lima-init.log looks completely fine (same as on my machine, where it starts normally).

and finally

...
INFO[0587] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0597] [hostagent] Waiting for the essential requirement 1 of 4: "ssh" 
INFO[0597] [hostagent] Waiting for the essential requirement 2 of 4: "sshfs binary to be installed" 
FATA[0601] did not receive an event with the "running" status

From this it looks like it takes a very long time for ssh to respond for the first time, and then the host agent times out. Is it always the same that it goes to installing sshfs, and then aborts?

I don't really have any good idea why this would be happening with Alpine and Arch, but not with Ubuntu.

Did you ever try rebooting your Mac and checking if the problem persists beyond that?

LATBauerdick · 2021-09-18T21:40:46Z

I rebooted and also re-installed from scratch, no change. I’ll now try with another user account.

LATBauerdick · 2021-09-19T19:08:33Z

I think I now know what happened. When I installed alpine for some reason the DNS did not work (it worked just fine with the default ubuntu installation) and apk update etc would be unable to run. So I did a
echo "nameserver 8.8.8.8" >> /etc/resolv.conf and then apk update and apk add curl zsh xz and my nix setup just worked.

However, that screwed up the limactl start alpine, for some reason (I reproduced that on a different laptop).

I made a re-installation after a reboot of my Mac, and then DNS worked without having to add 8.8.8.8. Once I left out that addition to /etc/resolv the limactl start alpine works! I don't know why, but fine.

HOWEVER... (and now I feel completely stupid), each stop and start cycle completely wipes out my installation!! (again, that woks just fine with default):

❯ limactl start Dev/lima/examples/alpine.yaml
? Creating an instance "alpine" Proceed with the default configuration
...etc...
❯ ssh -p 60020 -i ~/.lima/_config/user -o NoHostAuthenticationForLocalhost=yes 127.0.0.1
Welcome to Alpine!
...etc...
lima-alpine:~$ su
lima-alpine:/home/bauerdic.linux# apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
std-3.13.5 [/media/cdrom/apks]
v3.13.6-19-g03009d53f7 [https://dl-cdn.alpinelinux.org/alpine/v3.13/main]
v3.13.6-21-g62a6abd4b7 [https://dl-cdn.alpinelinux.org/alpine/v3.13/community]
OK: 13890 distinct packages available
lima-alpine:/home/bauerdic.linux# apk add curl zsh xz
(1/8) Installing brotli-libs (1.0.9-r3)
(2/8) Installing nghttp2-libs (1.42.0-r1)
(3/8) Installing libcurl (7.79.0-r0)
(4/8) Installing curl (7.79.0-r0)
(5/8) Installing xz (5.2.5-r0)
(6/8) Installing zsh (5.8-r1)
Executing zsh-5.8-r1.post-install
(7/8) Installing apk-tools-zsh-completion (2.12.7-r0)
(8/8) Installing openrc-zsh-completion (0.42.1-r20)
Executing busybox-1.32.1-r6.trigger
OK: 41 MiB in 73 packages
lima-alpine:/home/bauerdic.linux# 
lima-alpine:~$ which curl
/usr/bin/curl
lima-alpine:~$ ls -l /usr/bin/curl
-rwxr-xr-x    1 root     root        239560 Sep 15 09:16 /usr/bin/curl
lima-alpine:~$ ^D
Connection to 127.0.0.1 closed.
❯ limactl stop alpine
INFO[0000] Sending SIGINT to hostagent process 20136    
...etc...
INFO[0004] [hostagent] QEMU has exited                  
❯ limactl start alpine
INFO[0000] Using the existing instance "alpine"         
...etc...
INFO[0020] READY. Run `limactl shell alpine` to open the shell. 
❯ ssh -p 60020 -i ~/.lima/_config/user -o NoHostAuthenticationForLocalhost=yes 127.0.0.1
Welcome to Alpine!
...etc...
lima-alpine:~$ which curl
lima-alpine:~$ ls -l /usr/bin/curl
ls: /usr/bin/curl: No such file or directory

what?? Anything else I install is gone, too. Any ideas what is going on here? I'm sure I am missing something completely obvious (maybe I need to read trough the alpine twiki? busybox, no idea what that is?), but your help is much appreciated!

jandubois · 2021-09-19T21:15:38Z

Any ideas what is going on here?

Yes, the examples/alpine.yaml config is running from an ISO image (like a live-dvd), so each reboot reverts to the pristine image. Lima moves some directories to a data volume (/etc, /home, /tmp, /usr/local, and /var/lib), so they will persist, but everything else is served from a tmpfs.

If you want to have persisted changes to the OS itself, then you should probably run from a normal disk image instead of an ISO (all other examples use regular disk images).

You could move additional directories to the data volume in a provisioning script, but which ones would you need? Just /usr, or also /bin, /lib, /sbin? Anything else? This feels brittle, so I think once you need more than /usr/local you should move to a regular disk image instead.

Alternatively you could build a custom "edition" of alpine-lima to include any extra packages you want to have installed.

I've created alpine-lima originally for Rancher Desktop, to have a small OS image that is bundled with the application, so it should be as small as possible, and not include any unnecessary software because the only update mechanism would be an update of the image itself when the app is updated.

Note how the std edition of alpine-lima is just 50MB whereas the Ubuntu image is 500MB. But just installing the ISO into a disk image will probably at least double the file size.

LATBauerdick · 2021-09-19T22:18:30Z

Thank you Jan for explaining this to me. I think I will experiment a bit and go the regular disk image way.

Thanks again!

LATBauerdick changed the title ~~stuck on "Waiting for the essential requirement 1 of X: "ssh" w~~ stuck on "Waiting for the essential requirement 1 of 4: "ssh" with non-Ubuntu and libslirp v4.6.1 Sep 17, 2021

AkihiroSuda added the status/more-info-needed label Sep 18, 2021

AkihiroSuda added the guest/alpine Guest: Alpine label Sep 18, 2021

jandubois added the guest/archlinux Guest: Arch Linux label Sep 18, 2021

LATBauerdick closed this as completed Sep 19, 2021

na0x2c6 mentioned this issue May 7, 2023

Stuck on "ssh" or "user session is ready for ssh" #1288

Open

wq9578 mentioned this issue May 1, 2024

[User name: admin] Waiting for the essential requirement 1 of 4: "ssh" #2117

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stuck on "Waiting for the essential requirement 1 of 4: "ssh" with non-Ubuntu and libslirp v4.6.1 #255

stuck on "Waiting for the essential requirement 1 of 4: "ssh" with non-Ubuntu and libslirp v4.6.1 #255

LATBauerdick commented Sep 17, 2021 •

edited

Loading

jandubois commented Sep 18, 2021

LATBauerdick commented Sep 18, 2021

jandubois commented Sep 18, 2021

LATBauerdick commented Sep 18, 2021

jandubois commented Sep 18, 2021

LATBauerdick commented Sep 18, 2021

jandubois commented Sep 18, 2021 •

edited

Loading

LATBauerdick commented Sep 18, 2021

LATBauerdick commented Sep 19, 2021

jandubois commented Sep 19, 2021 •

edited

Loading

LATBauerdick commented Sep 19, 2021

stuck on "Waiting for the essential requirement 1 of 4: "ssh" with non-Ubuntu and libslirp v4.6.1 #255

stuck on "Waiting for the essential requirement 1 of 4: "ssh" with non-Ubuntu and libslirp v4.6.1 #255

Comments

LATBauerdick commented Sep 17, 2021 • edited Loading

jandubois commented Sep 18, 2021

LATBauerdick commented Sep 18, 2021

jandubois commented Sep 18, 2021

LATBauerdick commented Sep 18, 2021

jandubois commented Sep 18, 2021

LATBauerdick commented Sep 18, 2021

jandubois commented Sep 18, 2021 • edited Loading

LATBauerdick commented Sep 18, 2021

LATBauerdick commented Sep 19, 2021

jandubois commented Sep 19, 2021 • edited Loading

LATBauerdick commented Sep 19, 2021

LATBauerdick commented Sep 17, 2021 •

edited

Loading

jandubois commented Sep 18, 2021 •

edited

Loading

jandubois commented Sep 19, 2021 •

edited

Loading