MacOS M1 Limactl start hangs on Waiting for essential requirement 3 of 5: sshfs binary to be installed (Could not connect to 127.0.0.1:48100 (127.0.0.1). - connect (111: Connection refused))

[richtong]

I have MacOS Monterey on an M1 Macbook, but I think the problem has something to do with my .ssh file, I get this warning, about a gpg key in .ssh not being in ssh format (why is Lima going through my .ssh I'm confused about that), but here is the output. I have confirmed I have 4.6.1 of the slirp and it get's past requirement 1 anyway, but hangs on the sshfs binary to be installed.

I'm a noob so confused about what magic lima is doing to get the first image bootstrapped.

limactl start

? Creating an instance "default" Proceed with the default configuration
INFO[0003] Attempting to download the image from "~/Downloads/impish-server-cloudimg-arm64.img" digest=
INFO[0003] Attempting to download the image from "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-arm64.img" digest=
INFO[0003] Using cache "/Users/rich/Library/Caches/lima/download/by-url-sha256/a9f81252e41821dac2357ea4c9b5a5a1c71526b41bc4473d6365fa3594b86dd9/data"
INFO[0003] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.15.0/nerdctl-full-0.15.0-linux-arm64.tar.gz" digest="sha256:dd8639ce868bab394467576f55375c4b40a8288badb579d2e30c3487da6004ea"
INFO[0003] Using cache "/Users/rich/Library/Caches/lima/download/by-url-sha256/e1229ceee50fc4392d95361c97a8746e6f6fbd872da89c050af58054d5730c56/data"
WARN[0003] [hostagent] public key "/Users/rich/.ssh/[email protected]" doesn't seem to be in ssh format
INFO[0004] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/rich/.lima/default/serial.log")
INFO[0004] SSH Local Port: 60022
INFO[0004] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0049] [hostagent] The essential requirement 1 of 5 is satisfied
INFO[0049] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0049] [hostagent] The essential requirement 2 of 5 is satisfied
INFO[0049] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0090] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"

[AkihiroSuda]

Please provide :

• /Users/rich/.lima/default/serial.log (host)
• /var/log/cloud-init*.log (guest)

[richtong]

Yes Akihiro-san, here they are. Happy New Year btw, it looks like the download of sshfs has failed, when I do a lima to get into the VM, it says /User/rich not found, so I don't think the sshfs worked...Here are the debug1G files

serial.log
cloud-init.log
cloud-init-output.log

[AkihiroSuda]

Err:1 http://ports.ubuntu.com/ubuntu-ports impish InRelease
Could not connect to 127.0.0.1:48100 (127.0.0.1). - connect (111: Connection refused)

Looks like a proxy issue
Cc @jandubois

[richtong]

Thanks much @AkihiroSuda I did go into the lima default and there does seem to be a problem with the networking. I can't find the documentation on how this is supposed to be set up by default, but getting into the default container with the limactl command and I tried:

curl https://www.google.com
curl: (7) Failed to connect to 127.0.0.1 port 48100: Connection refused

Looking at the network with

ip route
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 100
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15
192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 100

I can see it has a network at 192.168.5.15

ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:60:91:3d brd ff:ff:ff:ff:ff:ff
    altname enp0s1
    inet 192.168.5.15/24 brd 192.168.5.255 scope global dynamic eth0
       valid_lft 85828sec preferred_lft 85828sec
    inet6 fec0::5055:55ff:fe60:913d/64 scope site dynamic mngtmpaddr noprefixroute
       valid_lft 86255sec preferred_lft 14255sec
    inet6 fe80::5055:55ff:fe60:913d/64 scope link
       valid_lft forever preferred_lft forever

Which all seems correct given https://github.com/lima-vm/lima/blob/master/docs/network.md

[jandubois]

It certainly looks like some proxy interference, but Lima is supposed to rewrite all proxy reference to localhost and 127.0.0.1 to use the SLIRP gateway address instead...

@richtong Can you check if any proxy environment variables are set?

$ env | grep -i proxy | sort
$ cat /etc/environment

Also, are you aware of a proxy running on your host on port 48100?

[jandubois]

why is Lima going through my .ssh I'm confused about that

Lima adds all of your public keys to the authorized keys file inside the VM, so you can connect to it with any of them. This is typically not needed, and you can disable this by setting ssh.loadDotSSHPubKeys to false in the lima.yaml file.

[richtong]

Got it @jandubois ok, your point about a proxy on 48100 was well taken and make me think that I should try a reboot, which worked, it started successfully. May need to put this into the README.md somewhere that after an installation a reboot is needed. I'm not sure why.

There was nothing on port there judging from lsof in either guest or in MacOS before the reboot, but env definitely had proxy set.

However on reboot the two commands came out very differently, so in the lima guest there is no proxy set so env | grep proxy returns null.

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
#LIMA-START
#LIMA-END

[jandubois]

However on reboot the two commands came out very differently,

Can you share how they looked before your reboot?

Assuming you did have proxy settings pointing to 127.0.0.1, I still don't understand why they wouldn't get translated.

[jandubois]

Assuming you did have proxy settings pointing to 127.0.0.1, I still don't understand why they wouldn't get translated.

I just took another look at the code, and the replacement logic is only applied to values from the host environment, but not from to the system proxy settings (nor the lima.yaml settings from env). So that could explains this, but only if you did have a local proxy configured before in your network settings.

I'll fix the code to always apply the translation, but I would still really like to understand where these proxy settings came from.

[m-housh]

For me, I think that the new Apple Private Relay feature was causing problems as I was also getting a hang-up and proxy issues that were resolved when I turned private relay off on my computer. FWIW.

[richtong]

yes they were pointed at 127.0.0.1:48100

[richtong]

Ah good point, I didn't check the private relay stuff.

[jandubois]

@m-housh Were you using both private relay and a proxy at the same time?

I've enabled just "private proxy", and everything continues to work fine for me.

[jandubois]

yes they were pointed at 127.0.0.1:48100

@richtong In that case I assume this setting came from your system proxy settings, which Lima didn't properly translate. This should be fixed by #508, so I'm going to close this issue.

@m-housh If you have a reproducible failure with "private relay" please open a separate issue for that, as it seems independent of this one.

[richtong]

OK here is what I see, it has to do with if NordVPN is running when lima start is running

1. Private relay seems to work whether enabled or not. I got it to successfully run.
2. If NordVPN is running (it doesn't have to have a connection engaged), then it hangs on that step 3 of 5 and SSHFS. Not sure I understand MacOS networking well enough to know why, but you might want to put a note about at least that particular VPN just on installation. Once it is installed it seems to work fine either running or connected.

So the last time it worked for me, I didn't notice this as I usually have NordVPN running in the background at startup.

[jandubois]

@richtong I cannot reproduce this. Can you explain exactly what you do to run NordVPN?

I've installed the latest NordVPN app from the App store, ran it, and did a "Quick Connect". I can still create a new default VM with Lima while the VPN is running...

[jandubois]

Going to re-open the issue until we understand the failure mode with NordVPN.

[fantasywind]

It seems Surge VPN making network refusing too.

LAN routing in colima ssh environment

[richtong]

Yes I'm not quite sure on NordVPN or why it's not reproducible. When I look at the nord installation i don't see anything obvious but I have seen that it does cause certain other things to fail. If you let me know what I should look for I will.

[snej]

I’m getting the same problem, but I don’t have Private Relay or any VPN or proxy. Also an M1 Mac. I had installed lima early this year and played with it a bit, but then didn’t use it until yesterday. Today I decided to brew upgrade lima and then factory-reset my instance. After that, it won’t start; limactl start has the same problem described above, keeps logging Waiting for the essential requirement 3 of 5: "sshfs binary to be installed”.

I’m on macOS Ventura 13.0; any known compatibility problems?

[AkihiroSuda]

@snej
The logs may contain some hint

• $HOME/.lima/<INSTANCE>/serial.log
• /var/log/cloud-init-output.log (inside the guest)

[snej]

@AkihiroSuda I didn’t see any errors in the serial.log file. It ends with a login prompt; is that normal?
serial.log

I don’t know how to look at files in the guest without starting it, and I can’t start it.

[AkihiroSuda]

I don’t know how to look at files in the guest without starting it, and I can’t start it.

You can just run limactl shell <INSTANCE> in another terminal while booting it (if the sshd in the guest is working)

[snej]

Interestingly, the cloud-init-output.log shows the same error I was getting in my previous instance that caused me to try a factory reset in the first place — apt-get fails:

…
+ apt-get update
Ign:1 http://ports.ubuntu.com/ubuntu-ports impish InRelease
Ign:2 http://ports.ubuntu.com/ubuntu-ports impish-updates InRelease
Ign:3 http://ports.ubuntu.com/ubuntu-ports impish-backports InRelease
Ign:4 http://ports.ubuntu.com/ubuntu-ports impish-security InRelease
Err:5 http://ports.ubuntu.com/ubuntu-ports impish Release
  404  Not Found [IP: 185.125.190.36 80]
Err:6 http://ports.ubuntu.com/ubuntu-ports impish-updates Release
  404  Not Found [IP: 185.125.190.36 80]
Err:7 http://ports.ubuntu.com/ubuntu-ports impish-backports Release
  404  Not Found [IP: 185.125.190.36 80]
Err:8 http://ports.ubuntu.com/ubuntu-ports impish-security Release
  404  Not Found [IP: 185.125.190.36 80]
Reading package lists...
E: The repository 'http://ports.ubuntu.com/ubuntu-ports impish Release' no longer has a Release file.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports impish-updates Release' no longer has a Release file.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports impish-backports Release' does not have a Release file.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports impish-security Release' no longer has a Release file.
LIMA| WARNING: Failed to execute /mnt/lima-cidata/boot/30-install-packages.sh
…

cloud-init-output.log

[snej]

Maybe this has something to do with it? Ubuntu 21.10 (Impish Indri) reaches End of Life on July 14 2022

Is lima hardcoded to use this release or is there a way to update to a newer one?

OK, by deleting the instance and starting a new one I’ve picked up the current template, which uses release 22.10. It’s still downloading, but I assume it won’t have this issue. [Update: it successfully started. Whew!]

Not sure what the best solution for this is. Scan the log for this error and report it to the user? Preflight ubuntu-ports to make sure the distro is still supported?