NetBSD - VM doesn't start after a 120s timeout

[kobalicek]

I'm having the following occasional issue when running NetBSD runner:

  Pseudo-terminal will not be allocated because stdin is not a terminal.
  ssh: connect to host localhost port 2847: Connection refused
  Waiting for VM to be ready...
  Executing command inside VM: true
  /usr/bin/ssh -t runner@localhost
  Pseudo-terminal will not be allocated because stdin is not a terminal.
  ssh: connect to host localhost port 2847: Connection refused
  Waiting for VM to be ready...
  Executing command inside VM: true
  /usr/bin/ssh -t runner@localhost
  Pseudo-terminal will not be allocated because stdin is not a terminal.
  ssh: connect to host localhost port 2847: Connection refused
  Waiting for VM to be ready...
  Executing command inside VM: true
  /usr/bin/ssh -t runner@localhost
  Pseudo-terminal will not be allocated because stdin is not a terminal.
  ssh: connect to host localhost port 2847: Connection refused
  Waiting for VM to be ready...
  Executing command inside VM: true
  /usr/bin/ssh -t runner@localhost
  Pseudo-terminal will not be allocated because stdin is not a terminal.
  ssh: connect to host localhost port 2847: Connection refused
  Terminating VM
  /usr/bin/sudo kill -s TERM 1370
  kill: 1370: No such process
Error: Waiting for VM to become ready timed out after 120 seconds

I'm using QEMU to run it.

Basically the VM is not ready after 120 seconds, which causes the action to be terminated.

I'm not sure what is the problem in this case - if the GHA runner is simply overloaded or whether there is a race or something caused by the action itself, which results in inability to connect to the SSH server inside the VM.

I'm wondering - is this something we have to live with or do you think that this can be fixed somehow? It's very hard to diagnose as it doesn't happen every time, but it happens frequently enough to have my attention.

[jacob-carlborg]

Yeah, it's difficult to say. Could be both something inside the VM and something outside. Perhaps it's possible to run through DTrace to debug it. Not sure if that works on a GHA runner though. Perhaps it's possible to redirect the output of the VM to some file and print that, to see what's going on.

[jacob-carlborg]

Do you have a link to a failing job?

[kobalicek]

I have - actually two failing jobs within 2 days:

• https://github.com/asmjit/asmjit/actions/runs/6272224096/job/17033373759
• https://github.com/asmjit/asmjit/actions/runs/6284915251/job/17066768332

I'm not sure that would help though, as nothing interesting happens in these runs, it just stops at the beginning.

[manxorist]

I am seeing the same problem:

https://github.com/OpenMPT/openmpt/blob/ea8aafbdcbf07b1d2a96a0d213edb64e7872f6ae/.github/workflows/NetBSD-9.3-Makefile.yml

and a couple of failing jobs:

• https://github.com/OpenMPT/openmpt/actions/runs/6282259563
• https://github.com/OpenMPT/openmpt/actions/runs/6282412664
• https://github.com/OpenMPT/openmpt/actions/runs/6282449434
• https://github.com/OpenMPT/openmpt/actions/runs/6288164462
• https://github.com/OpenMPT/openmpt/actions/runs/6288329988
• https://github.com/OpenMPT/openmpt/actions/runs/6288995972
• https://github.com/OpenMPT/openmpt/actions/runs/6314983647
• https://github.com/OpenMPT/openmpt/actions/runs/6317196952
• https://github.com/OpenMPT/openmpt/actions/runs/6321362086

The last successful NetBSD VM run was on 2023-09-21, https://github.com/OpenMPT/openmpt/actions/runs/6264853726 .

[kobalicek]

And one more:

• https://github.com/asmjit/asmjit/actions/runs/6328512506/job/17186877536

I think that this is the most unstable runner at the moment - it fails in like 50% of time like this

[jacob-carlborg]

it fails in like 50% of time like this

Oh, that's pretty bad. I'll see if I can debug the issue.

[jacob-carlborg]

Seems like GitHub made some breaking changes again. This happens when trying to run QEMU:

dyld[1372]: Library not loaded: '/usr/local/opt/capstone/lib/libcapstone.4.dylib'

But it should always fail.

This makes it much easier to fix. I thought all the dependencies were statically linked to avoid this exact problem, but it looks like I missed one.

BTW, this is not specific to NetBSD, it applies for all platforms when QEMU is used as the hypervisor. But since xhyve if the default hypervisor for FreeBSD and OpenBSD on macOS runners it doesn't affect those platforms unless explicitly switching hypervisor to QEMU.

If you're in a hurry you can switch to using Linux runners instead of macOS as a workaround, but macOS has better performance.

[kobalicek]

Yeah it always fails.

I'm removing netbsd from my CI as this just makes all builds to fail.

I think this is just really unfortunate reality that it's not natively supported by github.

[jacob-carlborg]

Fixed in https://github.com/cross-platform-actions/action/releases/tag/v0.19.1. I've added a test to make sure this doesn't happen again. In doing that I also found another non-system dependency. But that is fixed as well.