lima-bin: init binary derivation at 0.14.2 #209171

tricktron · 2023-01-05T14:11:47Z

Description of changes

Adds lima binary derivation.

Why?

Lima added support for Apple's Virtualization.framework aka vz in 0.14.0 which needs the apple sdk 13 to build it from source. In nixpkgs, we are currently on sdk 11 and I tried to make it work with sdk11 but failed. See #206285 (comment).

Instead, it makes more sense to provide the lima binary, which this pr does.

Things done

roblabla · 2023-01-05T15:13:04Z

Should colima be updated to depend on lima-bin? It currently pulls lima, which doesn't support vz.

tricktron · 2023-01-05T15:13:56Z

@roblabla Yes, that is my plan.

dhess · 2023-01-05T15:24:01Z

Should colima be updated to depend on lima-bin? It currently pulls lima, which doesn't support vz.

Probably only on aarch64-darwin and x86_64-darwin, though. There's no reason for Linux users to prefer the binary version of lima.

tricktron · 2023-01-05T16:23:27Z

Probably only on aarch64-darwin and x86_64-darwin, though. There's no reason for Linux users to prefer the binary version of lima.

Good point. So should I then only provide darwin as binaries?

jbgosselin · 2023-01-05T17:02:31Z

I think we can provide all available platforms as binaries. If users want to use them on linux they could override.

tricktron · 2023-01-05T17:48:31Z

@dennajort @roblabla @dhess
The pr is now ready for review.

roblabla · 2023-01-09T14:58:02Z

lima is complaining to not have the vz entitlements:

{"level":"fatal","msg":"Error Domain=VZErrorDomain Code=2 Description=\"Invalid virtual machine configuration. The process doesn’t have the “com.apple.security.virtualization” entitlement.\" UserInfo={\n    NSLocalizedFailure = \"Invalid virtual machine configuration.\";\n    NSLocalizedFailureReason = \"The process doesn\\U2019t have the \\U201ccom.apple.security.virtualization\\U201d entitlement.\";\n}","time":"2023-01-09T15:56:52+01:00"}

dhess · 2023-01-09T15:07:14Z

Are you using it via colima? If so, you need to replace the lima derivation that colima is using, as well.

I've ginned something up here in our overlay, and it works with vz in colima, which I've verified by creating --vmtype vz containers with colima 0.5.2:

https://github.com/hackworthltd/hacknix/blob/7bfc5c68c04f81f7cdef608eda668e1ed55832c1/nix/overlays/200-lima.nix

tricktron · 2023-01-09T15:09:55Z

lima is complaining to not have the vz entitlements:

{"level":"fatal","msg":"Error Domain=VZErrorDomain Code=2 Description=\"Invalid virtual machine configuration. The process doesn’t have the “com.apple.security.virtualization” entitlement.\" UserInfo={\n    NSLocalizedFailure = \"Invalid virtual machine configuration.\";\n    NSLocalizedFailureReason = \"The process doesn\\U2019t have the \\U201ccom.apple.security.virtualization\\U201d entitlement.\";\n}","time":"2023-01-09T15:56:52+01:00"}

@roblabla What command did you run?

roblabla · 2023-01-09T15:11:54Z

@roblabla What command did you run?

limactl start colima

I think I've found the source of the problem. If I add dontStrip = true; to the derivation, it works. I suppose strip is removing the entitlements from the binary.

tricktron · 2023-01-09T15:16:04Z

@roblabla I cannot reproduce this. Works on my machine:

result/bin/limactl start colima
INFO[0000] Using the existing instance "colima"
INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/tricktron/.lima/colima/serial.log")
INFO[0000] SSH Local Port: 61892
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0010] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0020] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0020] [hostagent] The essential requirement 1 of 5 is satisfied
INFO[0020] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0020] [hostagent] The essential requirement 2 of 5 is satisfied
INFO[0020] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0020] [hostagent] The essential requirement 3 of 5 is satisfied
INFO[0020] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
INFO[0020] [hostagent] The essential requirement 4 of 5 is satisfied
INFO[0020] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running"
INFO[0020] [hostagent] The essential requirement 5 of 5 is satisfied
INFO[0020] [hostagent] Mounting "/Users/tricktron" on "/Users/tricktron"
INFO[0020] [hostagent] Mounting "/tmp/colima" on "/tmp/colima"
INFO[0020] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0020] [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/tricktron/.colima/default/docker.sock" (host)
INFO[0020] [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/tricktron/.colima/docker.sock" (host)
INFO[0020] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/tricktron/.lima/colima/ga.sock" (host)
INFO[0021] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0021] READY. Run `limactl shell colima` to open the shell.

roblabla · 2023-01-09T15:20:26Z

So I checked the .limactl-wrapped binary, and indeed, I can see that there are no entitlements embedded in the binary:

$ codesign -d --entitlements :- result-nostrip/bin/.limactl-wrapped
Executable=/nix/store/xhi9zmhwqjsjghb9dafajd2igx39rk4p-lima-0.14.2/bin/.limactl-wrapped
Warning: Specifying ':' in the path is deprecated and will not work in a future release
$ codesign -d --entitlements :- result/bin/.limactl-wrapped
Executable=/nix/store/ridj9qx1y39i9m34dzgr0wxp1zws7qgk-lima-0.14.2/bin/.limactl-wrapped
Warning: Specifying ':' in the path is deprecated and will not work in a future release
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.network.client</key><true/><key>com.apple.security.network.server</key><true/><key>com.apple.security.virtualization</key><true/></dict></plist>

The former was generated using nix build .#lima-bin, while the later was generated similarly, but adding dontStrip = true to the derivation. Both were done on the tip of this PR (d5e7c6c6bed9db97323a90361646d1d28a85bff2).

I'm running this on a macos13 installation. Not sure what I can do to find what's different between our environments.

roblabla · 2023-01-09T15:25:39Z

@tricktron your VM is running qemu

INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/tricktron/.lima/colima/serial.log")

That works on my end too. But I'm trying to get a VZ VM to run, which requires the entitlement. You have to set vmType: vz in the configuration YAML (~/.lima/default/default.yaml)

roblabla · 2023-01-09T15:31:00Z

for an easier reproducer: limactl start --name vz template://experimental/vz should create a vz-based VM, and fail:

$ limactl start --name vz template://experimental/vz
? Creating an instance "vz" Proceed with the current configuration
INFO[0001] Attempting to download the image from "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"  digest=
613.88 MiB / 613.88 MiB [----------------------------------] 100.00% 51.60 MiB/s
INFO[0013] Downloaded the image from "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
INFO[0016] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v1.1.0/nerdctl-full-1.1.0-linux-arm64.tar.gz"  digest="sha256:3b613a1be5a24460c44bb93a3609b790ada94e06efd1a86467d45bec7da8b449"
190.14 MiB / 190.14 MiB [----------------------------------] 100.00% 15.34 MiB/s
INFO[0029] Downloaded the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v1.1.0/nerdctl-full-1.1.0-linux-arm64.tar.gz"
INFO[0030] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/roblabla/.lima/vz/serial.log")
INFO[0030] [hostagent] Setting up Rosetta share
FATA[0031] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/roblabla/.lima/vz/ha.stderr.log")

And here's the ha.stderr.log:

$ cat ~/.lima/vz/ha.stderr.log
{"level":"debug","msg":"Creating iso file /Users/roblabla/.lima/vz/cidata.iso","time":"2023-01-09T16:30:21+01:00"}
{"level":"debug","msg":"Using /var/folders/p7/cywn8zl53tn4_97kfys4lxc00000gn/T/diskfs_iso2028567672 as workspace","time":"2023-01-09T16:30:21+01:00"}
{"level":"debug","msg":"OpenSSH version 9.1.1 detected","time":"2023-01-09T16:30:21+01:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing [email protected] and [email protected]","time":"2023-01-09T16:30:21+01:00"}
{"level":"info","msg":"Starting VZ (hint: to watch the boot progress, see \"/Users/roblabla/.lima/vz/serial.log\")","time":"2023-01-09T16:30:21+01:00"}
{"level":"debug","msg":"Start tcp server listening on: 127.0.0.1:49858","time":"2023-01-09T16:30:21+01:00"}
{"level":"debug","msg":"Start udp server listening on: 127.0.0.1:49693","time":"2023-01-09T16:30:21+01:00"}
{"level":"info","msg":"Setting up Rosetta share","time":"2023-01-09T16:30:21+01:00"}
{"level":"fatal","msg":"Error Domain=VZErrorDomain Code=2 Description=\"Invalid virtual machine configuration. The process doesn’t have the “com.apple.security.virtualization” entitlement.\" UserInfo={\n    NSLocalizedFailure = \"Invalid virtual machine configuration.\";\n    NSLocalizedFailureReason = \"The process doesn\\U2019t have the \\U201ccom.apple.security.virtualization\\U201d entitlement.\";\n}","time":"2023-01-09T16:30:21+01:00"}

tricktron · 2023-01-09T15:39:19Z

@roblabla Great catch. I could reproduce it and disabled the stripping to fix it.

Thank you for the precise testing!

tricktron · 2023-01-09T16:03:09Z

Maybe we should only disable stripping on darwin because the entitlements are only in the darwin binaries?

jbgosselin · 2023-01-09T16:22:55Z

This is what I added on my PR for the non binary version of lima https://github.com/NixOS/nixpkgs/pull/206285/files#diff-adecdcf421866f9b6153fa6fa3608ee1e62ef5a5518ff91ef7fecf4a8908ce23R37

tricktron · 2023-01-09T16:53:24Z

This is what I added on my PR for the non binary version of lima https://github.com/NixOS/nixpkgs/pull/206285/files#diff-adecdcf421866f9b6153fa6fa3608ee1e62ef5a5518ff91ef7fecf4a8908ce23R37

Great stuff. Thanks for the hint!

jbgosselin · 2023-01-25T19:00:13Z

@tricktron you have merge conflicts (maybe due to my changes on lima that are now merged).

tricktron · 2023-01-26T08:48:04Z

@dennajort Thanks for the heads-up. I resolved the merge conflict.

nixos-discourse · 2023-01-26T23:16:24Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/815

pecigonzalo · 2023-01-27T15:53:37Z

Awesome! We should update Colima to reference this version as well.

tricktron · 2023-01-27T16:20:02Z

@pecigonzalo

Awesome! We should update Colima to reference this version as well.

Here you go: #212980.

tricktron mentioned this pull request Jan 5, 2023

lima: 0.13.0 -> 0.14.2 #206285

Merged

13 tasks

ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Jan 5, 2023

tricktron requested a review from SuperSandro2000 January 9, 2023 14:43

roblabla approved these changes Jan 9, 2023

View reviewed changes

tricktron force-pushed the f-lima-bin branch from e811c1a to 0b3e311 Compare January 9, 2023 16:49

tricktron added 2 commits January 26, 2023 09:46

lima-bin: init binary derivation at 0.14.2

ff22aff

lima-bin: use makeBinaryWrapper instead of makeWrapper

8ddef23

tricktron added 2 commits January 26, 2023 09:46

lima-bin: use autoPatchelfHook to fix build on linux

4d427f4

lima-bin: do not strip binary on darwin to keep entitlements

ece8915

tricktron force-pushed the f-lima-bin branch from 0b3e311 to ece8915 Compare January 26, 2023 08:47

SuperSandro2000 merged commit 379895e into NixOS:master Jan 27, 2023

xanderio pushed a commit to xanderio/nixpkgs that referenced this pull request Feb 13, 2023

lima-bin: init binary derivation at 0.14.2 (NixOS#209171)

b75176d

tricktron mentioned this pull request Jan 11, 2024

lima-bin: 0.19.0 -> 0.19.1 #279326

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lima-bin: init binary derivation at 0.14.2 #209171

lima-bin: init binary derivation at 0.14.2 #209171

tricktron commented Jan 5, 2023 •

edited

Loading

roblabla commented Jan 5, 2023 •

edited

Loading

tricktron commented Jan 5, 2023

dhess commented Jan 5, 2023

tricktron commented Jan 5, 2023

jbgosselin commented Jan 5, 2023

tricktron commented Jan 5, 2023

roblabla commented Jan 9, 2023

dhess commented Jan 9, 2023

tricktron commented Jan 9, 2023

roblabla commented Jan 9, 2023

tricktron commented Jan 9, 2023

roblabla commented Jan 9, 2023 •

edited

Loading

roblabla commented Jan 9, 2023 •

edited

Loading

roblabla commented Jan 9, 2023 •

edited

Loading

tricktron commented Jan 9, 2023

tricktron commented Jan 9, 2023

jbgosselin commented Jan 9, 2023

tricktron commented Jan 9, 2023

jbgosselin commented Jan 25, 2023

tricktron commented Jan 26, 2023

nixos-discourse commented Jan 26, 2023

pecigonzalo commented Jan 27, 2023

tricktron commented Jan 27, 2023

lima-bin: init binary derivation at 0.14.2 #209171

lima-bin: init binary derivation at 0.14.2 #209171

Conversation

tricktron commented Jan 5, 2023 • edited Loading

Description of changes

Why?

Things done

roblabla commented Jan 5, 2023 • edited Loading

tricktron commented Jan 5, 2023

dhess commented Jan 5, 2023

tricktron commented Jan 5, 2023

jbgosselin commented Jan 5, 2023

tricktron commented Jan 5, 2023

roblabla commented Jan 9, 2023

dhess commented Jan 9, 2023

tricktron commented Jan 9, 2023

roblabla commented Jan 9, 2023

tricktron commented Jan 9, 2023

roblabla commented Jan 9, 2023 • edited Loading

roblabla commented Jan 9, 2023 • edited Loading

roblabla commented Jan 9, 2023 • edited Loading

tricktron commented Jan 9, 2023

tricktron commented Jan 9, 2023

jbgosselin commented Jan 9, 2023

tricktron commented Jan 9, 2023

jbgosselin commented Jan 25, 2023

tricktron commented Jan 26, 2023

nixos-discourse commented Jan 26, 2023

pecigonzalo commented Jan 27, 2023

tricktron commented Jan 27, 2023

tricktron commented Jan 5, 2023 •

edited

Loading

roblabla commented Jan 5, 2023 •

edited

Loading

roblabla commented Jan 9, 2023 •

edited

Loading

roblabla commented Jan 9, 2023 •

edited

Loading

roblabla commented Jan 9, 2023 •

edited

Loading