lima: 0.13.0 -> 0.14.2 #206285

jbgosselin · 2022-12-15T18:39:46Z

Description of changes

Updates performed

Golang update
Added dependency to xcrun on darwin to detect if can compile with Virtualization.framework
Added dependency to sigtool on darwin for needed codesign command
Modify Makefile to --force codesign due to golang signing aarch64-darwin binaries by default

To inspect upstream changes

Things done

deejayem · 2022-12-16T12:08:50Z

Result of nixpkgs-review pr 206285 run on x86_64-darwin 1

2 packages built:

colima
lima

deejayem

Built and tested successfully (via colima and docker/docker-compose) on x86_64-darwin.

But ofborg says lima.passthru.tests on aarch64-darwin failed:

> building
> mkdir -p _output/bin
> cp -a ./cmd/lima _output/bin/lima
> # The hostagent must be compiled with CGO_ENABLED=1 so that net.LookupIP() in the DNS server
> # calls the native resolver library and not the simplistic version in the Go library.
> CGO_ENABLED=1 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v0.14.1" -tags "no_vz" -o _output/bin/limactl ./cmd/limactl
> codesign --entitlements vz.entitlements -s - ./_output/bin/limactl
> libc++abi: terminating with uncaught exception of type std::runtime_error: file is already signed. pass -f to sign regardless.
> make: *** [Makefile:182: codesign] Abort trap: 6

jbgosselin · 2022-12-16T16:14:49Z

Built and tested successfully (via colima and docker/docker-compose) on x86_64-darwin.

But ofborg says lima.passthru.tests on aarch64-darwin failed:

> building
> mkdir -p _output/bin
> cp -a ./cmd/lima _output/bin/lima
> # The hostagent must be compiled with CGO_ENABLED=1 so that net.LookupIP() in the DNS server
> # calls the native resolver library and not the simplistic version in the Go library.
> CGO_ENABLED=1 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v0.14.1" -tags "no_vz" -o _output/bin/limactl ./cmd/limactl
> codesign --entitlements vz.entitlements -s - ./_output/bin/limactl
> libc++abi: terminating with uncaught exception of type std::runtime_error: file is already signed. pass -f to sign regardless.
> make: *** [Makefile:182: codesign] Abort trap: 6

I have a little feeling that it may be because the binary was already built and not cleaned up or something like that. I added a rm -rf _output and will try again.

jbgosselin · 2022-12-16T17:10:44Z

Actually what happens is that go binary built on aarch64-darwin requires to be signed. I assume that it signs them by default. But we do want to sign them with vz.entitlements so I modified the Makefile to --force the signature.

pkgs/applications/virtualization/lima/default.nix

Co-authored-by: Sandro <[email protected]>

tricktron

Builds and works on my aarch64-darwin machine.

Until fixed upstream. Source: NixOS/nixpkgs#206285

dhess · 2023-01-01T17:44:01Z

Thanks for this.

vz appears not to be supported:

limactl start template://experimental/vz                                                                                                                                             ~
? Creating an instance "vz" Proceed with the current configuration
FATA[0005] vm driver 'vz' needs macOS 13 or later (Hint: try recompiling Lima if you are seeing this error on macOS 13)

I assume this is because nixpkgs doesn't use the macOS 13 SDK? All of the required entitlements are present on the executable, and I'm running (and building on) macOS 13.1, but I see that lima's Makefile disables it when the SDK is too old:

https://github.com/lima-vm/lima/blob/baca0e7ecd1b3515f471d2eb60fdbb0864e9a487/Makefile#L19

jbgosselin · 2023-01-01T23:47:46Z

Thanks for this.

vz appears not to be supported:
limactl start template://experimental/vz                                                                                                                                             ~
? Creating an instance "vz" Proceed with the current configuration
FATA[0005] vm driver 'vz' needs macOS 13 or later (Hint: try recompiling Lima if you are seeing this error on macOS 13) 
I assume this is because nixpkgs doesn't use the macOS 13 SDK? All of the required entitlements are present on the executable, and I'm running (and building on) macOS 13.1, but I see that lima's Makefile disables it when the SDK is too old:

https://github.com/lima-vm/lima/blob/baca0e7ecd1b3515f471d2eb60fdbb0864e9a487/Makefile#L19

Yeah you're right, looks like xcrun is only running at 10.12 whatever the version of macos installed. I don't know how to tell nix to use a more up to date version of the SDK if available on the host machine.

jonathanlking · 2023-01-03T21:22:45Z

I'm not that familiar with nix, and especially not with darwin/macOS, but I wanted to share what I tried just in case it's helpful.

I was naively hoping that an overlay like:

(final: prev: {
    lima = prev.lima.override (old: { xcbuild = old.xcbuild.override(_: { sdkVer = "13.0"; }); });
  })

would get things building, but unfortunately it errors with:

building
mkdir -p _output/bin
cp -a ./cmd/lima _output/bin/lima
# The hostagent must be compiled with CGO_ENABLED=1 so that net.LookupIP() in the DNS server
# calls the native resolver library and not the simplistic version in the Go library.
CGO_ENABLED=1 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v0.14.2" -tags "" -o _output/bin/limactl ./cmd/limactl
# github.com/Code-Hex/vz/v3
In file included from vendor/github.com/Code-Hex/vz/v3/audio.go:6:
In file included from ./virtualization_11.h:9:
./virtualization_helper.h:25:9: warning: macOS 12.3 API has been disabled [-W#pragma-messages]
#pragma message("macOS 12.3 API has been disabled")
        ^
./virtualization_helper.h:32:9: warning: macOS 13 API has been disabled [-W#pragma-messages]
#pragma message("macOS 13 API has been disabled")
        ^
In file included from vendor/github.com/Code-Hex/vz/v3/audio.go:6:
./virtualization_11.h:10:9: fatal error: 'Virtualization/Virtualization.h' file not found
#import <Virtualization/Virtualization.h>
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings and 1 error generated.
make: *** [Makefile:94: _output/bin/limactl] Error 2

I think lima is pulling in vz, which the lima derivation is building too?

virtualization_helper.h checks the value of __MAC_OS_X_VERSION_MAX_ALLOWED, so this might need to be set/overridden too?

jbgosselin · 2023-01-04T19:30:36Z

Thanks for the help @jonathanlking will look into the way you did if I can make it work.

I'm also a bit concerned how to implement this to be available for users using nix on machine prior to 13.0 as well as after.

dhess · 2023-01-04T19:35:48Z

lima provides binaries, and this may be a case where it makes sense to use them for macOS. nixpkgs could provide a lima-binary package so that the user has a choice about which one to use.

jbgosselin · 2023-01-04T19:37:46Z

lima provides binaries, and this may be a case where it makes sense to use them for macOS. nixpkgs could provide a lima-binary package so that the user has a choice about which one to use.

Could be an idea also.
I'm honestly more concern over having it breaks on users using a version of macos prior to 13.0.

roblabla · 2023-01-04T21:31:07Z

Using 13.x SDK should work even on older version of macos. It only makes new APIs available, but if the program doesn't call them (or only calls them after checking the host os version), it shouldn't cause issues. Lima uses vz to access the Virtualization API, and it has this kind of doc: https://github.com/Code-Hex/vz/blob/957dfd2add9a0badf25dee8cd1dc01a79e8fee06/bootloader.go#L137

// NewEFIBootLoader creates a new EFI boot loader.
//
// This is only supported on macOS 13 and newer, error will
// be returned on older versions.

Of course, vz-based VMs won't work on macos12 and before, but the qemu-based driver should still work without issues.

jbgosselin · 2023-01-05T00:39:59Z

@roblabla great insight, thanks !

jbgosselin · 2023-01-05T00:54:22Z

#206285 (comment)
@jonathanlking did you try this on a machine running macos Ventura ?

jonathanlking · 2023-01-05T01:21:12Z

Hey, I did a tiny bit more exploring earlier today (again, please take what I'm saying with extreme caution!).

Something concerning me was the fatal error: 'Virtualization/Virtualization.h' file not found error.

I think prior to lima-vm/lima@2679326 there was no dependency on macOS frameworks? (I'm not super confident about this claim). I think you might need to use a different callPackage to make them available (if this isn't already happening), I got that impression from reading:

To use it, reference apple_sdk_11_0 instead of apple_sdk in your derivation and use pkgs.darwin.apple_sdk_11_0.callPackage instead of pkgs.callPackage. On Linux, this will have the same effect as pkgs.callPackage, so you can use pkgs.darwin.apple_sdk_11_0.callPackage regardless of platform.

on #176661.

Anyway, there is certainly no Virtualization in the pkgs.darwin.apple_sdk_11_0, which makes sense as the framework wasn't yet introduced, so I think it's something we'd need to patch/introduce?
I think the way it works is the nix derivations just refer to/copy the libraries on your mac computer.

FYI I found a copy of Virtualization.h under /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Virtualization.framework/Versions/A/Headers/Virtualization.h.

My feeling at the moment is that a lima-binary package might be considerably easier 😅

jonathanlking · 2023-01-05T01:23:12Z

#206285 (comment) @jonathanlking did you try this on a machine running macos Ventura ?

@dennajort Yes, this is on macOS 13.0.1 running on an M1

dhess · 2023-01-05T10:00:30Z

To use it, reference apple_sdk_11_0 instead of apple_sdk in your derivation and use pkgs.darwin.apple_sdk_11_0.callPackage instead of pkgs.callPackage. On Linux, this will have the same effect as pkgs.callPackage, so you can use pkgs.darwin.apple_sdk_11_0.callPackage regardless of platform.

I tried this as well, but it doesn't help, as the vz feature requires macOS SDK 13.0, but the most recent nixpkgs has is 11.0 (via the apple_sdk_11_0 stdenv).

My feeling at the moment is that a lima-binary package might be considerably easier 😅

That's what I concluded after trying the apple_sdk_11_0 option. Even if someone started adding support for macOS SDK 13.0 today, it would likely be months before it was ready, if past experience is any guide.

tricktron · 2023-01-05T10:02:12Z

@jonathanlking @dhess
I tried going down that path with the following changes:

diff --git a/pkgs/applications/virtualization/lima/default.nix b/pkgs/applications/virtualization/lima/default.nix
index 29dd9b27232..a282e4c30de 100644
--- a/pkgs/applications/virtualization/lima/default.nix
+++ b/pkgs/applications/virtualization/lima/default.nix
@@ -7,6 +7,8 @@
 , xcbuild
 , sigtool
 , makeWrapper
+, Virtualization
+, Cocoa
 }:
 
 buildGoModule rec {
@@ -22,6 +24,8 @@ buildGoModule rec {
 
   vendorSha256 = "sha256-l53MTxLY/uid+0U/eY96l0aBWKImST1gN2BARilh2K0=";
 
+  buildInputs = lib.optionals stdenv.isDarwin [ Virtualization Cocoa ];
+
   nativeBuildInputs = [ makeWrapper installShellFiles ]
     ++ lib.optionals stdenv.isDarwin [ xcbuild.xcrun sigtool ];
 
@@ -29,7 +33,8 @@ buildGoModule rec {
   postPatch = ''
     substituteInPlace Makefile \
       --replace 'binaries: clean' 'binaries:' \
-      --replace 'codesign --entitlements vz.entitlements -s -' 'codesign --force --entitlements vz.entitlements -s -'
+      --replace 'codesign --entitlements vz.entitlements -s -' 'codesign --force --entitlements vz.entitlements -s -' \
+      --replace 'MACOS_SDK_VERSION=$(shell xcrun --show-sdk-version | cut -d . -f 1)' 'MACOS_SDK_VERSION=13'
   '';
 
   # It attaches entitlements with codesign and strip removes those,

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 8fa2336e176..601d86d8f43 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -36777,8 +36777,9 @@ with pkgs;
 
   colima = callPackage ../applications/virtualization/colima { };
 
-  lima = callPackage ../applications/virtualization/lima {
+  lima = darwin.apple_sdk_11_0.callPackage ../applications/virtualization/lima {
     inherit (darwin) sigtool;
+    inherit (darwin.apple_sdk_11_0.frameworks) Virtualization Cocoa;
   };
 
   logtop = callPackage ../tools/misc/logtop { };

This then leads to the next errors:

vendor/github.com/Code-Hex/vz/v3/virtualization_helper.h:25:9: warning: macOS 12.3 API has been disabled [-W#pragma-messages]
vendor/github.com/Code-Hex/vz/v3/virtualization_helper.h:32:9: warning: macOS 13 API has been disabled [-W#pragma-messages]
virtualization_12.m:14:52: error: property 'canStop' not found on object of type 'VZVirtualMachine *'
virtualization_12.m:27:42: warning: instance method '-stopWithCompletionHandler:' not found (return type defaults to 'id') [-Wobjc-method-access]
/nix/store/lv5xsqihygkhyb9djf840nhlymi7j77m-apple-framework-Virtualization-11.0.0/Library/Frameworks/Virtualization.framework/Headers/VZVirtualMachine.h:57:12: note: receiver is instance of class declared here
virtualization_12.m:42:18: error: unknown receiver 'VZGenericPlatformConfiguration'; did you mean 'VZSerialPortConfiguration'?
/nix/store/lv5xsqihygkhyb9djf840nhlymi7j77m-apple-framework-Virtualization-11.0.0/Library/Frameworks/Virtualization.framework/Headers/VZVirtualMachineConfiguration.h:16:8: note: 'VZSerialPortConfiguration' declared here
virtualization_12.m:42:56: error: 'init' is unavailable
/nix/store/lv5xsqihygkhyb9djf840nhlymi7j77m-apple-framework-Virtualization-11.0.0/Library/Frameworks/Virtualization.framework/Headers/VZSerialPortConfiguration.h:26:1: note: 'init' has been explicitly marked unavailable here
virtualization_12.m:55:50: warning: instance method '-setDirectorySharingDevices:' not found (return type defaults to 'id') [-Wobjc-method-access]
/nix/store/lv5xsqihygkhyb9djf840nhlymi7j77m-apple-framework-Virtualization-11.0.0/Library/Frameworks/Virtualization.framework/Headers/VZVirtualMachineConfiguration.h:46:12: note: receiver is instance of class declared here
virtualization_12.m:70:88: error: expected expression
virtualization_12.m:70:63: error: use of undeclared identifier 'VZPlatformConfiguration'

According to this pr it should be possible to use vz on sdk 11.0, however the guarding in https://github.com/Code-Hex/vz/blob/957dfd2add9a0badf25dee8cd1dc01a79e8fee06/virtualization_12.m#L9-L20
is not working properly.

Even if get all that to work, it seems fragile to me because the root problem is that we are two major sdk versions behind. So I guess a binary version of lima is indeed the better approach for those who want the bleeding edge experimental vz emulation.

tricktron · 2023-01-05T14:13:23Z

I opened a pr for the lima binary derivation #209171.

jbgosselin · 2023-01-05T17:00:45Z

Thanks for your help on that !
My idea is to not try to fiddle too much with the apple sdk at the moment. As is, if the new apple sdks are added to nix, it may work automatically if I understand ?

tricktron · 2023-01-05T17:12:16Z

My idea is to not try to fiddle too much with the apple sdk at the moment. As is, if the new apple sdks are added to nix, it may work automatically if I understand?

Yes. Bumping the apple sdk is hard work, however (see the monthly macOS roundup: https://discourse.nixos.org/t/nix-macos-monthly/12330), so don't expect that to happen anytime soon.

farcaller · 2023-01-05T17:50:48Z

Result of nixpkgs-review pr 206285 run on aarch64-darwin 1

2 packages built:

colima
lima

farcaller · 2023-01-05T17:54:06Z

lima uname -a works just fine 👍

tricktron · 2023-01-09T14:44:11Z

@SuperSandro2000, I think this is ready to merge.

wegank · 2023-01-23T13:20:33Z

@ofborg build lima colima

lima: 0.13.0 -> 0.14.1

4f67556

jbgosselin mentioned this pull request Dec 15, 2022

lima: 0.13.0 -> 0.14.2 #205976

Closed

ofborg bot requested a review from voanhduy1512 December 15, 2022 18:58

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Dec 15, 2022

deejayem reviewed Dec 16, 2022

View reviewed changes

jbgosselin added 2 commits December 16, 2022 11:17

add _output cleaning before build

dc05102

remove rm, add --force to codesign

1a5fda5

deejayem approved these changes Dec 16, 2022

View reviewed changes

SuperSandro2000 reviewed Dec 17, 2022

View reviewed changes

pkgs/applications/virtualization/lima/default.nix Outdated Show resolved Hide resolved

Combine substituteInPlace on Makefile

788a5c3

Co-authored-by: Sandro <[email protected]>

jbgosselin requested review from SuperSandro2000 and removed request for voanhduy1512 December 18, 2022 03:49

pecigonzalo mentioned this pull request Dec 29, 2022

colima: 0.5.0 -> 0.5.2 #208094

Merged

13 tasks

lima: 0.14.2

fa65a5e

jbgosselin changed the title ~~lima: 0.13.0 -> 0.14.1~~ lima: 0.13.0 -> 0.14.2 Dec 29, 2022

ofborg bot requested a review from voanhduy1512 December 29, 2022 19:23

tricktron approved these changes Dec 29, 2022

View reviewed changes

dhess added a commit to hackworthltd/hacknix that referenced this pull request Jan 1, 2023

feat: lima 0.14.2.

1976e9b

Until fixed upstream. Source: NixOS/nixpkgs#206285

dhess mentioned this pull request Jan 1, 2023

feat: lima 0.14.2. hackworthltd/hacknix#607

Merged

jonathanlking mentioned this pull request Jan 3, 2023

colima start --vm-type=vz failing when installed from Nix abiosoft/colima#556

Closed

5 tasks

tricktron mentioned this pull request Jan 5, 2023

lima-bin: init binary derivation at 0.14.2 #209171

Merged

12 tasks

farcaller approved these changes Jan 5, 2023

View reviewed changes

wegank merged commit 0797f5f into NixOS:master Jan 23, 2023

jbgosselin deleted the lima-0.14.1 branch January 23, 2023 16:43

Janik-Haag added the 12. first-time contribution This PR is the author's first one; please be gentle! label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lima: 0.13.0 -> 0.14.2 #206285

lima: 0.13.0 -> 0.14.2 #206285

jbgosselin commented Dec 15, 2022 •

edited

Loading

deejayem commented Dec 16, 2022

deejayem left a comment

jbgosselin commented Dec 16, 2022

jbgosselin commented Dec 16, 2022

tricktron left a comment

dhess commented Jan 1, 2023 •

edited

Loading

jbgosselin commented Jan 1, 2023

jonathanlking commented Jan 3, 2023

jbgosselin commented Jan 4, 2023

dhess commented Jan 4, 2023

jbgosselin commented Jan 4, 2023

roblabla commented Jan 4, 2023

jbgosselin commented Jan 5, 2023

jbgosselin commented Jan 5, 2023

jonathanlking commented Jan 5, 2023 •

edited

Loading

jonathanlking commented Jan 5, 2023

dhess commented Jan 5, 2023

tricktron commented Jan 5, 2023

tricktron commented Jan 5, 2023

jbgosselin commented Jan 5, 2023

tricktron commented Jan 5, 2023

farcaller commented Jan 5, 2023

farcaller commented Jan 5, 2023

tricktron commented Jan 9, 2023

wegank commented Jan 23, 2023

lima: 0.13.0 -> 0.14.2 #206285

lima: 0.13.0 -> 0.14.2 #206285

Conversation

jbgosselin commented Dec 15, 2022 • edited Loading

Description of changes

Things done

deejayem commented Dec 16, 2022

deejayem left a comment

Choose a reason for hiding this comment

jbgosselin commented Dec 16, 2022

jbgosselin commented Dec 16, 2022

tricktron left a comment

Choose a reason for hiding this comment

dhess commented Jan 1, 2023 • edited Loading

jbgosselin commented Jan 1, 2023

jonathanlking commented Jan 3, 2023

jbgosselin commented Jan 4, 2023

dhess commented Jan 4, 2023

jbgosselin commented Jan 4, 2023

roblabla commented Jan 4, 2023

jbgosselin commented Jan 5, 2023

jbgosselin commented Jan 5, 2023

jonathanlking commented Jan 5, 2023 • edited Loading

jonathanlking commented Jan 5, 2023

dhess commented Jan 5, 2023

tricktron commented Jan 5, 2023

tricktron commented Jan 5, 2023

jbgosselin commented Jan 5, 2023

tricktron commented Jan 5, 2023

farcaller commented Jan 5, 2023

farcaller commented Jan 5, 2023

tricktron commented Jan 9, 2023

wegank commented Jan 23, 2023

jbgosselin commented Dec 15, 2022 •

edited

Loading

dhess commented Jan 1, 2023 •

edited

Loading

jonathanlking commented Jan 5, 2023 •

edited

Loading