feat: mount additional disks when using vz

[pendo324]

This PR adds support for using the additional disks feature with Virtualization.framework. I noticed it was missing, so I figured I'd add it.

Considerations:

• Virtualization.framework can only mount RAW disk images. The additional disk feature currently creates QCOW2 images. Since QEMU can support RAW images, is it ok to change the additional disk feature to create RAW images by default?
  • This change already included in this diff, but I can remove it
• Should Lima automatically convert QCOW2 disks to RAW before trying to mount a QCOW2 additional with vmType: vz? There's already precedent for runtime conversion of disks in imgutil, just wanted to make sure we want to do it for additional disks before adding
  • Auto conversion is already included in this diff, but I can remove it
  • I was having issues doing an in place conversion (I saw in place conversion in other uses of imgutil.QCOWToRaw), causing data loss, which is why there's an intermediate file right now. Not sure if there's a better way to do this?

Test:

$ qemu-img info --output=json ~/.lima/_disks/test/datadisk
{
    "virtual-size": 53687091200,
    "filename": "/Users/alvajus/.lima/_disks/test/datadisk",
    "cluster-size": 65536,
    "format": "qcow2",
    "actual-size": 14221312,
    "format-specific": {
        "type": "qcow2",
        "data": {
            "compat": "1.1",
            "compression-type": "zlib",
            "lazy-refcounts": false,
            "refcount-bits": 16,
            "corrupt": false,
            "extended-l2": false
        }
    },
    "dirty-flag": false
}

# Create an instance using the default fedora.yaml template with addition of additionalDisks: - test (note: test was created as QCOW2 as seen above)
$ ./_output/bin/limactl start --name=fedora template://fedora
...
INFO[0129] READY. Run `limactl shell fedora` to open the shell.

$ ./_output/bin/limactl shell fedora
[alvajus@lima-fedora lima]$
[alvajus@lima-fedora lima]$ ls /mnt/lima-test/
lost+found
[alvajus@lima-fedora lima]$ echo "THIS IS MY TEST FILE" | sudo tee /mnt/lima-test/testfile
THIS IS MY TEST FILE
[alvajus@lima-fedora lima]$ cat /mnt/lima-test/testfile
THIS IS MY TEST FILE
[alvajus@lima-fedora lima]$ exit
logout

$ ./_output/bin/limactl stop fedora
...
INFO[0002] [hostagent] QEMU has exited

$ ./_output/bin/limactl remove fedora
...
INFO[0000] Deleted "fedora" ("/Users/alvajus/.lima/fedora")

# Modify fedora.yaml to still use the same disk, but also include vmType: "vz" and recreate the VM
$ ./_output/bin/limactl start --name=fedora template://fedora
? Creating an instance "fedora" Proceed with the current configuration
WARN[0000] `vmType: vz` is experimental
...
INFO[0146] READY. Run `limactl shell fedora` to open the shell.

$ ./_output/bin/limactl shell fedora
[alvajus@lima-fedora lima]$ ls /mnt/lima-test/
lost+found  testfile
[alvajus@lima-fedora lima]$ cat /mnt/lima-test/testfile
THIS IS MY TEST FILE
[alvajus@lima-fedora lima]$ exit
logout

$ qemu-img info --output=json /Users/alvajus/.lima/_disks/test/datadisk
{
    "virtual-size": 53687091200,
    "filename": "/Users/alvajus/.lima/_disks/test/datadisk",
    "format": "raw",
    "actual-size": 6107136,
    "dirty-flag": false
}

[balajiv113]

Should Lima automatically convert QCOW2 disks to RAW before trying to mount

IMO The goal of additionalDisks are to share disk between instances doing conversion from QCOW2 - RAW defeats that purpose

is it ok to change the additional disk feature to create RAW images by default

My take on this would be to provide support for creation of additionalDisks with formatType.

limactl disk create --type raw

If raw is used, it can be shared across drivers.

About default, my take would be to keep qcow2 as default as they have optimal storage.

[pendo324]

Thanks for the review, @balajiv113, I pushed a new revision to incorporate your comments.

Should Lima automatically convert QCOW2 disks to RAW before trying to mount

IMO The goal of additionalDisks are to share disk between instances doing conversion from QCOW2 - RAW defeats that purpose

is it ok to change the additional disk feature to create RAW images by default

My take on this would be to provide support for creation of additionalDisks with formatType.

limactl disk create --type raw

If raw is used, it can be shared across drivers.

About default, my take would be to keep qcow2 as default as they have optimal storage.

While QCOW2 is better optimized for storage, I wonder if RAW as a default is still viable given that RAW images can also be sparse?

RAW images are a little harder to deal with, in that if you (for example) cp them, the entire disk will be copied, even if its not all in use. This can be avoided by using something like rsync -S, but that's not exactly obvious.

The only reason I'm proposing this is for interoperability between QEMU and Virtualization.framework VMs, but its possible that Lima can just leave this for individual users to decide too.

Instead of automatic runtime conversion, I can:

• remove the runtime conversion code
• add the --format raw option to the disk create command
• add notes on how to do the conversion manually, if you want to share data between the two VM types

Let me know if that sounds better, thanks!

[jandubois]

IMO The goal of additionalDisks are to share disk between instances doing conversion from QCOW2 - RAW defeats that purpose

How does it defeat the purpose? The images would still work with QEMU.

I just tested this by switching an Alpine instance from QEMU to VZ, which converted the diffdisk to RAW format. I made some changes to the image, stopped the instance and switched it back to QEMU. The disk remained in RAW format, but the modified data was still there and everything seemed to work normally.

What is the functionality we would lose by switching to RAW format?

[balajiv113]

@jandubois

doing conversion from QCOW2 - RAW defeats that purpose

With automatic conversion we will endup with 2 different disks (one with qcow2 format and other will be raw).

From vz, whatever we change that won't be reflected to qcow2 disk.

As you said, we can mount qemu also with raw format in that case it is fine.
Considering this, it would be better to decide on the disk format on disk create itself limactl disk create --format raw instead of doing auto conversion to raw on the fly

[jandubois]

With automatic conversion we will endup with 2 different disks (one with qcow2 format and other will be raw).

Oh, I didn't catch this. To me "automatic conversion" meant: the first time the disk is mounted to a VZ instance it will be replaced with the converted RAW disk, and from then on both QEMU and VZ instances would only use the RAW disk going forward. This should be transparent to the QEMU instances and guarantee data consistency because there is always only a single copy of the data.

I didn't look at the code in the PR yet, so I didn't realize this is not how it was implemented.

My main concern is about what happens if there is an error during the conversion (e.g. you run out of disk space). The old QCOW2 disk must remain in working state until the conversion has completed successfully.

Storage size

I don't have any knowledge about what advantages QCOW2 would have for storage size. Given that both formats have support for sparse files, how is QCOW2 optimized for storage? Do you have any references I can read up on this? Any comparison on storage efficiency and access speed (e.g. if QCOW2 would store data in compressed format, I would expect the disk to be slower than RAW).

[balajiv113]

Oh, I didn't catch this

My bad, Am the culprit here. I overlooked and missed that the PR already takes care of moving QEMU disk to a new extension .qcow2. So it is as per you said only.

formats have support for sparse files

Again my bad :D I was looking into ls which was giving 100G for raw disks and qcow was showing only the used space and this was misleading. But checked the disk utility now it is fine.

@jandubois
Thanks for the details :) Its mostly things missed from my end. The current approach of using raw format looks good to me

[jandubois]

I was looking into ls

This is a good point though: If you are only using QEMU, and you are not excluding the Lima VMs and disks from your backups, then you may end up backing up the full sparse file for a RAW disk if you have overallocated it significantly, and much of the space remains permanently unused.

So I think it makes sense to allocate additional disks as QCOW2 by default (but provide a --raw option to override), and auto-convert to RAW the first time it is mounted to a VZ instance. That way everyone automatically gets the best setup:

• Users only using QEMU will still use QCOW2
• Users using VZ (or both) will convert to RAW and have interoperability

So this would be my recommendation, as long as the auto-conversion to RAW is done in a way that any failure keeps the QCOW2 copy unharmed.

---

PS: Thinking about this some more, I think we are already currently at risk of damaging the VM disk if we ever run out of host disk space while trying to commit additional sparse data blocks. I wonder if limactl start should display a warning when you start an instance and the remaining free space is not enough to completely allocate the sparse space for the disk (plus an additional buffer of e.g. at least 1GB)?

[pendo324]

Thank you for the great discussion. QCOW2 disks are definitely more user friendly, in that users can just use default/familiar tools like cp and ls to interact with them. Even if they can be sparse, that might not be obvious on first glance.

I updated the code to allow a new --format option for disk create, it defaults to qcow2. Right now we only really need qcow2 or RAW, but I figured it might be good to make it more extensible now rather than changing it later if needed. Let me know what you think.

I also added a bunch of information on how to best deal with sparse RAW files to the docs. Let me know if it looks good, or if I should move that info to another location.

[jandubois]

@pendo324 On macOS you can use clonefile (2) (via cp -c) to copy a sparse file. No need to use GNU tools or rsync. It even shares the data blocks between the source and destination until either of them is modified (copy-on-write), and of course preserves the sparse-ness as well.

[pendo324]

@pendo324 On macOS you can use clonefile (2) (via cp -c) to copy a sparse file. No need to use GNU tools or rsync. It even shares the data blocks between the source and destination until either of them is modified (copy-on-write), and of course preserves the sparse-ness as well.

Oh wow, I couldn't find anything like that on macOS when I looked through the docs for cp. This is much better for users. Thanks for the info! I just updated the docs.

[afbjorklund]

I updated the code to allow a new --format option for disk create, it defaults to qcow2. Right now we only really need qcow2 or RAW, but I figured it might be good to make it more extensible now rather than changing it later if needed. Let me know what you think.

I used .vdi format for the VirtualBox driver, it should support raw as well - as long as you add a .img suffix to the file...

[pendo324]

I updated the code to allow a new --format option for disk create, it defaults to qcow2. Right now we only really need qcow2 or RAW, but I figured it might be good to make it more extensible now rather than changing it later if needed. Let me know what you think.

I used .vdi format for the VirtualBox driver, it should support raw as well - as long as you add a .img suffix to the file...

I can add vdi to the list of supported formats, or add all of the formats from this list. Might be more suitable in a separate PR though?

[balajiv113]

LGTM 👍

[balajiv113]

I can add vdi to the list of supported formats

I don't think it is required as of now.
Maybe once we support virtualbox driver then we can add the required format later in a separate PR

[AkihiroSuda]

Thanks

[pendo324]

Hey @jandubois, please let me know if there's any further changes you think should be made to get this merged when you get a chance. Thanks!

[jandubois]

@pendo324 I've been extremely busy the last 2 weeks, so I barely looked at any outstanding code reviews; sorry about that!

I've looked at the PR now once more, and there is one thing I don't get: After converting the qcow2 disk to raw format, you rename the old disk to datadisk.qcow2:

$ ls -l ~/.lima/_disks/foo/
total 9360
-rw-r--r--@ 1 jan  staff  10737418240 17 Mar 21:36 datadisk
-rw-r--r--@ 1 jan  staff       196768 17 Mar 21:35 datadisk.qcow2
lrwxr-xr-x@ 1 jan  staff           24 17 Mar 21:35 in_use_by -> /Users/jan/.lima/default

What is the point of keeping the old disk around. It will never be accessed by Lima again, and there is no way delete it with any lima commands, except when you delete the raw disk with limactl disk delete .... Why don't you want to delete it right away, to save the additional disk space (in case it was converted after data was already copied into it)?

---

Probably unrelated to this PR: the first time I stopped the VM and tried to delete the disk, the in_use_by symlink was still in place. I don't understand how that could have happened, and also couldn't reproduce after force-deleting and re-creating the disk, and then starting/stopping the VM once more. Just in case anybody else has any ideas:

$ l stop
INFO[0000] Sending SIGINT to hostagent process 205
INFO[0000] Waiting for the host agent and the driver processes to shut down
INFO[0000] [hostagent] 2023/03/17 21:17:17 tcpproxy: for incoming conn 127.0.0.1:62994, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: no route to host
INFO[0000] [hostagent] Received SIGINT, shutting down the host agent
INFO[0000] [hostagent] Shutting down the host agent
INFO[0000] [hostagent] Stopping forwarding "/run/lima-guestagent.sock" (guest) to "/Users/jan/.lima/default/ga.sock" (host)
INFO[0000] [hostagent] Unmounting disk "foo"
INFO[0000] [hostagent] Unmounting "/Users/jan"
INFO[0000] [hostagent] Unmounting "/tmp/lima"
INFO[0000] [hostagent] Shutting down VZ
ERRO[0000] [hostagent] dhcp: unhandled message type: RELEASE
INFO[0000] [hostagent] [VZ] - vm state change: stopped

$ l disk delete foo
WARN[0000] Skipping deleting disk "foo", disk is referenced by one or more non-running instances: ["default"]
WARN[0000] To delete anyway, run "limactl disk delete --force foo"

[pendo324]

@pendo324 I've been extremely busy the last 2 weeks, so I barely looked at any outstanding code reviews; sorry about that!

No problem, thanks for taking a look at it again 👍.

I've looked at the PR now once more, and there is one thing I don't get: After converting the qcow2 disk to raw format, you rename the old disk to datadisk.qcow2:

$ ls -l ~/.lima/_disks/foo/
total 9360
-rw-r--r--@ 1 jan  staff  10737418240 17 Mar 21:36 datadisk
-rw-r--r--@ 1 jan  staff       196768 17 Mar 21:35 datadisk.qcow2
lrwxr-xr-x@ 1 jan  staff           24 17 Mar 21:35 in_use_by -> /Users/jan/.lima/default

What is the point of keeping the old disk around. It will never be accessed by Lima again, and there is no way delete it with any lima commands, except when you delete the raw disk with limactl disk delete .... Why don't you want to delete it right away, to save the additional disk space (in case it was converted after data was already copied into it)?

Good point, I just updated the PR to delete the old qcow2 disk after swapping the RAW disk to be named datadisk. I think only removing the qcow2 disk after the RAW disk is in a usable state makes it more of a transaction, and ensures that at least one disk is usable (or at least recoverable) if the rename / deletion somehow fails. Let me know what you think. The alternative is to just delete the qcow2 disk as soon as the RAW disk is created, I wouldn't be opposed to that either.

---

Probably unrelated to this PR: the first time I stopped the VM and tried to delete the disk, the in_use_by symlink was still in place. I don't understand how that could have happened, and also couldn't reproduce after force-deleting and re-creating the disk, and then starting/stopping the VM once more. Just in case anybody else has any ideas:

$ l stop
INFO[0000] Sending SIGINT to hostagent process 205
INFO[0000] Waiting for the host agent and the driver processes to shut down
INFO[0000] [hostagent] 2023/03/17 21:17:17 tcpproxy: for incoming conn 127.0.0.1:62994, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: no route to host
INFO[0000] [hostagent] Received SIGINT, shutting down the host agent
INFO[0000] [hostagent] Shutting down the host agent
INFO[0000] [hostagent] Stopping forwarding "/run/lima-guestagent.sock" (guest) to "/Users/jan/.lima/default/ga.sock" (host)
INFO[0000] [hostagent] Unmounting disk "foo"
INFO[0000] [hostagent] Unmounting "/Users/jan"
INFO[0000] [hostagent] Unmounting "/tmp/lima"
INFO[0000] [hostagent] Shutting down VZ
ERRO[0000] [hostagent] dhcp: unhandled message type: RELEASE
INFO[0000] [hostagent] [VZ] - vm state change: stopped

$ l disk delete foo
WARN[0000] Skipping deleting disk "foo", disk is referenced by one or more non-running instances: ["default"]
WARN[0000] To delete anyway, run "limactl disk delete --force foo"

Not sure why this would happen, I've never seen that before, but I'll try to look into it.

[pendo324]

Accidentally removed some reviewers and it seems like I can't re-request for some reason. All tests passed, please take a look at this and let me know if you have any additional requests when you get a chance, thanks! @AkihiroSuda @jandubois @balajiv113

[jandubois]

Thanks, LGTM.

I have one comment, but I'm also fine with merging the PR as-is.

[jandubois]

Technically we don't know that the current format is qcow2, except for the fact that limactl disk create currently only creates either raw or qcow2 formatted volumes. So this code could be kept more generic, like this:

Suggested change

	qcow2Path := fmt.Sprintf("%s.qcow2", extraDiskPath)
	if err = imgutil.QCOWToRaw(extraDiskPath, rawPath); err != nil {
	return fmt.Errorf("failed to convert qcow2 disk %q to raw for vz driver: %w", diskName, err)
	}
	oldFormatPath := fmt.Sprintf("%s.%s", extraDiskPath, extraDiskFormat)
	if err = imgutil.ConvertToRaw(extraDiskPath, rawPath); err != nil {
	return fmt.Errorf("failed to convert %s disk %q to raw for vz driver: %w", extraDiskFormat, diskName, err)
	}
	...

[pendo324]

Good point, I created a new issue to track this, I'll implement it after we get this merged if that's alright #1438