Support ZFS delegation to container

[dalbani]

This issue is about reviving the discussion in https://github.com/lxc/lxd/issues/4184 from back in 2018, to take into account recent changes in the features provided by OpenZFS.
Namely what openzfs/zfs#12263 has brought since September 2022, and which is to be included in the yet to be released OpenZFS 2.2: "Linux namespace delegation support".

As as I explained in https://github.com/lxc/lxd/issues/4184#issuecomment-1560552412, I've successfully managed to delegate ZFS management to a container by:

1. compiling OpenZFS packages for Debian from the master branch on the host
2. compiling LXD 5.0 from Debian 12 with a slight modification by swapping zfsutils-linux for openzfs-zfsutils in: https://github.com/lxc/lxd/blob/123231d27121223f20f11a65499ce8a5e35d704b/lxd/storage/drivers/driver_zfs_utils.go#L259
3. defining a "hook file" like:

    #!/bin/sh -eu
   
    # FIXME find a way to programatically find the parent dataset name
    ZFS_DATASET=default/containers/${LXC_NAME}
   
    if [ ${LXC_HOOK_TYPE} = start-host ]; then
      zfs set zoned=on ${ZFS_DATASET}
      zfs zone /proc/${LXC_PID}/ns/user ${ZFS_DATASET}
    elif [ ${LXC_HOOK_TYPE} = post-stop ]; then
      zfs set zoned=off ${ZFS_DATASET}
    fi
   

4. setting up a container with something:

    config:
      raw.lxc: |
        lxc.hook.version = 1
        lxc.hook.start-host = /etc/lxc/hooks/zfs-zone
        lxc.hook.post-stop = /etc/lxc/hooks/zfs-zone
    devices:
      zfs:
        path: /dev/zfs
        type: unix-char
   

5. installing the openzfs-zfsutils package in a Debian 12 container
6. enjoying being able to manage ZFS in the container (via K3s / OpenEBS ZFS LocalPV to be precise) 😄

The only limitation/issue that I've seen so far, is the inability to do actions on the host like lxc push or lxc edit which have an effect on the rootfs directory while the container is running, as the ZFS dataset is not accessible at that time.
But that's a limitation I'm more than ready to accept.

---

So the question is: instead of messing around which manually configured hooks, what do you think of integrating the support for ZFS delegation in LXD itself?
Where the feature could be activated at the container/profile level via a dedicated setting for example.
Would you have pointers to share if someone (me?) wants to have a try at the implementation?

---

PS: I've noted a couple of places in the documentation would be changed if ZFS delegation was to be officially supported:

1. https://linuxcontainers.org/lxd/docs/stable-5.0/reference/storage_zfs/#limitations

Delegating part of a pool

ZFS doesn’t support delegating part of a pool to a container user. Upstream is actively working on providing this functionality.

2. https://linuxcontainers.org/lxd/docs/stable-5.0/reference/storage_drivers/#feature-comparison

Storage driver usable inside a container
ZFS: no