-
-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] NixOS integration of nixpart #21403
Conversation
References a partition, disk, volume or whatever you like instead of using a device path or label. Creating those storage devices is done by nixpart and we can infer the right labels and/or poths from the device tree. I've added those hooks here, because duplicating things such as fsType, label, options or mountPoint in the storage configuration look kinda pointless to me. Signed-off-by: aszlig <[email protected]>
This is not the final version, because I'm not yet sure whether we want BTRFS as a special option here. Also, we don't check types properly yet. Signed-off-by: aszlig <[email protected]>
All of the types are just stubs right now, but I actually forgot volgroupType, as I didn't do tests of LVM in nixpart so far yet. Signed-off-by: aszlig <[email protected]>
Currently, this is still WIP and subject to change, but it helps to see whether our storage configuration options actually work out the way we want. Still needs a lot of cleanup, especially regarding the -m option, where I'm not sure whether we should do it with nixpart or write our own lightweight solution to be built into NixOS. Signed-off-by: aszlig <[email protected]>
Renames the test to closer match the NixOS module attribute and put it into release.nix. Of course, those tests still fail, because nixpart is still WIP and I haven't pushed the first version yet. Signed-off-by: aszlig <[email protected]>
We have renamed the function to nixpart() already, so let's make sure we rename it accordingly. Signed-off-by: aszlig <[email protected]>
We're internally calling nix-instantiate to get the required options from the configuration, so let's pass it through an option that skips this step. Signed-off-by: aszlig <[email protected]>
Let's use relative paths instead, because the version in <nixpkgs> isn't necessarily the same as the current nixpkgs tree. Signed-off-by: aszlig <[email protected]>
If we now use something like this: pkgs.libselinux.override { enablePython = true; python = pkgs.python3; } ... we now no longer will get the following error: selinuxswig_wrap.c:143:21: fatal error: Python.h: No such file or directory This happens because the Makefile is using the Python interpreter to get the right library dir for Python: PYTHONLIBDIR ?= $(shell $(PYTHON) -c " from distutils.sysconfig import *; print(get_python_lib(1)) ") Signed-off-by: aszlig <[email protected]>
This is basically multipath-tools patched into oblivion so that it's resulting into a "mpathconf" script. Although the patches apply to a lot more than just mpathconf and its manpage, we use filterdiff to only apply all the patches that apply to either mpathconf or mpathconf.[0-9]* so that we don't need to have the original source of multipath-tools. The reason I'm packaging this is that it's needed for libblockdev to configure multipath devices. As the description says, it's for editing /etc/multipath.conf so in the long run let's see whether we actually need this if we can patch libblockdev so it's not relying on a global multipath.conf at runtime. Signed-off-by: aszlig <[email protected]>
This is a package that is required for running common tests against projects from RedHat/Fedora, for example libbytesize. I'm removing nix_run_setup.py before the check phase because that files triggers a few pylint warnings and in turn causes the tests to fail. Note that I don't use rm -f here to make sure the build fails once we no longer need the nix_run_setup.py file so we can remove the reference from the pocketlint as well. Tested by building against Python 3.3, 3.4, 3.5 and 3.6. Building against Python 3.6 has failed because of a test failure in pylint, so it's only a transient failure. Signed-off-by: aszlig <[email protected]>
This is needed for libblockdev, one of its implicit dependencies. In order to successfully build the docs, we needed to pass XML_CATALOG_FILES so that xsltproc (called by gtkdoc-mkhtml) is able to find them. The package only works with Python 3, so I'm explicitly depending on python3Packages. Tested by building against i686-linux and x86_64-linux, which is why I set platforms to only Linux (don't have anything else to test on). Signed-off-by: aszlig <[email protected]>
The patch is from Fedora: http://pkgs.fedoraproject.org/cgit/rpms/parted.git/tree/0025-Add-libparted-fs-resize.pc.patch It adds the pkgconfig file via configure.ac and Makefile.am, so we need to use autoreconfHook and pkgconfig as well. The latter is needed so that PKG_CHECK_MODULES macros are properly replaced in the resulting configure script. Built with success on i686-linux and x86_64-linux. Signed-off-by: aszlig <[email protected]>
One of the requirements of libblockdev. I'm using "python" without specifying an explicit Python version, so that it's easy to override the Python version via something like this: volume_key.override { inherit (python3Packages) python; } Tested by building against i686-linux and x86_64-linux with Python 2.7 and Python 3.5. Signed-off-by: aszlig <[email protected]>
Needed for the current version of blivet. This library does a lot of calls to external binaries, so we need to patch in a lot of paths (bin-paths.patch). The checkPhase currently only checks whether the paths to these binaries are correct but doesn't run the real tests, which require root permissions. We also depend only on Python 3.x, because Python 2.x support seems broken at the moment and it really doesn't make sense to support it via patching on our side while it's eventually becoming obsolete. Other than the default output there is also a "tests" output which is going to be used by the upcoming NixOS VM test to run the libblockdev tests within a VM. Signed-off-by: aszlig <[email protected]>
Patch is originally from: https://bug735669.bugzilla-attachments.gnome.org/attachment.cgi?id=289405 It's part of the following bug report: https://bugzilla.gnome.org/show_bug.cgi?id=735669 I stumbled on this while running the libblockdev tests. Unfortunately we haven't seen a new release of parted yet, so I'll add this patch for now. Tested by building against i686-linux and x86_64-linux. Signed-off-by: aszlig <[email protected]>
The six library is a dependency of the library, so add it to the propagatedBuildInputs. In addition, pygobject3 and gobjectIntrospection need to be propagated as well if we want to use this library from within Python, which is its primary use-case. If we don't want this propagation, we could add a flag "withPython" or something similar so that the Python-specific parts are bound to a condition. Right now however, the only package which is going to use libblockdev is blivet. Signed-off-by: aszlig <[email protected]>
The main reason I'm updating this is because the libblockdev tests assume that lvm has proper default cache backend support, which only was added in recent versions. Upstream changelog can be found at: https://git.fedorahosted.org/cgit/lvm2.git/tree/WHATS_NEW?h=v2_02_168 Tested against the NixOS installer test (tests.installer.lvm). Signed-off-by: aszlig <[email protected]> Cc: @7c6f434c
First of all the purity.patch is no longer referenced, so instead of keeping it rotting around in the source tree, let's remove it. We can still bring it back if we really need it. Next, let's move version attribute to the derivation attributes, because this is what we adopted as a standard for other packages. Plus I couldn't find any package within the source tree where this is referenced. The systemd generator and unit files are now installed via the make files provided by the upstream package. This also means that the name of "blk_availability_systemd_red_hat.service" now is "blk-availability.service" and resides in $out/lib/systemd/system instead of $out/etc/systemd/system. Again, I haven't found anything that references either of these paths. We now also install the standard lvm config files in $out/etc/lvm, although lvm does not search these paths by default. We might want to fix this later. Tested against the tests.installer.lvm NixOS VM test. Signed-off-by: aszlig <[email protected]> Cc: @7c6f434c
For example the following fails on NixOS: lvcreate --profile thin-performance ... It tries to look for a thin-performance.profile file in /etc/lvm/profile, which by default doesn't exist. So in order to make those profiles work we now set a default profile directory which is used if lvm doesn't have a configuration file. Tested against the tests.installer.lvm NixOS VM test. Signed-off-by: aszlig <[email protected]> Cc: @7c6f434c
This is going to be used for libblockdev. I'm a bit hesitant to enable this by default though, to make sure we don't unnecessarily increase the closure size of lvm2 (especially if it comes to the initrd, this could be quite dangerous). Tested against the tests.installer.lvm NixOS VM test even though it shouldn't affect the result. Signed-off-by: aszlig <[email protected]>
While it probably doesn't make sense to include lvm2 with enabled thin provisioning to the initrd it certainly makes sense for libblockdev and in turn for nixpart. It remains to be seen whether we'd need to add this to a generated NixOS configuration once someone actually wants to use it with rootfs volumes but for now this is enough to let libblockdev's LVM tests pass. Signed-off-by: aszlig <[email protected]>
These tests are irrelevant on Nix(OS) because we ship libblockdev with all the binary paths directly built in (bin-paths.patch and "binPaths" attribute), so there is no way to actually fake these utilities except by rebuilding the package. Apart from that the tests try to recompile libblockdev on-the-fly to check whether plugin loading works correctly. This is also a non-issue with Nix, because *all* plugins are always available. So the tests.patch skips all of these tests involving these non-issues. Signed-off-by: aszlig <[email protected]>
Once again a patch from Fedora: http://pkgs.fedoraproject.org/cgit/rpms/parted.git/tree/0031-Use-BLKSSZGET-to-get-device-sector-size-in-_device_p.patch Submitted upstream at: http://lists.alioth.debian.org/pipermail/parted-devel/2016-March/004817.html This fixes the remaining failing tests for libblockdev. Tested by building against i686-linux and x86_64-linux. Signed-off-by: aszlig <[email protected]>
This includes a lot of cruft that has accumulated during my previous tries to get the tests working. After several tries (several revisions of libblockdev and blivet) now the libblockdev tests succeed so we can finally move over to get blivet working. Right now the blivet subtest is commented out entirely because it will fail to evaluate. First we need to refactor blivet and its tests and then we need to refactor the whole NixOS test with both sutests so they're as DRY as possible. Signed-off-by: aszlig <[email protected]>
Upstream changes are a bit longer, see: https://github.com/pyudev/pyudev/blob/v0.21.0/CHANGES.rst While doCheck is implicitly true, the tests aren't actually hooked into setup.py but need to be run via py.test. I'm not running them on purpose because pyudev's tests assume they're on a full-featured (GNU/)Linux system and thus the majority of the tests fail. Tested against Python 2.7, 3.4, 3.5 and 3.6. Note that Python 3.3 doesn't work because the package requires the enum module, which is only available since Python 3.4 so I've disabled support for Python 3.3 for pyudev. Signed-off-by: aszlig <[email protected]>
Watching 😄 |
Is ZFS support planned for this PR or is that going to be a separate PR? |
and what about luks and lvm?, which can be nested in either order for reference, here is how i automate most of my installs:
ive also got an unfinished installer project with an older example at https://github.com/cleverca22/nixos-installer/blob/master/installer/gui/bootconfig.cpp#L155 |
What's the status of this PR? |
@rvolosatovs: I still intend to rebase and put this forward within the next two months if possible. |
I really like the idea of being able to define a partition table in Nix. However, this PR is quite huge and hard to get up-to-date. Overwhelming to for me (and probably others) to help out. Could we somehow split this PR up? Introducing packages can already be done. Would it make sense to have PRs for:
There are also quite a few patches in here. Would it make sense to propose those upstream already? In particular:
We can then introduce If you think it's a good idea, defining which steps can be taken would be a good idea. After that it's easier to delegate where there is lack of time. |
@bobvanderlinden: I think most of the dependencies should already be in nixpkgs, so only the most direct dependencies should be left after a rebase. Also, most of the patches should already be applied upstream. The main hurdles I had previously, were with Sorry for not getting to that earlier, I'll try to do a rebase and check whether things got better before May the 5th 2019. If not I'll report back here. |
@bobvanderlinden: Okay, after giving this a try in rebasing, I found out that a lot has changed for blivet and related packages, so doing the rebase properly needs a lot more time to do, which I currently do not have :-/ So from the packages that were added here which are not yet part of nixpkgs, only Also, speaking of separating these things, I also made VM tests of Particularly if we could get |
Thank you for your contributions.
|
still important to me |
systemd now also has a partition manager by the way (only works with GPT though). |
@Mic92 do you have a link to it? |
The systemd-repart feature seems to be more intended to repartition the disk containing a base image more than partition an empty disk from an installer. |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
This looks like it is stalled to me. Please open a new PR. |
WIP branch of upcoming
nixpart
integration into NixOSStorage configuration is done like this:
https://github.com/aszlig/nixpart/wiki/Device-tree-representation
Todo list until the first merge to master:
storage.*
options experimental.This is part 1 of 2 of #2079 (
nixos-assimilate
).Test jobset: https://headcounter.org/hydra/jobset/aszlig/nixpart