[RFC 0049] Flakes #49

edolstra · 2019-07-15T23:19:33Z

Abstract: This RFC proposes a mechanism to package Nix expressions into composable entities called "flakes". Flakes allow hermetic, reproducible evaluation of multi-repository Nix projects; impose a discoverable, standard structure on Nix projects; and replace previous mechanisms such as Nix channels and the Nix search path.

Rendered

This follows the Rust naming.

samueldr · 2019-07-15T23:53:12Z

How will "channel advances" be published with flakes?

Flakes are looked up in a registry that maps identifiers such as nixpkgs to actual locations such as github:edolstra/nixpkgs/release-19.03. You can use such locations ("flake references") directly in the nix command:

Let's modify the location in the example to nixpkgs/release-19.03 (is this valid?). Will end-users need to somehow know that they should be using nixpkgs-channels/nixos-19.03 to get the tested channel advance?

If the "tested update" publish workflow is a branch on the main (nixpkgs) repository, it will become possible for contributors to accidentally push an untested, maybe even invalid, branch to the repo. Not a great idea.

Additionally, an orthogonal point.

Flakes replace channels [...]

I'm assuming this will be part of "Hammer out the details of NixOS/NixOps support for flakes", as this reduces a bit the reproducibility, if it isn't used in conjunction with nix flake pin when upgrading the system. Otherwise, rebuilding the system after a channel advance will produce a different system, instead of the expected no-op.

FRidh

The RFC is well written. I do think we should aim at getting answers to the unresolved questions. I think the proposed direction is good from a usability point of view, and to get a bit of order in our expressions.

rfcs/0049-flakes.md

FRidh · 2019-07-16T08:02:01Z

rfcs/0049-flakes.md

+
+Flakes are motivated by a number of serious shortcomings in Nix:
+
+* While Nix pioneered reproducible builds, sadly, Nix expressions are


Mention how pure evaluation mode fits in and if/why that is not sufficient.

rfcs/0049-flakes.md

FRidh · 2019-07-16T08:34:58Z

rfcs/0049-flakes.md

+GitHub archives, we cannot directly check that the contents match the
+commit hash.
+
+Note that lock files are only used at top-level: the `flake.lock` files


Consider the situation where one wants to offer a package set consisting of stable third-party applications, where each application is defined in a separate Flake and has a lock file to guarantee its reproducibility. In that case the lock files of the applications are ignored, no longer guaranteeing the stability of the applications.

The idea of not considering the lock files is good in case of libraries, but in case of applications not always.

Yes, good point. We could simply always use the lock file of a dependency if it exist (so good practice would be not to commit lock files for library flakes). It gets tricky for flakes that provide both libraries and applications (like Nixpkgs)...

This seems like something to consider carefully or it could become a big pain.

I'm not sure I understand the difference between packages and apps, but what about having whether a lock file is considered be based on which set the reference is in?

I think also that when a lock file is present, it should be used. It basically says "we provide a curated set, don't break it". It's then up to the community to ensure they don't use it incorrectly.

Now, in practice I think that is not something we cannot always rely on and so we want to be able to override/patch. Also, because upstreams tend to pin things too much; just look at the amount of patching we need with Python applications. We are given the attributeinputs, which is already imported, and won't allow us to override or patch the Flake when needed which, if correct, is a limitation.

We’re discussing things here that many other systems have stumbled over in the past.

Lockfiles shouldn't be committed to repositories under any circumstance, unless it is a distribution repository. This is no different to vendor bundling, which is rightly opposed. It's for a distributor to decide how to combine dependencies.

I agree that lockfiles in dependencies should be ignored. If this principle were abandoned, it should be strictly opt-in.

Could someone explain why lock files in dependencies should be ignored? Doesn't it break the reproducibility of evaluation if the inputs of the dependency got updated?

Without care, there is a tension between flakes offering composability on the one hand and reproducibility on the other. Ignoring dependent lockfiles is one way of reducing that tension: The flake.nix should specify the interface for the flake (eventually with version bounds or similar?) and the flake.lock should specify the current composition for people working on that project directly.

For example, I'd like us to eventually have a stdenv flake that abstracts out the bootstrapping logic and definition of mkDerivation from nixpkgs proper. When I'm developing on stdenv itself, I want some fixed definitions of things like gcc etc. so I have something to work with, but the entire point is that I ultimately want the caller to specify these! So when nixpkgs imports the stdenv flake, it should provide the pieces that make it all work.

Maybe it might make sense, for a first draft, to say “lock files in dependencies are a hard error for now” and to just add that as a to-be-resolved-later question?

I'm not sure we need what @FRidh wants right now (and it's something that apparently only makes sense for applications, so not that well-explored for build systems at least), and this way is forwards-compatible with any future road we might end up choosing for handling lock files in dependencies.

rfcs/0049-flakes.md

FRidh · 2019-07-16T08:57:14Z

rfcs/0049-flakes.md

+            "id": "import-cargo",
+            "inputs": {},
+            "narHash": "sha256-mxwKMDFOrhjrBQhIWwwm8mmEugyx/oVlvBH1CKxchlw=",
+            "uri": "github:edolstra/import-cargo/c33e13881386931038d46a7aca4c9561144d582e"


While understandable for Nix, these uri's are still a bit odd for other potential consumers. Is there a reason for not putting a concrete https url to a tarball here? I suppose to be able to still use git after all. In any case, the motivation seems to be missing.

It's because the github: scheme has some particular caching semantics, in particular using the fact that the ETag returned by GitHub is the Git revision. We can't assume that for https tarballs in general.

We should come up with a better name for this scheme, then, and make it so the full URL is still included. I don’t think Nix should be coupled to GitHub (and as proposed this wouldn’t even work with GitHub enterprise). If we used attribute sets instead of URLs, this could be an attribute in the set.

Hm, why wouldn't it work with GitHub enterprise?

There is definitely something to be said for attrsets instead of URLs, but it means that we would have different syntaxes for flake references on the command line and in .inputs.

Well, there’s no way to specify the domain GitHub is running on.

The command line is a good point. Could just be multiple arguments, maybe?

IMO, the best option here is to abandon the github: scheme and specially recognise URLs of the form https://github.com/... as supporting an alternate retrieval strategy. The advantages of this would be:

Separation of what is retrieved from how it is retrieved; github: and https://github.com/... are the same what, just being retrieved via a different method. But an URL is supposed to specify the what, not the how.

This avoids having to change github: URLs if the retrieval methods GitHub supports change.

Conversely, suppose some other hosting site https://example.com/ becomes popular in the future. If you want to add a more efficient way of retrieving Git repositories, with this model you'd be asking them to change to example:... to do so, which would break compatibility with older versions of Nix which don't understand example:....

Shouldn't it be up to the user whether they want to use https or something else? Similar to how you get the option to clone using https or ssh in the github interface.

While a URI does indeed merely identify a resource a URL definitely most commonly specifies a protocol to be used for the retrieval. So I'd rather see a scheme where git://github.com/... is valid than having https urls being subverted non-obviously.

IMO, the best option here is to abandon the github: scheme and specially recognise URLs of the form https://github.com/... as supporting an alternate retrieval strategy.

Special-casing certain HTTP URLs seems much uglier than having a separate URI scheme. You would expect all resources in a particular URI scheme to be interpreted uniformly.

Also, from a UX perspective, having to figure out / type https://github.com/edolstra/patchelf/archive/master.tar.gz (and hoping that that's the special URL recognized by Nix) is worse than github:edolstra/patchelf.

Special casing would be a very bad idea, because GitHub URLs don’t support all the features of git URLs. Like many other things in this rfc, I think we should be asking ourselves “is this feature absolutely critical for an initial release?” The answer to me in this case is quite clearly no. So I propose we drop GitHub URLs for now, in favour of git URLs, pending further, more focussed discussion.

Co-Authored-By: Frederik Rietdijk <[email protected]>

rfcs/0049-flakes.md

edolstra · 2019-07-16T10:15:56Z

How will "channel advances" be published with flakes?

Let's modify the location in the example to nixpkgs/release-19.03 (is this valid?). Will end-users need to somehow know that they should be using nixpkgs-channels/nixos-19.03 to get the tested channel advance?

Ideally, we would merge nixpkgs-channels into nixpkgs. The only reason why we have a separate channels repo is to prevent people from pushing channels, but nowadays GitHub allows per-branch access control. So we can restrict these branches to a team that only includes the channel update script.

So nixpkgs/release-19.03 would refer to the untested 19.03 branch, while nixpkgs/nixos-19.03 would be the tested one.

I'm assuming this will be part of "Hammer out the details of NixOS/NixOps support for flakes", as this reduces a bit the reproducibility, if it isn't used in conjunction with nix flake pin when upgrading the system. Otherwise, rebuilding the system after a channel advance will produce a different system, instead of the expected no-op.

Yes, exactly. The idea is that your NixOS system is itself a flake, so its flake.lock pins the exact version of Nixpkgs. FWIW a NixOS configuration flake looks like this:

{
  name = "eelco-configurations";

  edition = 201906;

  description = "My NixOS configurations";

  inputs =
    [ "nixpkgs/nixos-19.03"
      "dwarffs"
    ];

  outputs = inputs: rec {

    nixosConfigurations.my-server = inputs.nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules =
        [ ./configuration.nix
          inputs.dwarffs.nixosModules.dwarffs
        ];
    };

  };
}

which would be built by nixos-rebuild using

# nix build --profile /nix/var/nix/profiles/system ~/Misc/eelco-configurations:nixosConfigurations.my-server.config.system.build.toplevel

rfcs/0049-flakes.md

aanderse · 2019-07-16T12:34:44Z

@edolstra Am I correct in assuming that this will fix the issue where every machine in a nixops network is locked to the same channel?

Profpatsch

What I’m missing most from the current version is a prior work section with how other package managers solve this problem and how it has influenced the design of flakes. Especially the design of flakes’s lock files.

There’s been much practical research in that area over the last few years.

Profpatsch · 2019-07-16T16:03:22Z

rfcs/0049-flakes.md

+    packages.dwarffs =
+      with inputs.nixpkgs.packages;
+      with inputs.nixpkgs.builders;
+      with inputs.nixpkgs.lib;


It’s probably cleaner to a reader not in the know of nixpkgs if those are aliased instead of pulled in with with.

Hm, what do you mean with aliased?

Just something like:

let pkgs = inputs.nixpkgs.packages; inherit (inputs.nixpkgs) builders lib;

rfcs/0049-flakes.md

alyssais

I’m extremely wary of this. I don’t think that Nix should become a package registry, and I am extremely concerned by the lockfile issues others have raised. In particular, what should happen when dependencies pin to different versions of Nixpkgs, and the inability to patch flakes due to them not being exposed as derivations. I think it’s imperative that this RFC should not be merged until we have satisfactory solutions to these issues.

On the other hand, some things in here excite me a great deal. I love the syntax this will enable for the nix cli, for both pinned and unpinned versions. Channels and default.nix both have their own rather severe shortcomings, and I’m looking forward to seeing the back of them. This strikes me as an extremely complicated solution to that problem, but perhaps there is no simple one…

alyssais · 2019-07-16T18:34:18Z

rfcs/0049-flakes.md

+
+There are multiple registries:
+
+* The global registry


Why do we need a global registry at all? Could we not just, eg, provide hard-coded defaults for nix and nixpkgs, and let users do the rest? This would avoid becoming a package repository to any extent, which I think would be a very good thing.

@alyssais what would the defaults be?

Maybe

nix = https://github.com/NixOS/nix stable

nixpkgs = https://github.com/NixOS/nixpkgs unstable

We don't need a registry, but it seems important for UX. In fact it's more important here than in tools like Cargo, since the registry is used in a command line interface. We don't want to require people to type nix run github:NixOS/patchelf instead of nix run patchelf. Special-casing nixpkgs seems ugly, and it makes every other flake second-class compared to nixpkgs (which is bad since we should move away from the monorepo approach since it doesn't scale).

which is bad since we should move away from the monorepo approach since it doesn't scale

I disagree here, which is probably also the reason I'm a bit hesitant with the flakes approach in general. For me the monorepo approach is one of the major selling points of nixpkgs. Probably the selling point. It lowers the barrier to contribution a lot. It makes it very easy to find any information I want with a single grep. I can bisect any problem. I can test if my update breaks any known reverse dependencies. There is a central point of trust, even if that trust is very diffused at the moment.

I imagine a bunch of single-package repos loosely tied together with lockfiles will actually scale less. Why would I want patchelf to be in its own flake vs. in nixpkgs?

crates.io works very well so I don't think a registry needs to be a disaster. In any case,you don't have to use the registry. Of course then you have to type things like nix build github:NixOS/nixpkgs:... instead of nixpkgs:... all the time.

You could have the same ergonomics with a local registry only, with the nice commands for adding to it proposed in this rfc, though, with none of the need for us to maintain any sort of global registry.

Ultimately however the monorepo approach fundamentally doesn't scale: it shouldn't be necessary for everybody who just wants to share a NixOS module to have to add it to the nixpkgs repo.

I don’t want to argue this further in this thread, because I think it’s off-topic for the RFC, but I want to make clear that I still don’t agree and any such move would need to be discussed extensively at a later date before it were to happen.

crates.io works very well so I don't think a registry needs to be a disaster. In any case,you don't have to use the registry. Of course then you have to type things like nix build github:NixOS/nixpkgs:... instead of nixpkgs:... all the time.

You mentioned crates.io a lot in your NixCon talk and it's obvious that this RFC took a lot of inspiration of what Rust is doing. I'd like to chime in here that the public often doesn't see a lot of the effort and sweat that it takes on parts of people to run crates.io, and the architectural problems that it comes with. This doesn't just include the actual infrastructure and the team of people it takes to run it, but also a lot of policy decisions on things like DMCA requests, name squatting, etc that introduce a lot of work that general users of the service aren't aware of either. It's easy to say "look, it runs really well" and not see the weekly meetings on policy updates. I got the feeling that everybody involved in developing NixOS was already pretty booked out. I feel that adding yet another thing to maintain to the list of NixOS infrastructure things would be an extremely bad idea.

The crates.io ecosystem is fundamentally a different maintenance approach to NixOS. There are no "Rust maintainers". Sure, there's people who work on the compiler or who are active in various working groups and teams (:wave:) but there's no centralised development in the way that NixOS requires it. Building a bunch of libraries is different from building an operating system distribution and while a global registry of libraries works well for one, it wouldn't scale at all for the other because it's the same people involved in most of the pieces. It just adds overhead.

Re: the mono repo

As others have said previously, the fact that all the moving parts in NixOS are in a single place, can easily be bisected and tested against one another is a huge selling point for collaboration and also lowers the barrier to entry. NixOS is the first Linux distribution I've actually started working on, despite wanting and very much trying to get involved in both Debian and Fedora. I think it would be a shame if this were to change.

There's a relevant discussion/story from the Rust/ rustc repo about this. The repo itself is huge, unwieldy, takes ages to download etc. So it was proposed to remove certain components from it that are only tangentially related to the compiler core, such as rustdoc. Ultimately, it was decided to keep the repo as it is, because rustdoc is a component that frequently breaks with new PRs and so, it's common practise that if a compiler contribution breaks docs, that person then also fixes docs.

I think you could compare the development of rustc with the development of NixOS/nixpkgs. They are both mono repos, both part of an ecosystem with moving pieces around it, but strongly integrated i each other and benefit greatly from having all their parts kept in the same place.

rfcs/0049-flakes.md

edolstra · 2020-07-16T12:37:43Z

Yes, sounds good.

Mic92 · 2020-07-25T09:16:18Z

Since this RFC is not extensive enough to be used as documentation and the nix repository also does not provide any except for the source code I started writing a wiki article: https://nixos.wiki/wiki/Flakes

arianvp · 2020-07-25T10:16:06Z

Why not make those docs part of the nix manual?

…

On Sat, Jul 25, 2020, 11:16 Jörg Thalheim ***@***.***> wrote: Since this RFC is not extensive enough to be used as documentation and the nix repository also does not provide any except for the source code I started writing a wiki article: https://nixos.wiki/wiki/Flakes — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZNI3KE6RMCM7GE3VBTGTR5KPHFANCNFSM4ID3SJCQ> .

bqv · 2020-07-25T10:18:15Z

This is still an experimental feature

arianvp · 2020-07-25T11:04:53Z

That sounds like a reason to accept documentation to be out of date. (With a disclaimer) but not a reason not to have it in the manual and manpages. For discoverability and to persuade more people to experiment with it and give feedback I still think it would be nice to have it in the manual.

…

On Sat, Jul 25, 2020, 12:18 Tony O ***@***.***> wrote: This is still an experimental feature — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZNIZERWW7YGGT2JERBTLR5KWPRANCNFSM4ID3SJCQ> .

Mic92 · 2020-07-25T20:47:56Z

Why not make those docs part of the nix manual?
…
On Sat, Jul 25, 2020, 11:16 Jörg Thalheim @.***> wrote: Since this RFC is not extensive enough to be used as documentation and the nix repository also does not provide any except for the source code I started writing a wiki article: nixos.wiki/wiki/Flakes — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <github.com/notifications/unsubscribe-auth/AAEZNI3KE6RMCM7GE3VBTGTR5KPHFANCNFSM4ID3SJCQ> .

docbook... and I am not sure if those PRs are accepted.

Ekleog · 2020-07-27T20:17:26Z

Having a link to the wiki (that makes it very explicit it is an experimental feature) as part of the manual might make sense, I'd personally be wary of having the docs for experimental features that are expected to change alongside the docs for regular features — taking example on Rust that has an unstable book in addition to the regular (stable) rust book would seem like a good idea to me.

Mic92 · 2020-07-27T20:44:42Z

Having a link to the wiki (that makes it very explicit it is an experimental feature) as part of the manual might make sense, I'd personally be wary of having the docs for experimental features that are expected to change alongside the docs for regular features — taking example on Rust that has an unstable book in addition to the regular (stable) rust book would seem like a good idea to me.

The wiki has has now a complete reference on the nix command: https://nixos.wiki/wiki/Nix_command
This allows to link to new commands as well as extending examples. I also started fixing some examples that are currently broken in --help.

nixos-discourse · 2022-08-19T10:00:10Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/low-hanging-fruit-and-high-impact-tasks/21095/1

nixos-discourse · 2023-08-08T00:15:35Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-did-you-get-stuck-in-the-nix-ecosystem-tell-me-your-story/31415/6

nixos-discourse · 2024-04-28T01:00:54Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flake-hard-coded-fields-and-their-rationale/44225/2

nixos-discourse · 2024-04-30T10:06:43Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-is-not-dying-please-dont-spread-fear-actively/44310/91

edolstra and others added 7 commits July 9, 2019 21:03

Flakes RFC

13b4857

Updates

734d920

Update date

3b16530

fix typo

c83185c

Rename 'epoch' -> 'edition'

cc5d0a2

This follows the Rust naming.

Rename to RFC NixOS#49

f2cf8a8

Typo

1441693

FRidh suggested changes Jul 16, 2019

View reviewed changes

zimbatm and others added 2 commits July 16, 2019 09:24

Update rfcs/0049-flakes.md

9334af1

Co-Authored-By: Frederik Rietdijk <[email protected]>

Update rfcs/0049-flakes.md

57f139b

Co-Authored-By: Frederik Rietdijk <[email protected]>

lheckemann reviewed Jul 16, 2019

View reviewed changes

rfcs/0049-flakes.md Show resolved Hide resolved

lheckemann reviewed Jul 16, 2019

View reviewed changes