flakes: allow composing from multiple files

[andir]

Is your feature request related to a problem? Please describe.

Commonly I have multiple Nix files in my repository to keep certain matters isolated to some files only. Some complex haskell build doesn't have to be mixed up with the code for my rust program.

Describe the solution you'd like

I would like to declare dependencies aka. inputs in various files across my project. On the root of my project I'd just like to aggregate all the outputs of my projects. e.g. via something like this:

{
  outputs = {
    a = builtins.getFlake ./a;
    b = builtins.getFlake ./b;
  };
}

(curently failes with value is a path while a string was expected)

or maybe something like this:

{
  outputs = { self }: {
    something = builtins.getFlake (self + "/something");
  };
}

(currently this failes with the string '/nix/store/3zl72j46y56q5wb2vizl8vsnr9bwpya2-source/something' is not allowed to refer to a store path (such as '/nix/store/3zl72j46y56q5wb2vizl8vsnr9bwpya2-source'))

Describe alternatives you've considered
I could obviously put everything into one flake.nix but that is really messy in the long run and not really scalable in larger repositories.

[zimbatm]

This could be fixed if flakes supported relative paths. inputs = { a = { url = "./a"; }; };

As long as the relative path doesn't contain .. that should be fine.

[Kha]

Related: #3978

[andir]

Yeah, relative inputs might work but for a feature we are designing from scratch it looks awfully close to a hack.

…

On Wed, 4 Nov 2020, 09:27 zimbatm, ***@***.***> wrote: This could be fixed if flakes supported relative paths. inputs = { a = { url = "./a"; }; }; — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4218 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE365DZT5GJOFGHU74RBUTSOEF6XANCNFSM4TJQFFRQ> .

[kquick]

Would something like the following work?

{
    inputs.a.url = "./a";
    inputs.b.url = "./b";
    outputs = { a, b }: {
        packages = a.packages // b.packages;
    };
}

[andir]

That seems to be a hack and not a solution. As mentioned in #4089 (comment) for a format that isn't yet stabilized we shouldn't have to resort to all kinds of hacks already.

[edolstra]

What would you consider a non-hacky solution?

[andir]

As given in the initial comment but to stay in inline with the inputs example given by @kquick:

In a basic variant it could look like this:

Example 1:

{
inputs = {
   someInternalFlakeA = ./a;
};
outputs = { someInternalFlakeA }: {
   a = someInternalFlakeA;
};
}

but this wouldn't allow specifying any kind of .follows overrides unless we add more special casing (which we shouldn't?!) so probably something closer to the previously given example:

Example 2:

{
inputs = {
   nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
   someInternalFlakeA = {
     src = ./a; # could also be `url` but should just accept a path instead of requiring a string.
     inputs.nixpkgs.follows = "nixpkgs";
   };
};
outputs = { someInternalFlakeA }: {
   a = someInternalFlakeA;
};
}

Using a Path is probably least surprising for (new) users. The pattern src = ./. is very easy to understand and likely already something a user is familiar with. If this would instead be a string the question would be: Why is this any different from all the other nix expressions? Being consistent here (and in general) should make it less frustrating to deal with the language and more intuitive.

I am not yet sure about the lockfile behavior of "Example 2". It could be either of these (or something else):

1. The flake in ./a must have a valid lock file (at ./a/flake.lock) otherwise the input is not accepted. All of the inputs of ./a are as defined in the lockfile in that directory unless overriden (like nixpkgs in the example).
2. The lockfile of the flake is not required. If it is incomplete or does not exist locking will be done as if all of the inputs of ./a are declared in the current flake but still handled separately from any dependencies in the "root" flake (e.g. avoid name clashes).

I can (now) see how builtins.getFlake ./a would be hard to work within a flake.nix as in order to lock it you'd have to evaluate the flake expression. Given the restricted Nix featureset in a flake.nix that might still be possible without too much effort?

That being said I still think having support for builtins.getFlake ./a would be good. The flake that is referred to should be required to be "fully locked" in that case (unless impurity of running the locking code is accepted via a CLI switch?). This would also remove any ability to do overrides to the flake that is being referred to. My current use-case for this feature is to ease consuming flakes in code that isn't yet (or will never; for other reasons) be converted into a flake itself. It is the "free" version of the compat code without requiring additional Nix code.

[edolstra]

You mean a = someInternalFlakeA.<some-output>; right? Since someInternalFlakeA is a attrset of outputs.

Yeah, inputs.someInternalFlakeA.src = ./a seems like a good way to support this.

[andir]

You mean a = someInternalFlakeA.<some-output>; right? Since someInternalFlakeA is a attrset of outputs.

No, I mean it as an attribute set. Imagine you have a few teams/products and each of them has a flake in their root directory (which itself then refers to the flake of each of the individual programs). The directory layout in that repository could be something like this:

├── flake.nix
├── teamA
│   ├── flake.nix
│   ├── rabbit
│   │   ├── backend
│   │   │   └── flake.nix
│   │   ├── flake.nix
│   │   └── frontend
│   │       └── flake.nix
│   └── unicorn
│       ├── backend
│       │   └── flake.nix
│       ├── flake.nix
│       └── frontend
│           └── flake.nix
└── teamB
    ├── flake.lock
    ├── flake.nix
    └── snowflake
        ├── compiler
        │   └── flake.nix
        ├── debugger
        │   └── flake.nix
        └── flake.nix

So from the root of the repository you want to be able to do nix build .#teamB.snowflake.compiler & nix build .#teamA.unicorn.frontend. And respectively the equivalent commands in the folder of each of the teams.

[cole-h]

Small nit: I don't think a valid lockfile should be required, since flakes that don't specify any inputs other than self don't need a flake.lock.

[andir]

Small nit: I don't think a valid lockfile should be required, since flakes that don't specify any inputs other than self don't need a flake.lock.

I think that could be an optimisation in an edge-case. I could also imagine that you have only a lockfile in your "root flake" while all the other flakes do not have one (and do not need one as long as they are just leafs).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/wild-idea-how-about-splitting-nixpkgs-and-nixos/11487/11

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/possibility-for-subflakes/11589/5

[stale]

I marked this as stale due to inactivity. → More info

[andir]

Still relevant.

[stale]

I marked this as stale due to inactivity. → More info

[zimbatm]

https://github.com/hercules-ci/flake-parts helps a lot with that issue

[hraban]

An issue i'm running into, and which isn't solvable with just Nix hacks, is nested flake.lock files. I don't want to commit them to the repo, because the master flake.lock is the only one that matters, but I can't put them in .gitignore and still use the flake directly, because:

$ pwd
/Users/user/code/personal/dotfiles/bin
$ nix build
warning: Git tree '/Users/user/code/personal/dotfiles' is dirty
warning: creating lock file '/Users/user/code/personal/dotfiles/bin/flake.lock'
evaluating derivation 'git+file:///Users/user/code/personal/dotfiles?dir=bin#packages.aarch64-darwin.default'The following paths are ignored by one of your .gitignore files:
bin/flake.lock
hint: Use -f if you really want to add them.
hint: Turn this message off by running
hint: "git config advice.addIgnoredFile false"
error:
      … while updating the lock file of flake 'git+file:///Users/user/code/personal/dotfiles?dir=bin'

      error: program 'git' failed with exit code 1

This means I can only use the flakes from the "parent" directory. I guess that's still a minor win, but it's very tempting to support nix run and nix shell etc right there in the subdirectory.

(Alternatively you can just "mentally ignore" them but it's not ideal, now you risk accidentally adding them on a git add -A or git add .)

[deemp]

@hraban, you can add a default.nix with flake-compat near each of your flakes to make flakes importable from anywhere.

In my flakes, I replicated the flake logic in outputs with makeFlake and allowed to override inputs.

However, I don't know how to make overriding more declarative.

[hraban]

@deemp but you end up with flake.lock files everywhere, instead of just top level. Is my assumption correct that these lock files reflect "the last time I happened to run a nix command in that sub directory", rather than "the actual version this flake is pinned to when included from top level"?

I can import subdirectory flakes no problem (thanks to recent work allowing relative path inputs in flakes), but I want to:

1. Use the subdirectory flakes directly during dev, e.g. nix develop and even nix run
2. Also have a top-level flake which pins all the versions and inputs for all subflakes and injects them all (mostly because I want to ensure only one copy of any shared dependency is ever used)

Having a bunch of subdirectory flake.lock files littering the repo spoils nr 2.

What do you think? What's your approach?

[deemp]

@hraban, my current approach is to lock as little as possible.

flakes

I have a top flake (located in a repo root) flakes that provides all subflakes (flakes in subdirectories of the repo) and pinned inputs via the all attribute.
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/flake.nix#L129

The pinned inputs are provided by the source-flake. flake-compat is one of all attributes.
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/source-flake/flake.nix

The subflakes have a single input - flakes. Example:
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/drv-tools/flake.nix#L2

The corresponding flake.lock is tiny.
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/drv-tools/flake.lock

The flakes input is used to avoid using relative paths in default.nix for finding a rev of flake-compat. Example:
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/drv-tools/default.nix

consumers of flakes

I use pinned inputs in all my other top flakes via the flakes flake.

This is how I access pinned nixpkgs in a top flake.
https://github.com/deemp/projects/blob/6b38577a32ee7cf19ace1e567d17008c2c35b928/flake.nix#L7

All subflakes in this repo import the top flake to get flakes. Example:
https://github.com/deemp/projects/blob/6b38577a32ee7cf19ace1e567d17008c2c35b928/haskell/ts-serializable-test/flake.nix#L3

Sometimes, subflakes import parent flakes (located in parent directories). Example:
https://github.com/deemp/projects/blob/6b38577a32ee7cf19ace1e567d17008c2c35b928/haskell/ts-serializable-test/flake.nix#L7

All child flakes do have flake.locks, but these locks are almost empty. Example:
https://github.com/deemp/projects/blob/6b38577a32ee7cf19ace1e567d17008c2c35b928/haskell/ts-serializable-test/flake.lock

makeFlake

The makeFlake function used in flakes duplicates the flakes (I mean, Nix flakes) functionality. Example:
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/drv-tools/flake.nix#L3

Along with existing flake attributes, it provides an outputs function which can be called with custom inputs.
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/makeFlake.nix#L22

This function works with inputs specified in a flake, not with possibly empty inputs of that flake.
https://github.com/deemp/flakes/blob/b844ef681803fc5d6944fe6d16cb089a31ce4c22/flake.nix#L5

Discussion

Pros:

1. I can do nix develop in subflakes.
2. I have a remote flake flakes which pins all versions and inputs. Additionally, I have top flakes in repos that usually depend on flakes only.
3. flakes and top flakes can be imported by subflakes.
4. I mostly import flakes and have almost empty flake.locks.
5. Whenever there's a single inputs.flakes, a flake.lock is tiny.
6. I can access inputs (provided for makeFlake) of a subflake aSubflake via (import ./aSubflake).outputs.inputs.
7. I can override inputs of a flake via the outputs function added to that flake by makeFlake.

Cons:

1. I duplicated the inputs functionality of flakes via outputs.
2. I can't use inputs.nixpkgs.follows.

[hraban]

Very interesting. A bit too idiosyncratic for me but I love the creativity here.