Add overlays mechanism to Nixpkgs. #21243

nbp · 2016-12-17T19:42:26Z

This patch add a new argument to Nixpkgs default expression named "overlays".

By default, the value of the argument is either taken from the environment variable NIXPKGS_OVERLAYS,
or from the directory ~/.nixpkgs/overlays/. If the environment variable does not name a valid directory
then this mechanism would fallback on the home directory. If the home directory does not exists it will
fallback on an empty list of overlays.

The overlays directory should contain the list of extra Nixpkgs stages which would be used to extend the
content of Nixpkgs, with additional set of packages. The overlays, i-e directory, files, symbolic links
are used in alphabetical order.

The simplest overlay which extends Nixpkgs with nothing looks like:

self: super: {
}

More refined overlays can use super as the basis for building new packages, and self as a way to query
the final result of the fix-point.

An example of overlay which extends Nixpkgs with a small set of packages can be found at:
https://github.com/nbp/nixpkgs-mozilla/blob/nixpkgs-overlay/moz-overlay.nix

To use this file, checkout the repository and add a symbolic link to
the moz-overlay.nix file in ~/.nixpkgs/overlays directory.

Motivation

The goal of this patch is to provide an alternative to the pkgs.overridePackages function (#20927), which re-execute Nixpkgs fix-point with new inputs. Instead, the alternative would be use Nixpkgs top-level function, with an overlays attribute which would be a list with the same argument as what is today the argument given to pkgs.overridePackages.

Thus, in addition to provide a useful extensible layer to Nixpkgs third parties, such as Mozilla, or any other company. This would remove the existing leakage of Nixpkgs fix-point out of Nixpkgs. This feature would also be nicer for the security-updates branch (#10851), as the overlays would be under the fix-point on which the patching mechanism works on.

Things done

Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"

mention-bot · 2016-12-17T19:42:28Z

@nbp, thanks for your PR! By analyzing the history of the files in this pull request, we identified @Ericson2314, @errge and @edolstra to be potential reviewers.

Ericson2314 · 2016-12-17T21:33:46Z

Hmm I'll need to think more about the ramifications for security updates, but otherwise I'm confused by the motivation for this. To me, mkExtensible is one of the best parts of the fix-extends ideom, and one I hope to see used more widely, not less.

CC @cstrahan, who created that abstraction.

nbp · 2016-12-17T23:23:07Z

@Ericson2314, @cstrahan:

The security-updates branch is using the fixed function a first time within the fix-point for base packages, and then as a standalone function for patching packages, and as a guide for another fix-point for propagating the patches.

The makeExtensibleWithCustomName function create a new attribute which let you inject phases in the pipeline of fixed function. If we wanted to carry the security-updates mechanism for the pkgs.overridePackages, then we would have to add special cases for this function, which I consider to be more painful than useful.

By adding the overlays list of extra phases, you can easily combine orthogonal extensions of Nixpkgs, which surpass the pkgs.overridePackages which only take one extra phase. Moreover, the pkgs.overridePackages function can easily be emulated as demonstrated in #20927 (comment)

Thus, I see no need to keep the pkgs.overridePackages function.

the-kenny

Some directory-resolution related questions.

the-kenny · 2016-12-25T08:46:31Z

doc/overlays.xml

+
+  <listitem>
+
+    <para>As the directory located at


How does this option work with different users? Will only the overlay set of the user executing nixos-rebuild considered or will it always use root's?

With the current patch, it will take the overlays of the user which is executing the command.

With the above suggestion, none should be considered unless they are specified in the configuration.nix file, or any other module.

In the directory <filename>~/.nixpkgs/overlays/</filename>.

the-kenny · 2016-12-25T08:47:07Z

doc/overlays.xml

+
+  <listitem>
+
+    <para>As argument of the imported attribute set. When importing Nixpkgs,


Is it possible to hard-code this directory for nixos-rebuild from configuration.nix? I think that might be useful.

I agree.

This would be a terrible experience if one configuration.nix is not evaluated the same way across computer depending on the ~/.nixpkgs/overlays directory content.

I will add the overlays argument to nixos/modules/misc/nixpkgs.nix.

This is fixed in commit 46ae6b5

grammar: As an argument ...

teh

This is really amazing work & a lot of power for such a small change! I'm very much in favour.

teh · 2016-12-26T16:56:13Z

doc/overlays.xml

+<title>Overlays</title>
+
+<para>This chapter describes how to extend and change Nixpkgs content using
+overlays.  Overlays are used to add phases in the fix-point used by Nixpkgs


"phases" sounds like pre or post-phase. Is a "phase" one file (and one function), and executed in alphabetical order?

Yes, one phase / layer is one overlay, which is a single function self: super: { ... }. Maybe I should use the term layer instead of phase?

I think "layer" would be better, too.

teh · 2016-12-26T16:57:12Z

doc/overlays.xml

+<title>Installing Overlays</title>
+
+<para>Overlays are looked for in the following order, the first valid one is
+considered, and all the rest are ignored:


Why would we ignore invalid files instead of throwing an error? This seems like a recipe for "silent, hard-to-debug" problems?

What I meant is the fallback logic, we consider the environment variable NIXPKGS_OVERLAYS, and check if this is a directory. Then we do the same check for the $HOME/.nixpkgs/overlays.

I agree, we should throw if NIXPKGS_OVERLAYS is set, but the content does not validate.
But I do not think we should do anything if the $HOME/.nixpkgs/overlays directory does not exists.

teh · 2016-12-26T16:58:13Z

doc/overlays.xml

+  <listitem>
+
+    <para>As a directory pointed by the environment variable named
+<varname>NIXPKGS_OVERLAYS</varname>. This directory can contain symbolic


Does NIXPKGS_OVERLAYS support colon separated paths like PATH?

For the moment no, I think this would be a nice addition in a future patch.

I also think being able to pull a Nix expression from a trusted remote (such as https://my.company/overlay.nix) would also be interesting.

teh · 2016-12-26T16:59:42Z

nixos/doc/manual/release-notes/rl-1703.xml

@@ -88,6 +90,11 @@ following incompatible changes:</para>
     <literal>networking.timeServers</literal>.
   </para>
  </listitem>
+
+  <listitem>
+    <para><literal>pkgs.overridePackages</literal> function no longer exists.     Instead import Nixpkgs a second time using <literal>import pkgs.path {


That's a very long space after "exists." Maybe a missing line break?

Ericson2314 · 2016-12-27T17:10:58Z

Note in 2278414#diff-19c9515018ff76dea736b57000a62d5bR49 I use overridePackages in a way that I worry would be hard to replicate with this.

Ericson2314 · 2016-12-27T17:21:42Z

More broadly, as I wrote in #16531 (comment) I think I thinking instead of having nixpkgs vs nixos, we should have nixpkgs vs "nixuser" vs "nixos". nixkpgs would have no config attrset, and no impure.nix, and basically just be packages. nixuser would have some sort of config system (probably module system), things like firefoxWrapper, dotfiles management if we ever get that.

I think splitting the pure and impure parts of this would be an excellent further justification for factoring out this "nixuser".

domenkozar

Overall looks good.

I think documentation needs someone who speaks English natively to flesh it out.

For Overlays I think we should have a tool that manages them. Something similar to git submodule that would sync git repositories. We should decide to go this route from the first day, prohibiting manipulation of ~/.nixpkgs/overlays by hand.

domenkozar · 2016-12-28T14:08:19Z

nixos/doc/manual/release-notes/rl-1703.xml

-    <para></para>
+    <para>Nixpkgs is now extensible through <link
+    xlink:href="https://github.com/NixOS/nixpkgs/pull/21243">overlays</link>.
+    See the Nixpkgs manual for more information.</para>


There should be a link to the manual with docs.

@domenkozar How would you link to the new Nixpkgs documentation from the release notes?

domenkozar · 2016-12-28T14:09:01Z

nixos/doc/manual/release-notes/rl-1703.xml

+  <listitem>
+    <para><literal>pkgs.overridePackages</literal> function no longer exists.
+     Instead import Nixpkgs a second time using <literal>import pkgs.path {
+    overlays = [ <replaceable>...</replaceable> ]; }</literal>.</para>


This is not clear what pkgs is and what would be the contents of overlays. It's better to point to overlays section in the manual.

teh · 2017-01-02T23:20:23Z

I gave this a spin and I'm not super convinced that a directory is the best public API (it's not terrible either). The additional overlays parameter however is awesome!

@domenkozar I think the docs are OK - do you have a specific thing in mind? Also, what kind of tool do you have in mind?

Both @nbp & @domenkozar - what do you think about splitting the filesystem API (~/.nixpkgs/overlays and NIXPKGS_OVERLAYS) into a separate branch so we can get the uncontroversial parts merged soon? I can do the work :)

aneeshusa

Overall docs look good, this is mostly small grammar fixes.

Also a request for a special callPackage to make using overlays easier.

aneeshusa · 2017-01-14T17:04:12Z

doc/overlays.xml

+
+<title>Overlays</title>
+
+<para>This chapter describes how to extend and change Nixpkgs content using


nit: content is a bit strange here, maybe use packages or just remove it

aneeshusa · 2017-01-14T17:04:30Z

doc/overlays.xml

+<title>Overlays</title>
+
+<para>This chapter describes how to extend and change Nixpkgs content using
+overlays.  Overlays are used to add layers in the fix-point used by Nixpkgs


nit: Only use one space between sentences

aneeshusa · 2017-01-14T17:06:38Z

doc/overlays.xml

+<section xml:id="sec-overlays-install">
+<title>Installing Overlays</title>
+
+<para>The set of overlays are looked for in the following order, only the


grammar: set of overlays is singular but are is for plurals; I would say The set of overlays is looked for in the following places.

I would split this into two sentences between order and only.

aneeshusa · 2017-01-14T20:23:09Z

nixos/modules/misc/nixpkgs.nix

+              openssh = super.openssh.override {
+                hpnSupport = true;
+                withKerberos = true;
+                kerberos = self.libkrb5


nit: missing semicolon

aneeshusa · 2017-01-14T20:23:57Z

nixos/doc/manual/release-notes/rl-1703.xml

+    pkgs.overridePackages (self: super: ...)
+</programlisting>
+
+    Should be replaced by:


nit: don't capitalize, use should

aneeshusa · 2017-01-14T21:20:08Z

doc/overlays.xml

+</orderedlist>
+</para>
+
+<para>For the second and third option, the directory contains either


..., the directory should contain Nix expressions defining the overlays. Each overlay can be a file, a directory containing a default.nix file, or a symlink to one of those. The expressions should follow the syntax described in ...

Also, this section should describe any ordering guarantees (or lack thereof) about overlays retrieved from a directory.

I added the following paragraph right after to describe the ordering of overlays, which has an impact both when providing a list of overlays as arguments of Nixpkgs, or when using a directory.

The order of the overlay layers can influence the recipe of packages if two layers override the same recipe. In the case where overlays are loaded from a directory, these are loaded in alphabetical order.

aneeshusa · 2017-01-14T21:22:05Z

doc/overlays.xml

+following the syntax described in <xref
+linkend="sec-overlays-layout"/>.</para>
+
+<para>To install an overlay, using the last option.  Clone the repository of


grammar: To install an overlay using the last option, you can clone the overlay's repository and add a symbolic link to it in the ...

aneeshusa · 2017-01-14T21:27:06Z

doc/overlays.xml

+
+<para>This chapter describes how to extend and change Nixpkgs content using
+overlays.  Overlays are used to add layers in the fix-point used by Nixpkgs
+to bind the dependencies of all packages.</para>


Overlays can also be used for adding packages, so I would make this a bit more general, maybe used by Nixpkgs to compose the set of all packages.

aneeshusa · 2017-01-14T21:28:39Z

doc/overlays.xml

+
+    <para>As a directory pointed by the environment variable named
+<varname>NIXPKGS_OVERLAYS</varname>. This directory can contain symbolic
+links to Nix expressions.


No need to mention symlinks here since it's described below.

aneeshusa · 2017-01-14T21:28:56Z

doc/overlays.xml

+
+    <para>As the directory located at
+<filename>~/.nixpkgs/overlays/</filename>. This directory can contain
+symbolic links to Nix expressions.


No need to mention symlinks here since it's described below.

nbp · 2017-01-14T23:48:28Z

@aneeshusa , Thanks a lot of the review, I will fix the nits tomorrow.

For the custom callPackage, I think that is a good idea, I will think about it and the consequences/applications with the security-updates branch.

aneeshusa · 2017-01-15T00:00:12Z

FYI, just noticed I had made a typo in my example for a custom callPackage, I fixed it at #21243 (comment).

All review comments are addressed. I am slowly working on a nix-overlay tool.

aneeshusa

A few more grammar nits, and some confusion about the usage of callPackage here.

IDK why GitHub is showing these as outdated, they should still be valid comments at time of writing...

aneeshusa · 2017-01-15T18:38:51Z

doc/overlays.xml

-to the entry Nix expression of the overlay.  These Nix expressions are
-following the syntax described in <xref
-linkend="sec-overlays-layout"/>.</para>
+<para>For the second and third option, the directory should contain Nix expressions defining the


grammar: options

aneeshusa · 2017-01-15T18:39:38Z

doc/overlays.xml

-the overlay, and add a symbolic link to it in the
-<filename>~/.nixpkgs/overlays/</filename> directory.</para>
+<para>The order of the overlay layers can influence the recipe of packages if multiple layers override
+the same recipe. In the case where overlays are loaded from a directory, these are loaded in


they are loaded

aneeshusa · 2017-01-15T18:39:57Z

doc/overlays.xml

+alphabetical order.</para>
+
+<para>To install an overlay using the last option, you can clone the overlay's repository and add
+a symbolic link to in the <filename>~/.nixpkgs/overlays/</filename> directory.</para>


link to it in

aneeshusa · 2017-01-15T18:40:16Z

doc/overlays.xml

-<para>An overlay is a Nix expression, which is a function which accepts 2
-arguments.</para>
+<para>Overlays are expressed as Nix functions which accept 2 arguments and return a set of
+packages</para>


Needs a period or colon at the end.

aneeshusa · 2017-01-15T18:42:18Z

doc/overlays.xml

+  boost = super.boost.override {
+    python = self.python3;
+  };
+  rr = super.callPackage ./pkgs/rr {


Why is this super.callPackage? I would think super.callPackage would use packages from super for dependencies, where the documentation says to use packages from self.

E.g. I would think we need to provide a custom callPackage for each self layer using
lib.callPackagesWith.

Technically you could use both, but this would get some extra meaning in the futur (security-update branch), because self.callPackage goes twice through the self attribute evaluation. The first one to resolve the callPackage function, and the second time to resolve the dependencies from self which are captured by the callPackage function.

Note that self is the same provided to all layers. So using self.callPackage is for the moment identical to super.callPackage, except for this extra hop through the fix-point.

I documented it as functions should be taken from super, because they already capture self if they have to.

Also note that this way, we make it syntactically verifiable, as anything which comes around the dependencies uses super, and anything which is use to find dependencies uses self.

That sounds reasonable, but can you explain how functions already capture self if they need to? E.g. if I have an overlay that adds two new packages:

self: super: { # Let's say all deps are already in super, # so I can see this working with super.callPackages or self.callPackages foo = super.callPackage {}; # But this depends on foo, which isn't in super bar = super.callPackage {}; }

How does the second super.callPackage get a reference to foo to provide to bar?

OK, I tried it out and looked at the implementation and I now understand why this works, albeit the current behavior is slightly surprising to me.

self: super: { callWithSelf = f: f self; }

Any layer defined after could then do:

self: super: { mySelfClone = super.callWithSelf (x: x); }

nbp · 2017-01-15T23:51:58Z

Tested locally on top of nixpkgs-unstable channel:

./maintainers/scripts/travis-nox-review-pr.sh nixpkgs-verify nixpkgs-manual nixpkgs-tarball nixpkgs-unstable
./maintainers/scripts/travis-nox-review-pr.sh nixos-options nixos-manual

teh · 2017-01-16T10:58:14Z

nixos/doc/manual/release-notes/rl-1703.xml

+
+<programlisting>
+  let
+    pkgs = import &lt;nixpkgs&gt; {}; in


just noticed a double in (see next line) - @nbp if you have direct push permissions maybe a quick fix? Otherwise I'll try to mop it up with some other change.

This is fixed as part of commit 0214d94. Thanks!

Ericson2314 · 2017-01-16T16:05:46Z

Wait, do overlays currently apply to bootstrapping stages after all?

Ericson2314 · 2017-01-16T16:08:22Z

pkgs/top-level/default.nix

@@ -80,7 +83,7 @@ in let
  boot = import ../stdenv/booter.nix { inherit lib allPackages; };


If we do want all bootstraping stages to include the overlay unconditionally, we could instead "partially apply" overlays to all packages (3 lines above the inherit lib nixpkgsFun;, but I cannot comment there because unchanged).

nbp · 2017-01-17T19:42:47Z

@Ericson2314

Wait, do overlays currently apply to bootstrapping stages after all?

Yeah, but I think it would makes sense to disable that as soon as we can move the bootstraping process in Nixpkgs fix-point, as this would no longer be needed. For the moment I would not recommend anybody to rely on overlays for changing the stdenv, except by doing it explicitly.

@teh Thanks, I will fix it later, if nobody does it before ;)

Ericson2314 · 2017-01-17T21:14:05Z

@nbp, well, to be clear, we can disable it in the meantime too. But that plan sounds fine to me.

edolstra · 2017-01-31T14:09:27Z

IMHO we should not add more ad hoc environment variables like $NIXPKGS_OVERLAYS. Why not use the NIX_PATH mechanism for this?

edolstra · 2017-02-01T15:10:58Z

I've changed $NIXPKGS_OVERLAYS to <nixpkgs-overlays>, and ~/.nixpkgs/overlays to ~/.config/nixpkgs/overlays in order to not add more non-XDG-compliant paths.

BTW, since the point of overlays is that they can be stacked easily, should the overlays in <nixpkgs-overlays> and ~/.config/nixpkgs/overlays be merged, rather than having the former override the latter?

nbp · 2017-02-01T19:10:55Z

@edolstra

BTW, since the point of oveprlays is that they can be stacked easily, should the overlays in <nixpkgs-overlays> and ~/.config/nixpkgs/overlays be merged, rather than having the former override the latter?

I honestly do not know yet what would be the most desirable behaviour.

My initial though was that we could replace the default configuration and test with a different one, which is interesting in terms of reproducibility.

nbp added 0.kind: enhancement Add something new 2.status: work-in-progress This PR isn't done 6.topic: hygiene 9.needs: changelog 9.needs: community feedback 9.needs: documentation labels Dec 17, 2016

nbp added this to the 17.03 milestone Dec 17, 2016

nbp self-assigned this Dec 17, 2016

nbp force-pushed the overlays branch from 5cd514c to 4186c06 Compare December 18, 2016 02:22

Mic92 mentioned this pull request Dec 18, 2016

Package request: Android Open Source Project #21222

Closed

cstrahan self-assigned this Dec 19, 2016

nbp force-pushed the overlays branch from 4186c06 to f291b29 Compare December 24, 2016 15:47

nbp added 8.has: changelog 8.has: documentation and removed 9.needs: changelog 9.needs: documentation 2.status: work-in-progress This PR isn't done labels Dec 24, 2016

nbp force-pushed the overlays branch 3 times, most recently from 8a1226f to 57e54d6 Compare December 24, 2016 16:21

the-kenny reviewed Dec 25, 2016

View reviewed changes

teh approved these changes Dec 26, 2016

View reviewed changes

nbp removed the 9.needs: community feedback label Dec 26, 2016

domenkozar previously requested changes Dec 28, 2016

View reviewed changes

aneeshusa suggested changes Jan 14, 2017

View reviewed changes

Update overlay documentation by following nits from aneeshusa.

4205b8a

Fix typo in nixpkgs.nix module.

7fda136

aneeshusa suggested changes Jan 15, 2017

View reviewed changes

nbp added 4 commits January 15, 2017 18:48

Fix missing overlays argument in stdenv/linux/default.nix

5b9131c

Fix extra nits from aneeshusa

120a603

Fix typo, lib.listOf --> types.listOf

17efce4

Fix nixpkgs manual generation, missing para closing tag.

8798264

nbp force-pushed the overlays branch from aa11d93 to 8798264 Compare January 15, 2017 19:09

aneeshusa approved these changes Jan 15, 2017

View reviewed changes

Fix release-notes compilation.

97e75f2

nbp force-pushed the overlays branch from 82122ec to 97e75f2 Compare January 15, 2017 23:30

nbp merged commit 8366525 into NixOS:master Jan 16, 2017

teh reviewed Jan 16, 2017

View reviewed changes

Ericson2314 reviewed Jan 16, 2017

View reviewed changes

globin mentioned this pull request Jan 19, 2017

ccacheWrapper broken #21984

Closed

grahamc mentioned this pull request Jan 23, 2017

#2 weekly news NixOS/nixos-weekly#14

Closed

17 tasks

grahamc added the weekly label Jan 23, 2017

aneeshusa mentioned this pull request Jan 28, 2017

NixOS: Respect nixpkgs.overlays #22221

Merged

7 tasks

michaelpj mentioned this pull request Aug 13, 2017

Overlays: allow overlays to be specified as a file #28228

Merged

8 tasks


		<listitem>

		<para>As argument of the imported attribute set. When importing Nixpkgs,


		<title>Overlays</title>

		<para>This chapter describes how to extend and change Nixpkgs content using

		@@ -80,7 +83,7 @@ in let
		boot = import ../stdenv/booter.nix { inherit lib allPackages; };

Add overlays mechanism to Nixpkgs. #21243

Add overlays mechanism to Nixpkgs. #21243

Conversation

nbp commented Dec 17, 2016 • edited Loading

Motivation

Things done

mention-bot commented Dec 17, 2016

Ericson2314 commented Dec 17, 2016

nbp commented Dec 17, 2016

the-kenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teh Dec 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 commented Dec 27, 2016

Ericson2314 commented Dec 27, 2016

domenkozar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teh commented Jan 2, 2017

aneeshusa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nbp commented Jan 14, 2017

aneeshusa commented Jan 15, 2017

aneeshusa left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nbp commented Jan 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 commented Jan 16, 2017

Choose a reason for hiding this comment

nbp commented Jan 17, 2017

Ericson2314 commented Jan 17, 2017

edolstra commented Jan 31, 2017

edolstra commented Feb 1, 2017

nbp commented Feb 1, 2017

nbp commented Dec 17, 2016 •

edited

Loading

teh Dec 26, 2016 •

edited

Loading

aneeshusa left a comment •

edited

Loading

nbp commented Jan 15, 2017 •

edited

Loading