nix-shell dependencies can be garbage collected any time now / persistent nix-shell envs

[goodwillcoding]

There does not seem to be a way to prevent nix-shell dependencies from being garbage collected at any time. Related to this, for a while now there been possible to create persistent nix-shell environments that would be spared garbage collection. Neither of the two methods (see below) are now working.

shell.nix

with import <nixpkgs> {};
stdenv.mkDerivation {
    src = "./.";
    name = "test-0.1";
    buildInputs = [ hello ];
}

Previous Method 1 (https://nixos.org/nix-dev/2016-November/022173.html):

nix-instantiate shell.nix --show-trace  --indirect --add-root "$(pwd)/"shell.drv
nix-shell $(readlink $(pwd)/shell.drv)

While this does create a gcroot to the derivatation, it does not work (my guess cause it is not a gcroot to the build output)

$ tree /nix/var/nix/gcroots/auto/
/nix/var/nix/gcroots/auto/
└── rz3awpwi7lxpwfjmhqbab0rdfc3nj94m -> /home/user/tmp/persist-nix-shell/shell.drv

If nix-collect-gc is run while the shell is active it cleans up everything this derivation depends on

Previous Method 2:

mkdir gcroot
nix-shell shell.nix --indirect --add-root "gcroot/gc"

This used to create symlinks to dependencies, how ever since --indirect is now ignored as of 19477e8 this also no longer works as gcroot to the output is no longer created.

Please advise on this

[goodwillcoding]

@edolstra any thoughts? suggestion on this ^

[srghma]

Method 1 makes gcroot for shell derivation, this indeed doesnt prevent it dependencies to be garbage collected
Method 2 should makes gcroot for each dependency of shell derivation, but it's not working indeed

I have made script that makes same as method 2

#! /bin/sh -eu

# make gcroot for each dependency of shell and run nix-shell

mkdir -p .gcroots
nix-instantiate shell.nix --indirect --add-root $PWD/.gcroots/shell.drv
nix-store --indirect --add-root $PWD/.gcroots/shell.dep --realise $(nix-store --query --references $PWD/.gcroots/shell.drv)
exec nix-shell $(readlink $PWD/.gcroots/shell.drv)

tnx to http://nix-dev.science.uu.narkive.com/QjeSH005/truly-persistent-nix-shell

[lheckemann]

Do you have the keep-derivations and keep-outputs options set to true in your nix.conf? This is necessary for this to work iirc.

[srghma]

@lheckemann no, I dont have this options

 ~  cat /etc/nix/nix.conf
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
build-users-group = nixbld
max-jobs = 40
cores = 0
sandbox = true
extra-sandbox-paths =
substituters = https://cache.nixos.org https://cachix.cachix.org https://srghma.cachix.org
trusted-substituters =
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cachix.cachix.org-1:eWNHQldwUO7G2VkjpnjDbWwy4KQ/HNxht7H4SSoMckM= srghma.cachix.org-1:EUHKjTh/WKs49hFtw6bwDE9oQLeX5afml0cAKc97MbI=
auto-optimise-store = false
require-sigs = true

trusted-users = root srghma
allowed-users = *
builders =

auto-optimise-store = true

[lheckemann]

keep-derivations (sorry, got the name wrong in my previous comment) defaults to true, so that should be fine.
Try adding nix.extraOptions = "keep-outputs = true"; to your configuration.nix and rebuilding, then see if your shell dependencies still get gc'd when you use method 1.

keep-outputs
    If true, the garbage collector will keep the outputs of non-garbage derivations. If false (default), outputs
    will be deleted unless they are GC roots themselves (or reachable from other roots).
    In general, outputs must be registered as roots separately. However, even if the output of a derivation is
    registered as a root, the collector will still delete store paths that are used only at build time (e.g., the C
    compiler, or source tarballs downloaded from the network). To prevent it from doing so, set this option to true.

[srghma]

@lheckemann with keep-outputs and using this script

#! /bin/sh -eu

mkdir -p .gcroots

# add shell as gc-root in /nix/var/nix/gcroots/auto
nix-instantiate shell.nix --indirect --add-root $PWD/.gcroots/shell.drv

exec nix-shell $(readlink $PWD/.gcroots/shell.drv) "$@"

garbage collector indeed keep dependencies of shell, neat trick, but I still use the previous version of the script since keep-outputs = true is not defaut

[Elvecent]

While the script above works, I'd really love to have a first-class feature like this. Protecting a shell's dependencies from garbage collection is something I'd expect to be immediately configurable, I mean, per shell.

Is it possible for nix-shell to check some "protected" setting in shell.nix and accordingly update relevant gc roots? Said roots should probably also be removed when the setting is switched off.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pinned-nixpkgs-keeps-getting-garbage-collected/12912/5

[sleirsgoevy]

Found a solution:

nix-build -A inputDerivation -o gcroots

This will build a dummy derivation that has the shell's inputs as runtime dependencies, and place a GC root into gcroots.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/starting-nix-shell-without-rebuilding/16394/7

[con-f-use]

Isn't keep-outputs effectively equivalent to turning off garbage collection completely (except for build dependencies which still get collected)?

[kevincox]

I was surprised that nix-shell --add-root result-shell runs successfully but doesn't actually add a root. I think if we could make that option work it would be a great first step. Then we can decide if it makes sense to do this by default.

[con-f-use]

Gentle bumb @edolstra , since this seems like essential functionality. I suspect a lot of people would like to not download the entire internet every time an often used nix-shell gets garbage collected. Can we have an easy to use option to keep a nix-shell and its dependencies? See kevincox's post directly above.

[tengkuizdihar]

@sleirsgoevy will the old derivation be included in the GC when I change the shell.nix? I was hoping for that kind of behavior, where the only one that's GC'd is everything but the current one that's being used.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-09-nix-team-meeting-minutes-15/23951/1

[fricklerhandwerk]

Discussed in Nix team meeting on 2022-12-19.

Decision: closing as won't fix.
Reason: This is what nix develop will support, and fixing nix-shell is too complicated to be worth the effort. Previous attempts did not succeed. Nixpkgs implements this behavior using the Nix language, and probably so should nix develop – but that is a separate isuse.

Complete discussion

• @thufschmitt: apparently the shell is not GC'd any more while running due to concurrent GC
  • nix develop can already dump a GC root to the shell environment
    • related issue requesting a primop to do that from the Nix language, then nix-shell could have that, too
    • there is already an implementation of the same behavior within Nixpkgs
  • bigger problem: we can't add roots to eval-time inputs
  • proposal: close as wontfix, because it's really complicated to fix nix-shell, which is not worth it
    • there is an implementation attempt that does not do the right thing
    • @fricklerhandwerk: apparently people want that feature so much that they do all kinds of things to get it. should we consider this more generally?
      @thufschmitt: apparently people want the behavior of nix develop
      - that should be implemented in the Nix language, but that's a different question
• @Ericson2314: What about grabbing the inputs from a GC?
  • there is a request for a primop for that: provide getDerivationEnvironment as a primop to reference nix develop drv from Nix code #7468
  • @thufschmitt: We sholdn't need a nix lang primop for that

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-16-nix-team-meeting-minutes-17/24120/1

[tobiasBora]

What's the nix develop way to solve this? I tried to use nix develop --profile shell_profile_nix_develop but I can't find any link added in /nix/var/nix/gcroots, and I'm affraid to manually do something like nix develop --profile /nix/var/nix/gcroots/auto