"nix profile" allows installing duplicate packages

[dtzWill]

Describe the bug

The "nix profile" command allows installing duplicate packages, which isn't what I expected (nix-env would reject this).

It does seem to check for conflicting binaries/files among different packages (two versions of bash, for example).

This bug is about having duplicate identical installation items, which at least seems unlikely to be intentional and easy enough to check for (I'm seeing some profile-related bits having custom operator <'s that look likely to be useful for catching at least these trivially duplicate entries?).

Steps To Reproduce

1. nix profile install --profile ~/test-profile nixpkgs#bash
2. nix profile install --profile ~/test-profile nixpkgs#bash
3. nix profile list --profile ~/test-profile
4. Observe duplicates in output:

0 flake:nixpkgs#legacyPackages.x86_64-linux.bash github:NixOS/nixpkgs/80e6ae766ae5d928da056dee74b747660f6c1178#legacyPackages.x86_64-linux.bash /nix/store/svxmj4xahyjqq4lciisvgi1fwl5bxpdd-bash-5.1-p8
1 flake:nixpkgs#legacyPackages.x86_64-linux.bash github:NixOS/nixpkgs/80e6ae766ae5d928da056dee74b747660f6c1178#legacyPackages.x86_64-linux.bash /nix/store/svxmj4xahyjqq4lciisvgi1fwl5bxpdd-bash-5.1-p8

Expected behavior

Reject attempt to install duplicate package. Or silently keeping the first, so one can install same items repeatedly without worrying about what's already installed.

nix-env --version output

nix-env (Nix) 2.5pre20211007_844dd90

EDIT: updated after testing conflict handling among different packages

[tomberek]

Follow-up questions:

• at what level should the de-duplication occur? The fully-resolved uri? the original Uri? storePaths?
• what does it mean to install separate attrPaths that resolve to the same storePath?
• why does this not currently error with a priority conflict?
• do we retain the removal by integer?

Initial attempt: flox@efef797

[Ericson2314]

@dtzWill

nix-env would reject this

What exactly do you mean by this? I am struggling to understand. Do you mean the nix-env policy to make the names modulo versions unique?

[dtzWill]

What exactly do you mean by this? I am struggling to understand. Do you mean the nix-env policy to make the names modulo versions unique?

Oh, I meant the mechanism whereby two separate packages can't provide same files. I was thinking that would prevent installing same package twice as well.

It turns out that installing exactly the same package with nix-env is not an error, although it does seem to replace the existing package as an "upgrade" (as opposed to introducing an additional entry).

$ nix-env -p ~/test-profile -f channel:nixos-unstable -iA bash
...
$ nix-env -p ~/test-profile -f channel:nixos-unstable -iA bash
replacing old 'bash-5.1-p16'
installing 'bash-5.1-p16'
$ nix-env -p ~/test-profile -q
bash-5.1-p16

[Ericson2314]

I thought nix profile would still complain if non-identical files were to be put at the same merged location?

[zhengpd]

I'm new to Nix and don't know much about Nix packages yet, but the exact-same entries from nix profile list are confusing and counter-intuitive. I thought the hash (f2e9a130461950270f87630b11132323706b4d91) would be different if they are installed as different packages.

❯ nix profile list               
0 flake:nixpkgs#legacyPackages.aarch64-darwin.git github:NixOS/nixpkgs/f2e9a130461950270f87630b11132323706b4d91#legacyPackages.aarch64-darwin.git /nix/store/pb3dc2gmarp5yfpczlzqrsyijgxl3h65-git-2.40.0
1 flake:nixpkgs#legacyPackages.aarch64-darwin.git github:NixOS/nixpkgs/f2e9a130461950270f87630b11132323706b4d91#legacyPackages.aarch64-darwin.git /nix/store/pb3dc2gmarp5yfpczlzqrsyijgxl3h65-git-2.40.0
2 flake:nixpkgs#legacyPackages.aarch64-darwin.git github:NixOS/nixpkgs/f2e9a130461950270f87630b11132323706b4d91#legacyPackages.aarch64-darwin.git /nix/store/pb3dc2gmarp5yfpczlzqrsyijgxl3h65-git-2.40.0
3 flake:nixpkgs#legacyPackages.aarch64-darwin.git github:NixOS/nixpkgs/f2e9a130461950270f87630b11132323706b4d91#legacyPackages.aarch64-darwin.git /nix/store/pb3dc2gmarp5yfpczlzqrsyijgxl3h65-git-2.40.0
4 flake:nixpkgs#legacyPackages.aarch64-darwin.git github:NixOS/nixpkgs/f2e9a130461950270f87630b11132323706b4d91#legacyPackages.aarch64-darwin.git /nix/store/pb3dc2gmarp5yfpczlzqrsyijgxl3h65-git-2.40.0

[bobvanderlinden]

I thought nix profile would still complain if non-identical files were to be put at the same merged location?

Indeed:

$ nix profile install --profile ~/test-profile nixpkgs#bash
$ nix profile install --profile ~/test-profile nixpkgs#busybox
error: files '/nix/store/j80wcbm6bb1nsqsgmniq43rg980p27q9-bash-5.2-p15/bin/sh' and '/nix/store/f82310kkwad5jsfm5vgnghjbv8h17441-busybox-1.36.0/bin/sh' have the same priority 10; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' or type 'nix profile install --help' if using 'nix profile' to find out how to change the priority of one of the conflicting packages (0 being the highest priority)

Same thing happens when installing bash from a different version:

$ nix profile install --profile ~/test-profile github:nixos/nixpkgs/nixpkgs-unstable#bash
$ nix profile install --profile ~/test-profile github:nixos/nixpkgs/nixos-21.05#bash
error: files '/nix/store/j80wcbm6bb1nsqsgmniq43rg980p27q9-bash-5.2-p15/lib/bash/push' and '/nix/store/dzrvibwj2vjwqmc34wk3x1ffsjpp4av7-bash-4.4-p23/lib/bash/push' have the same priority 10; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' or type 'nix profile install --help' if using 'nix profile' to find out how to change the priority of one of the conflicting packages (0 being the highest priority)

Note that this is with 2.13.3, as that is the version marked stable in NixOS. The UX of these messages has improved in #7788.

But if it is the same package it will still silently create multiple entries in profile. In addition, it also doesn't upgrade the package.

nix-env -if replaces the existing package with the new one and logs that is does so. This is sometimes convenient, but personally I think it should fail and suggest the upgrade command.

I've added this issue as part of the nix profile bad UX issue: #7966

[zhengpd]

I created many duplicate items with nix profile install when trying nix commands, so for convenience I wrote a small script to quickly remove those duplicates, named nix-remove-duplicate-pkgs:

#!/usr/bin/env ruby

require "open3"

# clean up duplicate nix packages
# this should be removed after nix officially fix the duplicate packages bug
# see https://github.com/NixOS/nix/issues/5587
list, _ = Open3.capture2("nix", "profile", "list")
groups = list.split("\n").group_by { |line| line.split(" ")[1] }

# collect positions for nix profile remove
positions_to_delete = groups.map do |_, lines|
  positions = lines.map { |line| line.split(" ")[0].to_i }
  (positions.size > 1) ? positions[1..] : []
end.flatten

# must delete high position first, otherwise the positions would change
positions_to_delete.sort.reverse_each do |pos|
  system("nix", "profile", "remove", pos.to_s)
end

Now just one command to clean up duplicates before Nix fix the issue.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-as-a-unified-format-for-profiles/29476/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakey-profile-declarative-nix-profiles-as-flakes/35163/7

[bobvanderlinden]

#8678 has been merged. This means we can look into letting profile entries conflict based on the inferred name.

Installing packages with 2 urls that infer the same profile entry name should then fail.