ghcWithPackages environment doesn't build with integer-simple #153319

cdepillabout · 2022-01-03T12:18:36Z

Describe the bug

ghcWithPackages doesn't appear to work with integer-simple.

On 4acff2b, I get an error with the following command:

$ nix-shell -I nixpkgs=./. -p 'haskell.packages.integer-simple.ghc8107.ghcWithPackages (p: [p.random])'
this derivation will be built:
  /nix/store/8xw19mncwzyavib9rdwibyz4sh28mljh-ghc-integer-simple-8.10.7-with-packages.drv
building '/nix/store/8xw19mncwzyavib9rdwibyz4sh28mljh-ghc-integer-simple-8.10.7-with-packages.drv'...
/nix/store/7w5cdwwmcccmr5bb8qxayn7969sv743w-random-1.2.0/nix-support:
propagated-build-inputs: /nix/store/7mas5p1hd3xyw8bmgqkkwr1h2jabagpn-splitmix-0.1.0.4/nix-support/propagated-build-inputs
rm: cannot remove '/nix/store/48xd2pxvbi00ph96qr5ba6mwn1ypx9iy-ghc-integer-simple-8.10.7-with-packages/lib/ghc-integer-simple-8.10.7/package.conf.d/package.cache.lock': No such file or direc
tory

This command works correctly with non-integer-simple:

$ nix-shell -I nixpkgs=./. -p 'haskell.packages.ghc8107.ghcWithPackages (p: [p.random])'
$ runghc <<< $'import System.Random\nmain = (randomIO :: IO Int) >>= print'
6838749691054437585

My guess is that this is caused by the code block in pkgs/development/haskell-modules/with-packages-wrapper.nix right here:

nixpkgs/pkgs/development/haskell-modules/with-packages-wrapper.nix

Lines 132 to 140 in 18c84ea

    
               ${lib.optionalString hasLibraries '' 
        
                # GHC 8.10 changes. 
        
                # Instead of replacing package.cache[.lock] with the new file, 
        
                # ghc-pkg is now trying to open the file.  These file are symlink 
        
                # to another nix derivation, so they are not writable.  Removing 
        
                # them allow the correct behavior of ghc-pkg recache 
        
                # See: https://github.com/NixOS/nixpkgs/issues/79441 
        
                rm $out/lib/${ghc.name}/package.conf.d/package.cache.lock 
        
                rm $out/lib/${ghc.name}/package.conf.d/package.cache

The rm command fails with the error: rm: cannot remove '/nix/store/48xd2pxvbi00ph96qr5ba6mwn1ypx9iy-ghc-integer-simple-8.10.7-with-packages/lib/ghc-integer-simple-8.10.7/package.conf.d/package.cache.lock': No such file or directory

This error can be worked around with a change like the following:

diff --git a/pkgs/development/haskell-modules/with-packages-wrapper.nix b/pkgs/development/haskell-modules/with-packages-wrapper.nix
index e5fe60a0ae5..72cf37f8277 100644
--- a/pkgs/development/haskell-modules/with-packages-wrapper.nix
+++ b/pkgs/development/haskell-modules/with-packages-wrapper.nix
@@ -136,8 +136,8 @@ symlinkJoin {
      # to another nix derivation, so they are not writable.  Removing
      # them allow the correct behavior of ghc-pkg recache
      # See: https://github.com/NixOS/nixpkgs/issues/79441
-     rm $out/lib/${ghc.name}/package.conf.d/package.cache.lock
-     rm $out/lib/${ghc.name}/package.conf.d/package.cache
+     rm ${packageCfgDir}/package.cache.lock
+     rm ${packageCfgDir}/package.cache
 
      $out/bin/${ghcCommand}-pkg recache
      ''}

If you look at the definition of packageCfgDir, it appears that ghc.name is being used where ${ghcCommand}-${ghc.version} should be used.

However, even with this change, the resulting environment is not correct. The environment does not correctly contain the random package:

$ nix-shell -I nixpkgs=./. -p 'haskell.packages.integer-simple.ghc8107.ghcWithPackages (p: [p.random])'
this derivation will be built:
  /nix/store/5k5fk87zjcdkjz1blrqmf0dvca9vsqwf-ghc-integer-simple-8.10.7-with-packages.drv
building '/nix/store/5k5fk87zjcdkjz1blrqmf0dvca9vsqwf-ghc-integer-simple-8.10.7-with-packages.drv'...
/nix/store/7w5cdwwmcccmr5bb8qxayn7969sv743w-random-1.2.0/nix-support:
propagated-build-inputs: /nix/store/7mas5p1hd3xyw8bmgqkkwr1h2jabagpn-splitmix-0.1.0.4/nix-support/propagated-build-inputs
$ runghc <<< $'import System.Random\nmain = (randomIO :: IO Int) >>= print'

/run/user/1000/runghcXXXX3395958-0.hs:1:1: error:
    Could not find module ‘System.Random’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
1 | import System.Random
  | ^^^^^^^^^^^^^^^^^^^^

My guess is that this is because the ghcWithPackages environment is created with symlinkJoin, but the path to the package config directory is different for integer-simple packages and the integer-simple GHC.

Here's what the paths look like for normal GHC:

$ nix-build ./. -A haskell.compiler.ghc8107
/nix/store/ch0a40krsc4zaa9wv8qd1i56pswjzwx0-ghc-8.10.7
$ ls /nix/store/ch0a40krsc4zaa9wv8qd1i56pswjzwx0-ghc-8.10.7/lib/ghc-8.10.7/package.conf.d/
...
$ nix-build ./. -A haskell.packages.ghc8107.random
/nix/store/qamvzkgcl3qcafjc3kngirdvwxhdpqj6-random-1.2.0
$ ls /nix/store/qamvzkgcl3qcafjc3kngirdvwxhdpqj6-random-1.2.0/lib/ghc-8.10.7/package.conf.d/
random-1.2.0-4yTZ53A8d79AtqKspztz45.conf

You can see that random's package.conf file ends up in lib/ghc-8.10.7/package.conf.d/.

Here's what the paths look like for the integer-simple GHC:

$ nix-build ./. -A haskell.compiler.integer-simple.ghc8107
/nix/store/dbi6hx9fa4lsd3bilq62bqxz0sm41fwb-ghc-integer-simple-8.10.7
$ ls /nix/store/dbi6hx9fa4lsd3bilq62bqxz0sm41fwb-ghc-integer-simple-8.10.7/lib/ghc-8.10.7/package.conf.d/
...
$ nix-build ./. -A haskell.packages.integer-simple.ghc8107.random
/nix/store/7w5cdwwmcccmr5bb8qxayn7969sv743w-random-1.2.0
$ ls /nix/store/7w5cdwwmcccmr5bb8qxayn7969sv743w-random-1.2.0/lib/ghc-integer-simple-8.10.7/package.conf.d/
random-1.2.0-4yTZ53A8d79AtqKspztz45.conf

You can see here that haskell.compiler.integer-simple.ghc8107 has it's package.conf.d in lib/ghc-8.10.7/package.conf.d/, but random has it's .conf file in lib/ghc-integer-simple-8.10.7/package.conf.d/. So this .conf file doesn't end up in the right place in the symlinkJoin in with-packages-wrapper.nix.

Possible Solution

I'm not sure what would be the best solution for this, but I imagine we have to be very careful to make sure GHC and our Nix scripts are looking at the right place for the package conf dir in the following files:

pkgs/development/haskell-modules/generic-builder.nix
pkgs/development/haskell-modules/with-packages-wrapper.nix
pkgs/development/compilers/ghc/*.nix

The packageCfgDir variable that is currently defined in with-packages-wrapper.nix should likely be an attribute exported by the GHC derivations, and reused in the other files.

Notify maintainers

@NixOS/haskell

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.81, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.5pre20211126_55275fc`
 - channels(root): `"nixos-21.11.334247.573095944e7"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

The text was updated successfully, but these errors were encountered:

nixos-discourse · 2022-01-03T12:30:45Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/overriding-ghc-only-for-final-package/16953/2

pranaysashank · 2022-03-31T06:48:03Z

I had the same issue, but with musl and integer-gmp instead. So it's similar to what is described above except ghc-integer-simple-8.10.7 is replaced with ghc-musl-8.10.7. I worked around this by changing the derivation to not add a variant suffix.

sternenseemann · 2022-03-31T10:14:36Z

I'll look into fixing this finally, sorry this fell through the cracks. Since the workaround is so simple it should be quite easy to track down all uses of ghc.name that remain.

This is an incorrectness pointed out in NixOS#153319 which we already have a proper solution for.

sternenseemann · 2022-04-05T10:34:33Z

In addition to the testing in the linked PR, I tested both examples from the issue again, which now work correctly in haskell-updates.

cdepillabout · 2022-04-05T10:44:05Z

Great, thanks for looking into this!

sternenseemann · 2022-04-05T10:45:16Z

This will be closed automatically when the fix is in master, let's keep it open until then.

locallycompact · 2023-01-19T13:30:44Z

I'm getting this error again using ghcHEAD.withPackages.

       > propagated-build-inputs: /nix/store/0dfk2vhxr3m1rsd42nlbl6gb67frqvdc-lens-5.2/nix-support/propagated-build-inputs
     > /nix/store/vxsirk4xqaj4m7sij4gshz8gcss03zmi-adjunctions-4.4.2/nix-support:
     > propagated-build-inputs: /nix/store/0dfk2vhxr3m1rsd42nlbl6gb67frqvdc-lens-5.2/nix-support/propagated-build-inputs
     > /nix/store/66v6g3xy5xw58z08cgl7bagj9adazkxi-invariant-0.6/nix-support:
     > propagated-build-inputs: /nix/store/0dfk2vhxr3m1rsd42nlbl6gb67frqvdc-lens-5.2/nix-support/propagated-build-inputs
     > /nix/store/rdga81w7k42j01y4sf400bgxp7shicwp-primitive-0.7.4.0/nix-support:
     > propagated-build-inputs: /nix/store/0dfk2vhxr3m1rsd42nlbl6gb67frqvdc-lens-5.2/nix-support/propagated-build-inputs
     > /nix/store/zjzwf9c738nngmb3y2qjydcm16lbwy6k-vector-stream-0.1.0.0/nix-support:
     > propagated-build-inputs: /nix/store/0dfk2vhxr3m1rsd42nlbl6gb67frqvdc-lens-5.2/nix-support/propagated-build-inputs
     > rm: cannot remove '/nix/store/d6rf4q2v5xm5fdi9x9vl98s9xw87i9lr-ghc-9.7.20221224-with-packages/lib/ghc-9.7.20221224/package.conf.d/package.cache.lock': No such file or directory

sternenseemann · 2023-01-19T13:41:51Z

What is the precise expression you are using?

locallycompact · 2023-01-19T14:05:58Z

https://gitlab.homotopic.tech/horizon/horizon-core

nix run .#procex

I also tried it with your 9.6.1 branch and I get the same. There's a withPackages expression in shell/

sternenseemann · 2023-01-19T16:53:33Z

@locallycompact It'd be nice if you could try narrowing down the issue further in the future! In this case it's a piece of cake, as it affects all ghcWithPackages environments that are not “empty” using a hadrian-built GHC: #211608.

locallycompact · 2023-01-20T13:52:05Z

Thanks for finding it. I don't really get it but I'll take a look.

sternenseemann · 2023-01-20T18:06:24Z

No worries, I'll take care of it.

locallycompact · 2023-10-16T10:21:44Z

FYI I think this may have changed once more. Using latest commit of haskell-updates with GHC 9.8.1-release.

nix develop git+https://gitlab.horizon-haskell.net/package-sets/horizon-core#aeson.env

error: builder for '/nix/store/8whjnslj9pmf3b5hplxxh5n0pdvn07d9-ghc-9.8-with-packages.drv' failed with exit code 1;
       last 10 log lines:
       > propagated-build-inputs: /nix/store/a98mnbsv91qlgvlm8rnygvav80ivxd1v-data-fix-0.3.2/nix-support/propagated-build-inputs
       > /nix/store/0i9p75nxsvs9ljxciypbwrnl0zjg9xy8-transformers-compat-0.7.2/nix-support:
       > propagated-build-inputs: /nix/store/a98mnbsv91qlgvlm8rnygvav80ivxd1v-data-fix-0.3.2/nix-support/propagated-build-inputs
       > /nix/store/0i9p75nxsvs9ljxciypbwrnl0zjg9xy8-transformers-compat-0.7.2/nix-support:
       > propagated-build-inputs: /nix/store/a98mnbsv91qlgvlm8rnygvav80ivxd1v-data-fix-0.3.2/nix-support/propagated-build-inputs
       > /nix/store/siq0cq13mympkzqf3fqmi125v82vad4y-StateVar-1.2.2/nix-support:
       > propagated-build-inputs: /nix/store/a98mnbsv91qlgvlm8rnygvav80ivxd1v-data-fix-0.3.2/nix-support/propagated-build-inputs
       > /nix/store/siq0cq13mympkzqf3fqmi125v82vad4y-StateVar-1.2.2/nix-support:
       > propagated-build-inputs: /nix/store/a98mnbsv91qlgvlm8rnygvav80ivxd1v-data-fix-0.3.2/nix-support/propagated-build-inputs
       > rm: cannot remove '/nix/store/v0c2wwlw3zb1p2j774msadv4yg8njszi-ghc-9.8-with-packages/lib/ghc-9.8/lib/package.conf.d/package.cache.lock': No such file or directory
       For full logs, run 'nix log /nix/store/8whjnslj9pmf3b5hplxxh5n0pdvn07d9-ghc-9.8-with-packages.drv'.
error: 1 dependencies of derivation '/nix/store/a0yf3di39y5flirjw2rh0m8fhqrr4g9b-ghc-shell-for-aeson-2.2.1.0-env.drv' failed to build

cdepillabout added 0.kind: bug Something is broken 6.topic: haskell labels Jan 3, 2022

sternenseemann added a commit to sternenseemann/nixpkgs that referenced this issue Mar 31, 2022

ghcWithPackages: use packageCfgDir over ghc.name where appropriate

383fbfa

This is an incorrectness pointed out in NixOS#153319 which we already have a proper solution for.

sternenseemann mentioned this issue Mar 31, 2022

[haskell-updates] ghcWithPackages: fix ghclibdir computation with variantSuffix #166548

Merged

13 tasks

sternenseemann linked a pull request Mar 31, 2022 that will close this issue

[haskell-updates] ghcWithPackages: fix ghclibdir computation with variantSuffix #166548

Merged

13 tasks

cdepillabout closed this as completed Apr 5, 2022

sternenseemann reopened this Apr 5, 2022

sternenseemann closed this as completed Apr 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ghcWithPackages environment doesn't build with integer-simple #153319

ghcWithPackages environment doesn't build with integer-simple #153319

cdepillabout commented Jan 3, 2022 •

edited

Loading

nixos-discourse commented Jan 3, 2022

pranaysashank commented Mar 31, 2022 •

edited

Loading

sternenseemann commented Mar 31, 2022

sternenseemann commented Apr 5, 2022 •

edited

Loading

cdepillabout commented Apr 5, 2022

sternenseemann commented Apr 5, 2022

locallycompact commented Jan 19, 2023

sternenseemann commented Jan 19, 2023

locallycompact commented Jan 19, 2023

sternenseemann commented Jan 19, 2023

locallycompact commented Jan 20, 2023

sternenseemann commented Jan 20, 2023

locallycompact commented Oct 16, 2023

ghcWithPackages environment doesn't build with integer-simple #153319

ghcWithPackages environment doesn't build with integer-simple #153319

Comments

cdepillabout commented Jan 3, 2022 • edited Loading

Describe the bug

Possible Solution

Notify maintainers

Metadata

nixos-discourse commented Jan 3, 2022

pranaysashank commented Mar 31, 2022 • edited Loading

sternenseemann commented Mar 31, 2022

sternenseemann commented Apr 5, 2022 • edited Loading

cdepillabout commented Apr 5, 2022

sternenseemann commented Apr 5, 2022

locallycompact commented Jan 19, 2023

sternenseemann commented Jan 19, 2023

locallycompact commented Jan 19, 2023

sternenseemann commented Jan 19, 2023

locallycompact commented Jan 20, 2023

sternenseemann commented Jan 20, 2023

locallycompact commented Oct 16, 2023

cdepillabout commented Jan 3, 2022 •

edited

Loading

pranaysashank commented Mar 31, 2022 •

edited

Loading

sternenseemann commented Apr 5, 2022 •

edited

Loading