cabal build doesn't find public sub package within development shell but builds fine with nix build.

[MangoIV]

When trying to cabal build a package that contains a public sublibrary, cabal fails to find this library within a shell provided by haskell.nix.

Somehow everything builds just fine with nix build.

Reproducer fails with:

Warning: No remote package servers have been specified. Usually you would have
one specified in the config file.
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: blup-0.1.0.0 (user goal)
[__1] next goal: bla (dependency of blup)
[__1] rejecting: bla-0.1.0.0/installed-Hsuvz2e1aJzCilZPkGBZy8 (does not
contain library 'blabla', which is required by blup)
[__1] fail (backjumping, conflict set: bla, blup)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: blup, bla

[L-as]

Does this happen with source-repository-package?

[MangoIV]

like, when I declare the dependency in a cabal.project file?

[L-as]

yeah

[L-as]

but if you do it through cabalProjectLocal when calling cabalProject'.

[MangoIV]

Can anyone point me at where this

bla-0.1.0.0/installed-Hsuvz2e1aJzCilZPkGBZy8

comes form? Maybe I can fix it if I know.

[MangoIV]

If I put a sourc-repository-package into the cabalProjectLocal, it doesn't get picked up by the shell either. (This time, the entire package is missing)

I still works for the nix build.

[MangoIV]

I'm thinking that something's wrong with shell. (or the way I used it.)

[MangoIV]

I think I found the issue:

 131   │   # Collect all (transitive) Haskell library dependencies of a
 132   │   # component.
 133   │   ## flatLibDepends :: Component -> [Package]
 134   │   flatLibDepends = component:
 135   │     let
 136   │       makePairs = map (p: rec { key=val.name; val=(p.components.library or p); });
 137   │       closure = builtins.genericClosure {
 138   │         startSet = makePairs component.depends;
 139   │         operator = {val,...}: makePairs val.config.depends;
 140   │       };
 141   │     in map ({val,...}: val) closure;

this doesn't include components.sublibs

[hamishmack]

Unfortunately I don't think that is it. The reason it does not have components.sublibs is that the nix-tools output for dependencies is explicit for sublibs (so project.plan-nix should have something like depends = [ bla.components.sublibs.bla ] in it). We were thinking about making it explicit for dependencies on the default library was well (#1679), but I am not sure it is worth increasing the size of the nix output.

I think the problem is more likely that the way we are making package DB entries for sublibs is not what cabal expects. So it works with builders/component-builder.nix, but not with cabal build.

I think we should try to compare the the package DB entries that cabal build creates when not in a haskell.nix shell (I guess they will be in dist-newstyle somewhere), with those provided in the haskell.nix nix-shell (as in the ones in the ghc-pkg list output in the shell).

[MangoIV]

if I add the sublibs in the flatLibDepends function, the derivation gets added to buildInputs which before isn't the case ghc-pkg list includes the sublib dependency but marks it as blue. cabal also doesn't recognise the sublib.

[MangoIV]

It also seems like its given a weird name, in case of my reproducer:

/nix/store/5kyyvr8cd18a9bhfklrqk7hwprjwadaj-ghc-shell-for-blup-ghc-8.10.7-env/lib/ghc-8.10.7/package.conf.d
    array-0.5.4.0
    base-4.14.3.0
    bla-0.1.0.0
    z-bla-z-blabla-0.1.0.0 -- this is blue
    deepseq-1.4.4.0
    ghc-boot-th-8.10.7
    ghc-prim-0.6.1
    integer-gmp-1.0.3.0
    pretty-1.1.3.6
    rts-1.0.1
    template-haskell-2.16.0.0
/home/mangoiv/.ghc/x86_64-linux-8.10.7/package.conf.d
    (no packages)

[MangoIV]

so the reason it does not look up the components.sublib is that when you're trying to add the library you need to be explicit about it and if not, it just assumes that it's a sublib?
Can this not be documented in all modules this knowledge is needed?

[MangoIV]

It seems the normal nixpkgs haskell infrastructure also can't handle sublibs.

[MangoIV]

NixOS/nixpkgs#191481

[MangoIV]

@hamishmack as you suggested I investigated the package-db that's generated by haskell.nix a bit more and it seems to be exactly the same if I compare it to the one generated when trying to build via cabal.project.
I have some suspicions atm:

• the abi hash being not what ghc expects
• the resolver running although it doesn't have to (can I make cabal skip the resolving step?)
• something that gets put into the ~/.cabal folder that's being skipped when building. (I have not investigated that yet)

[hamishmack]

I've done some digging too. Here are my notes (not sure how useful they are).

I add a cabal.project file so that I could use hix to test. hix is an easy way to test stuff without relying on the nix in the project directory at all (unless you add a nix/hix.nix file).

$ echo 'packages: ./bla ./blup' > cabal.project

Then I used hix to look in the project.plan-nix for the sublibs

$ grep sublib -r $(nix run github:input-output-hk/haskell.nix#hix-build -- -A plan-nix)
/nix/store/9w92fmiby9rvrb8qcasih0g1x455n7b1-haskell-project-plan-to-nix-pkgs/.plan.nix/blup.nix:            (hsPkgs."bla".components.sublibs.blabla or (errorHandler.buildDepError "bla:blabla"))
/nix/store/9w92fmiby9rvrb8qcasih0g1x455n7b1-haskell-project-plan-to-nix-pkgs/.plan.nix/bla.nix:      sublibs = {
/nix/store/9w92fmiby9rvrb8qcasih0g1x455n7b1-haskell-project-plan-to-nix-pkgs/default.nix:          "bla".components.sublibs."blabla".planned = lib.mkOverride 900 true;

So hsPkgs."bla".components.sublibs.blabla is in there.

As you pointed out the blap builds ok with nix:

$ nix run github:input-output-hk/haskell.nix#hix-build -- -A hsPkgs.blup.components.exes.blup

But in a shell for the just blup it fails:

$ echo 'packages: ./blup' > blup.project
$ nix run github:input-output-hk/haskell.nix#hix-shell -- --arg 'shell' '{ packages = p: [p.blup]; }' --run 'cabal build all --project-file blup.project'
...
Error: cabal: Could not resolve dependencies:
[__0] trying: blup-0.1.0.0 (user goal)
[__1] next goal: bla (dependency of blup)
[__1] rejecting: bla-0.1.0.0/installed-Hsuvz2e1aJzCilZPkGBZy8 (does not
contain library 'blabla', which is required by blup)
[__1] rejecting: bla-2009.10.20, bla-2009.10.13 (does not contain library,
which is required by blup)
[__1] fail (backjumping, conflict set: bla, blup)
...

Letting cabal rebuild the package bla like this works:

$ nix run github:input-output-hk/haskell.nix#hix-shell -- --arg 'shell' '{ packages = p: [p.blup]; }' --run 'cabal build all'

Leaves us with a dist-newstyle/packagedb/ghc-8.10.7/bla-0.1.0.0-inplace.conf and dist-newstyle/packagedb/ghc-8.10.7/bla-0.1.0.0-inplace-blabla.conf. Unfortunately I can't see anything in these that links bla to the sublib. Comparing them to the ones in ghc-pkg list for the look ok.

I'm not actually sure what mechanism cabal build all is using. Adding -v3 only gives useful output when work needs to be done and nothing in there jumps out.

And then there is this, "ghc-pkg doesn't work well with the v2 store because the v2 store basically uses a bunch of semi-documented/proprietary extensions to the package database that ghc-pkg hasn't been taught about". I worried sublibs (and probably backpack) are examples of those extensions.

[MangoIV]

Leaves us with a dist-newstyle/packagedb/ghc-8.10.7/bla-0.1.0.0-inplace.conf and dist-newstyle/packagedb/ghc-8.10.7/bla-0.1.0.0-inplace-blabla.conf. Unfortunately I can't see anything in these that links bla to the sublib. Comparing them to the ones in ghc-pkg list for the look ok.

The only thing that's suspicious afaict is that the sublib is marked blue up on ghc-pkg list which means that it is being ignored. I tried adding the sublib to the ghc-environment file with haskell.nix which worked but it didn't change anything about it being marked as ignored.

[MangoIV]

Something I was wondering, too: there's a cabal plan generated by haskell.nix which can be found in the maybeJson outpath of the plan-nix derivation. I wonder whether we can somehow pass that to cabal to stop making it do its own resolving. I haven't found a way though.

[MangoIV]

I also opened an issue on the cabal repo to find out about the inner workings of package resolving

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[michaelpj]

This is still an issue.

[andreabedini]

I started looking in this again. Just a couple of notes so far:

• Just to be clear, each cabal component is a ghc package.
• ghc-pkg displays a package with the blue colour if "hidden by default" (source, search for blue).
• In our example, cabal writes in the main library .conf file exposed: True while the sublibrary's .conf does not have that field.
• Looking at the code this is what cabal-install does in general for sublibraries (in Distribution.PackageDescription.Configuration.finalizePD). I will need to read-up a bit before I can understand the design.

[hamishmack]

Does haskell/cabal#6039 have to be fixed first?

[andreabedini]

Does haskell/cabal#6039 have to be fixed first?

It could be related but I cannot tell yet.

[andreabedini]

Ok, I read a bunch of code and yes, that issue is defintely the problem.

A bit of background first: cabal-install has two different and unrelated mechanism for "reusing already compiled packages".

One involves reusing packages already available in the packagedb. This is what happens when you have something that depends on a boot package (e.g. mtl). If the "pre-existing" version of mtl (and its dependencies) is the most preferable (and I haven't figured out how this "preference" works) then the solver will just use that package rather than recompiling a new one. A pre-existing package will appear in plan.json like this

    {
      "type": "pre-existing",
      "id": "mtl-2.2.2",
      "pkg-name": "mtl",
      "pkg-version": "2.2.2",
      "depends": [
        "base-4.14.3.0",
        "transformers-0.5.6.2"
      ]
    },

(Note that if you force a difference version of transformers, then the solver will have to recompile both transformers and mtl (and this is what causes headaches when you import ghc as a library, because this effectively fix all of the boot packages)).

A completely difference mechanism intervienes after the solver and it's the "store". Once the solver has made an install plan with the packages that should be compiled, cabal looks into the store and checks if any of those have already been compiled. It then it replaces packages in the plan with indentical packages already available in the store; making what gets called an improved plan. The packages that the solver wants to compile appear in plan.json as (type: "configured"). Note that plan.json describes the original plan, not the improved plan!

So, what happens is this: the cabal solver is not aware of packages with multiple libraries, and always ends up compiling them, but the cabal store makes this unnoticeable (given that none of the boot libraries have sublibs). But haskell.nix provides a global packagedb and not a cabal store, so cabal has to do what the solver says and recompiles those packages.

Note: the cabal store is implemented as packagedb, just like the global one. Nevertheless, it is used in a completely different way (i.e. it's not passed as a packagedb to the solver).

Note: one should probably mention that the solver itself does not know about components at all, but only about packages. Once the solver has made a plan with all dependencies at the package level, cabal "elaborates" the plan turning it into a plan between components.

I have a couple of ideas of how we can solve this in a simple way; but fixing cabal could also be a worthy endeavor.