Manage a global package environment for easy GHCi access

[ezyang]

See https://www.reddit.com/r/haskell/comments/507h0p/cabal_newbuild_is_a_package_manager/d72bzce?context=3

I think, essentially, the request is for cabal new-install to put the requested package into some global packages set, which we keep up-to-date upon invocations to new-install. Maybe we should even write the package environment in the place GHC looks by default, so a bare ghci invocation works.

@hvr thought extra-packages might be what you want (if it were implemented) but I'm not sure; I think it would cause any such mentioned packages to be considered local, and thus built inplace. But maybe that's a misinterpretation.

CC @cocreature

[cocreature]

Is there any documentation on extra-packages? Having them treated as local packages doesn’t sound like what I want.

Personally I like viewing the global package set as just another project that is chosen if no other project is found. That removes the need for separate commands in the UI. Thinking about it at least the name extra-packages sounds like what I want, so just a list of packages that should be available in a project without them being declared as a dependency in a cabal file. For packages cabal new-install could just add the package to that list automatically and build it. If I am in a project it gets installed there or if I am somewhere else it gets installed into the global project.

[ezyang]

There is no documentation on extra-packages; but to its credit, they are also not implemented (only the syntax bits are in.) It's hard to say what the semantics were supposed to be... @dcoutts ? I do think your interpretation makes sense.

[dcoutts]

The motivation for extra-packages is things like ghc-vis, libraries that you want to have available in your development environment to use in ghci, but that is not a dependency of any package in your project.

So extra-packages are to be added into the project environment, solved for etc. They can come from the store or be built locally following the usual rule. They're additional to the ones automatically pulled in by deps of local packages. So where this really shows up is that you can use them as a target of build (e.g. if it's not built in the store already) and that they should go into the .ghc.environment file.

If the extra-packages is a exe then I guess it means (apart from building etc) just make it available in any managed shell environment or managed bin dir we might do.

[hvr]

If the extra-packages is a exe then I guess it means (apart from building etc) just make it available in any managed shell environment or managed bin dir we might do.

For consistency with the per-components framework, I'd expect extra-packages to build only library components (although maybe it should be renamed to extra-libs), and for when exe components are to be requested there should be a separate extra-tools directive (c.f. #3708)

[cocreature]

Reading @dcoutts’ description, extra-packages in combination with a global project that all commands default to if there is no cabal.project (or cabal file) seems like it would be exactly what I’m looking for.

[23Skidoo]

@dcoutts Do we plan to make cabal new-install foo update extra-packages in the global project?

[dcoutts]

@23Skidoo I'm not sure I'd totally nailed down what global install of libs should mean, but if we can make it behave more or less like a project then that would be easier to implement and might be a good UI as it'd follow the same mental model. So if we can say that the interpretation of install of libs user-wide is to update a virtual or real user-wide project with extra-packages and rebuild then great. But note that this implies full consistency, as with projects, and possibly rebuilding other packages that were previously installed to make them consistent (though at least we can do atomic switchovers so we don't bork things if rebuilding fails).

[dcoutts]

Oh and then note that this is not at all as simple as what Simon Marlow was suggesting in that same discussion which was to just blindly register the lib in the user db ("I know what I'm doing dammit!").

So I'd suggest that we proceed by implementing extra-packages for local projects and in parallel think about whether it fully works UI-wise to have the user-wide environment (ghc/ghci libs and exes) managed like a local project.

First thing to note there is that you don't want full consistency for deps of exes. I should be able to install an exe user-wide even if it doesn't have the same consistent deps as other user-wide binaries. So perhaps we should say that libraries available user-wide are managed like a project, but exes are done differently (or equivalently they're solved for independently aka --independent-goals but --independent-exe-goals only).

[ezyang]

You are right that there is a tension between (1) minimizing build time (if I say cabal new-install, I want the package to install ASAP) and (2) keeping a consistent database. Old cabal install always tried to minimize build time, by trying to use preexisting packages and being willing to install inconsistent things.

You might think that if you are willing to wait, it would always be right to keep the database consistent. But this might not be possible if a user wants to add two separate packages to their environment which don't have compatible dependencies. So inconsistency can be useful, even if it is not completely right. So, we could try the "global project" model, but it definitely sounds like letting each of the library roots be an independent goal would let more plans go through. Fortunately, I don't see why extra-packages couldn't be setup to allow independent goals, so you still ought to be able to use the project codepath. But maybe you want to warn when some of the dependencies are inconsistent!

[simonmar]

I'd like to suggest something really really simple: have cabal new-install for a library build and install the library into the store.

Additionally it would be nice to have a way to do ghci -package-db ~/.cabal/store/ghc-8.0.1/package.db via cabal, but I can always do that myself.

I'm sure you're thinking "but why would you do that? You'll get a random collection of packages". My point is that usually it does the right thing, and when it doesn't do the right thing I can give it a hint by adding a -package flag. I don't want to mess with global state or a global package environment or environment files or anything - just let me build a package and use it.

[hvr]

@simonmar You don't get a random collection of packages, but rather a forrest of install-plans into scope. But your suggestion would work fine, if you care only about a single package. However, if quickly testing something in GHCI is your interest, there's a related feature #3843 which has a similiar problem. So what about having a convenient command: cabal ghci --transitive-deps <list-of-deps> which throws you into a GHCi session with those packages in scope (possibly building them first), as well as their transitive deps (but in a consistent plan)?

EDIT: Just to clarify, I think the cabal new-install <library-package> suggestion is reasonable in its own right (I currently emulate it by creating a dummy package .cabal file, which contains a build-depends with the libraries I want to get populated into the nix store); my comment is rather about the 2nd part with ghci-invocation sugar.

[ezyang]

Well, I don't see any reason why we can't go ahead and implement @simonmar's suggestion. I don't think it will be that useful, but it would be pretty easy to do.

[phadej]

I realised on the plane: It would be useful, to tell cabal to only use packages in the store, or solver to consider only versions with tars cached. I had to reinvent constraints manually to make offline work possible :) in that light new-install will be useful to populate caches

[23Skidoo]

@phadej Do you know that cabal install has an --offline flag?

[phadej]

@23Skidoo but it doesn't work with new-build :(

[ezyang]

Yes, offline mode would be great. But it does make the process less deterministic (because the solver has to prefer versions for which we have local tarballs.)

[tom-bop]

For people who don't think this feature will be useful, can I clarify?: it's my understanding that, without using "old-install", the ghci command becomes effectively useless, except to toy with functions from the Prelude. Is that right?

[fgaz]

@tom-bop
Well, except that now there are ghc environment files. So if you invoke ghci in a project directory it will include the libraries your packages depend on.

new-install for libraries will basically be a global env file plus #4825 plus some a lot of consistency checking

[fgaz]

@simonmar #4825 does what you want

[tom-bop]

@fgaz #4825 is awesome, but is for nonlocal executables, and @simonmar and I are talking about nonlocal libraries. (Unless you're saying that #4825 additionally installs packages in the global store?)

[fgaz]

@tom-bop Yeah, it does. It installs the selected package in the store and symlinks its exes, if there are any. So you can use it to add libraries to the store too. You'll have to manually create a ghc environment file or explicitly pass -package* flags to ghci though.

[hvr]

@tom-bop @fgaz @simonmar you may want to take a look at https://gist.github.com/a076fbd34083a44faecd45ffd12b9772 which is a shell script I've been using to create package environment files (which exposes all transitive deps for the dependency spec specified at creation). It requires cabal 2.1 and GHC 8.0.2 (or later).

[fgaz]

@phadej duplicate of #6481?

[phadej]

@fgaz, if #6481 is accepted as "the way we want to proceed" then yes, otherwise no. So until we know what we want, I wouldn't close this.