Backpack for Stack

[ezyang]

Hello all,

I want to open up the discussion about how Backpack can be integrated with Stack; in particular, I'd be willing to work on and submit a patchset adding support for it in Stack. But before I do that, I want to hash out some general architectural concerns. The specification for the GHC and Cabal support for Backpack is here: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst ; please read it for more motivation on why you might like to support it.

The design process over two years lead us to a design where Backpack requires the package manager (Stack) to know how to instantiate packages that have required signatures in them (I spent the better part of six months trying to figure out how to do it all in GHC; if you are interested in the sordid details I can let you know). Things seem to work most smoothly if:

1. The package manager has a component oriented build model; that is, the unit of work in the install plan is a component rather than a package. (I don't believe Stack is architected in this way, although issues like stack 0.1.8 lost ability to build individual executables or library #1406 suggest it might be a good architecture for Stack anyway.) See also https://github.com/ezyang/ghc-proposals/blob/master/proposals/0000-componentized-cabal.rst for the proposal that added per-component configure to Cabal
2. The package manager knows enough to be able to finalize the package description (i.e., resolve all of its conditionals, getting a PackageDescription); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.

I'll note that private use of Backpack (i.e., the public library does not have any signatures) should be supported out-of-the-box, as there is no change to the outward-facing API of Setup scripts in this case (Stack will need to understand how to register multiple libraries produced by a Setup script, however). In the meantime, public libraries which have signatures would not be supported.

One of the major applications of Backpack I would like to see in the near future is the Backpack-ification of GHC's boot libraries to solve the string problem. Backpack would make it easy to write modules which are parametric over a particular desired string representation; the original package names would continue to export String-based interfaces, but versions using ByteString etc could be made available, and the availability of a package abstracted over holes means that an end-user could specify their own. Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.

Please let me know if you have any questions or concerns; I'd be happy to answer them.

CC @mgsloan, @snoyberg, @Blaisorblade , @borsboom , @harendra-kumar

[aljce]

This would be awesome.

[Blaisorblade]

@ezyang First let me thank you for all your work on this.

TL;DR. Does stack need this shipped tomorrow, in a month or in a year?

Before I can look into details and ask more substantive questions: is there a timeline for Backpack being merged/released/used, and an estimate how much work was required on cabal-install?

Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.

I agree this is probably essential. Having feature parity also matters—having to choose between stack and Backpack, or stack and sane strings, would be pretty bad.

[borsboom]

@mgsloan, I think you know the most about the parts of Stack that @ezyang is talking about. Once you're back from BM, can you weigh in?

@ezyang: how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code.

[ezyang]

Does stack need this shipped tomorrow, in a month or in a year?

Definitely not tomorrow or a month. If all goes well Backpack will get merged to GHC and Cabal, and people can start experimenting with it, but there were certainly be bugs and I won't claim that anyone should start using it for real immediately. Backpack'ing base is at least blocked on teaching GHC's build system how to deal, which will take some time and I am not planning for 8.2. On the other hand, base is probably the most plausible, widely used package that will get Backpack'ed, because (1) the potential upside is so great, and (2) it closely tracks GHC versions, so we don't have to worry about supporting old GHCs.

how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code.

There are some key library functions for configuring and instantiating packages, and cabal-install (new-build only; the legacy codepath doesn't support it) just calls out to them. At present, there are some things that are duplicated because I couldn't figure out an API that could be reused in both cases, but this is definitely a case where I can try harder.

The two key functions have these signatures:

type ConfiguredComponentMap =
        (Map PackageName (ComponentId, PackageId), -- libraries
         Map String ComponentId)                   -- executables
toConfiguredComponent
    :: PackageDescription
    -> ComponentId
    -> Map PackageName (ComponentId, PackageId) -- external
    -> ConfiguredComponentMap
    -> Component
    -> ConfiguredComponent

type LinkedComponentMap = Map ComponentId (UnitId, ModuleShape)
toLinkedComponent
    :: Verbosity
    -> PackageId
    -> LinkedComponentMap
    -> ConfiguredComponent
    -> LogProgress LinkedComponent

and a LinkedComponent has all the information to actually build with Backpack (the model is that every source package produces a configured component, which is mix-in linked into a linked component; then you go ahead and instantiate all the linked components getting the final graph of build products you need to build.)

[chrisdone]

The package manager knows enough to be able to finalize the package description (i.e., resolve all of its conditionals, getting a PackageDescription); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.

Stack does this. We use Cabal to parse the .cabal file but then resolve the GenericPackageDescription manually. Here is an example: https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L765 The whole of Stack.Package is pretty much dedicated to resolving package info. See also https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L198

[ezyang]

Great. Another question I have (since I am looking into this) is how difficult it would be to get Stack to support a per-component build model. I'm hoping that I can take anywhere we talk about 'Package' and actually it's talking about a 'Component', but I am not familiar enough with the code to say if this is the case or not.

[mgsloan]

@ezyang Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:

1. Dependencies vary based on whether --enable-tests or --enable-benchmarks is passed to Cabal, so this needs to be figured out on a package level
2. Avoiding package reconfiguration - best to configure all the components we want at once

And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'.

[ezyang]

OK. So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package (another thing where I'm not sure how well Stack will deal with this change.)

[ezyang]

I've been fishing around the codebase, and here are some initial architectural thoughts on what Backpack needs.

• Stack defines a simplified Package data type, which it reads out the contents of PackageDescription into. Backpack introduces some new fields, most notably mixins, which need to be preserved so that Backpack can figure out how packages are being instantiated. At the moment, Package doesn't really preserve any information like exposed modules which we would need to do this. It seems the most convenient way of keeping all the information Backpack needs is to keep PackageDescription verbatim in Package? This will also make it easier to call into Cabal to do mix-in linking.

• Once we have all the packages, we need to do a "mix-in" linking step to figure out exactly how packages have been instantiated. For example, if I have build-depends: regex-indef, str-string, I need to figure out that regex-indef's signature named Str was instantiated with the Str from str-string.

  It is an interesting question on how to integrate this with Stack. Stack's model is that it goes ahead and reads in all of the local source packages, but then lazily reads in the external package information using the accessor function created by withLoadPackage. The idea is that if a package is already installed, then we can avoid traversing its dependency graph. This should integrate reasonably well with Backpack: if a package is installed, we can read off the module shape (that's the metadata you need to mix-in link) from the installed package info; if it is not, we can compute it when we installPackage.

• Mix-in linking is not enough: after we figure out how everything is linked, we have to instantiate, creating a build task for every configuration by which a package was instantiated. In Stack, a build Plan consists of some number of Tasks to be done. I'm not sure how dependencies between tasks are computed, but each task is uniquely identified by the PackageIdentifier it provides (taskProvides). A Task roughly corresponds to a configure/build pair on Cabal. So, we need to modify taskProvides to be a bit more flexible: rather than only build a PackageIdentifier, it builds both a package identifier and an instantiation (to be defined shortly). Let's call such an identifier a UnitIdentifier, following the terminology I've used in Cabal and my thesis. Roughly speaking, these data structures should look like this:

  data UnitIdentifier = UnitIdentifier PackageIdentifier [(ModuleName, Module)]
  data Module = Module UnitIdentifier ModuleName
  

If Stack was modified in this way, would you take the patch? How would you like the design to be different? What is unclear?

[mgsloan]

Hey @ezyang , sorry for letting this languish, got caught up in other things. Passing this over to @snoyberg

[snoyberg]

I'm just coming up to speed on this now Edward. It seems relevant to work I'm already doing upgrading to Cabal 2.0. I'm also in favor of pushing Stack towards a component base build system. My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version.

I realize this issue is a bit dated now, and perhaps you have different thoughts. Maybe we could jump on a text chat and synch up?

[ezyang]

@snoyberg Sure thing; ping me on IRC some time? As far as I am concerned, the thinking on this ticket hasn't changed much (although maybe Stack's internals have!)

My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version.

Yeah, so in cabal new-build, this is done by treating these packages as non-per-component, single packages in the plan. It's a little bit of complexity but not too bad, and keeps BC.

[snoyberg]

Just chatted with @ezyang on IRC about this. Here's my summary:

1. We first need to get Cabal 2.0 support merged into Stack, which is on me (ETA: 2 weeks)
2. Once that's done, we will switch Stack to a per-component build plan. This is something that I'm very excited about in fact, see Specify components when configuring, not building haskell/cabal#2802
3. The biggest wrinkle is continuing support for pre-2.0 Cabals, which will be necessary for any LTS or Nightly release before GHC 8.2. Basic plan of attack is:
   1. Every build plan is calculated on a per-component basis
   2. If we're using Cabal 2.0 or later, we will also build on a per-component basis (separate configure with a separate dist dir for each and every component)
   3. If we're using pre-2.0 and there are no cycles in dependencies (which can happen in test suites and benchmarks), then we convert the per-component plan to a simple per-package plan and build
   4. If we're using pre-2.0 and there are cycles, we will convert into a staged per-package plan: any packages that have cycles will be split into lib/exe builds and full builds (including the tests and benchmarks). This is essentially the same as we have today, but can hopefully be streamlined by relying on the per-component plan logic. (Example use case: running stack test inside the bytestring package.)

I'll ping this issue with updates when Cabal 2.0 is merged in

[snoyberg]

Pinging appropriately: Cabal 2.0 support has just been merged to master :)

[mgsloan]

One thing for switching stack to a per-component build plan will also be adding support for internal libraries - #3361

Also support for foreign libraries, not sure if that's as relevant - #3364

[centromere]

What work needs to be done to finish adding Backpack support to stack?

[snoyberg]

There's still quite a bit of work to be done, see my comment above #2540 (comment). All that we've done so far is get support for Cabal 2.0 into Stack.

[ezyang]

I'm working on per-component build for Stack in https://github.com/ezyang/stack/tree/per-component-build

It doesn't build yet, but I've finally gotten it to typecheck to ConstructPlan, which means the real work can begin :)

[mgsloan]

Great, thanks @ezyang ! I skimmed it, and it looks good so far. Feel free to open a PR before its ready, as soon as you are ready to start getting review feedback.

[mikeplus64]

How's this come/coming along?

[ezyang]

Very, very slowly XD

[LeanderK]

any progress? 🙂
I don't want to be rude, it's just that I would really like to delve deeper into backpack. But not being able to integrate it into stack-projects is pretty discouraging (all of my projects use stack and there are also a lot out there 😕). I think it's a bit of a chicken or the egg problem, what comes first? Support or usage?

[stites]

I noticed that this ticket has a call for help via "A year into Backpack." Since this is needed for the hasktorch project to get into the LTSs, I've posted a bounty for this issue.

[matthew-piziak]

From @ezyang's blog.

In Stack issue #2540 I volunteered to implement Backpack support for Stack. However, over the past year, it has become abundantly clear that I don't actually have enough spare time to implement this myself. Looking for brave souls to delve into this; and I am happy to advise about the Backpack aspects.

Hey @ezyang, it was nice of you to volunteer, but when you realize you don't have time for a ticket it's good etiquette to say so in the ticket thread. That way people know it's unassigned and up for grabs. Don't lick the cookie—pass the plate!

You've done great work in characterizing this issue so far and I don't want to diminish that. But as they say, the last responsibility of a maintainer is to find a replacement, and it's good to do that in a centralized way. Many developers are on tenterhooks by this feature, so it's important that its status—viz. is somebody working on it / does it need a maintainer—is clearly advertised.

[mboes]

@matthew-piziak Cookie licking sounds pretty incongruous to this particular situation. @ezyang did a tremendous amount of work to actually get Backpack merged into GHC, and also on this particular ticket. This task isn't assigned to anyone and @ezyang makes it clear in his blog post that he'd be happy to see someone else pick this up.

[matthew-piziak]

@mboes You're right, I realized now that I had a poor understanding of the ticket's history. I apologize for the misleading language in my comment. In any case I am happy that this issue's status has been clarified.

[haitlahcen]

I don't know if its related but I faced the same issue (Invalid package ID) on a backpacked project and fixed it by adding + in the parser.

[cartazio]

this seems like a pretty egregious bug. why isn't it fixed?

[jkachmar]

@cartazio For what it's worth, backpack is currently not fully supported by any build tools outside of cabal-install. In light of this I wouldn't consider the lack of support for it within stack to be a particularly "egregious" bug, but rather a sign that there is limited interest in supporting this functionality beyond cabal-install.

For reference, please note the following tracking issues showing that support for backpack is not a particularly high priority elsewhere in the Haskell build tool ecosystem:

nixpkgs
rules_haskell
haskell.nix

[cartazio]

while i can empathize, i dont sympathize.
This is a bug in those tools, and perhaps they just haven't been motivated enough to take care of them.

[cartazio]

if they support ghc, this is a bug. If they just support haskell 2010, fine. but this means any post Haskell 2010 feature support is underspecified.

Do these tools support extensions/changes relative to Haskell 2010? What reference implementation is guiding them? I think you're saying "not ghc/cabal, but something more nebulous"

[jkachmar]

if they support ghc, this is a bug

I suppose my point here is that we've gone several years with all of the alternative build tools to cabal-install not supporting this feature, which (to me) means that regardless of whether it's considered to be a bug upstream it's certainly not a pressing problem for consumers of these tools.

Do these tools support extensions/changes relative to Haskell 2010? What reference implementation is guiding them?

My point was more that there doesn't seem to be much pressure on any build tools to support backpack, and there doesn't seem to be much negative backlash from the community for not having solved them (that rules_haskell issue is nearly two years old at this point).

To be fair, this is one of those tricky things in the space of open source community management. If the compiler implements a feature, but not enough people care to use it (or push for its adoption), is it really a bug not to support that?

EDIT:
This hadn't occurred to me when I wrote this up initially, but I think it's also important to consider this from the perspective that others may not consider a missing feature to be synonymous with a bug.

In the nixpkgs issue, you can see that the maintainer has even marked this functionality as an "enhancement".

[flip111]

it's certainly not a pressing problem for consumers of these tools

Most of the projects can move forward without backpack. Just that it offers really nice features which people like to use. "Not a pressing problem" doesn't justify the attention this issue got and the nearly $500 bounty on it

[cartazio]

I’m starting some work in progress evaluations on how / if folks such as backpack can dramatically improve the experience of both maintaining vector and making it easier for users to write their own factor families. It’s still up In the air if I can get it to work the way I want. But if it pays for itself in the ways I hope, the longest I’m going to sit on such a release to allow other tools to catchup is 6 months. I’m pretty confident that if I get all the pieces lined up that It can essentially just be a minor version bump of vector. But if so, I’ll have to give y’all a grace period to fix this bug. I guess this is your 6 month notice. Since waiting to be certain it pays off will just drag this ticket out further.

…

On Mon, Feb 3, 2020 at 1:14 PM flip111 ***@***.***> wrote: it's certainly not a pressing problem for consumers of these tools Most of the projects can move forward without backpack. Just that it offers really nice features which people like to use. "Not a pressing problem" doesn't justify the attention this issue got and the nearly $500 bounty on it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2540?email_source=notifications&email_token=AAABBQQ2GUDBLLN35GVIGNTRBBNJJA5CNFSM4CN45YHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKU22VA#issuecomment-581545300>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABBQSBFUPIVGEOSBJA5ODRBBNJJANCNFSM4CN45YHA> .

[cartazio]

to be clear: (and a stack contributor nicely pm'd me on IRC to help me clarify what i mean)

1. i'm the maintainer for vector
2. i'm exploring ways to make vector better for everyone (even users who just assume the api is gonna be the same always)
3. one direction that seems the most humane (vs commit war crimes of CPP or TH) is evaluating if backpack can support evolving vector while reducing the current material code footprint of the current vector code base (the Boxed/Storable/Unboxed vector apis are mostly the same/ share the same code, but with type specializations for improved inference and friendly UX for newcomers).

many packages depend on vector directly or indirectly. If vector starts using backpack, this impacts users of tools that dont support it.

If backpack is an effective tool to support evolving vector aggressively while maintaing super stable user interfaces and much more friendly "write your own new vector instances " workflow for users who want, idk, memmory mapped or gpu or tape-drive fusioned vector data types, i want to be able to do this. Or I want YOU to be able to this.

Many people use stack as a personal choice, and I dont want stack users or vector users to suffer. Accordingly it seems pretty important for backpack to be supported should this design mix pan out. Otherwise any library using vector directly or indirectly will not be buildable with stack.

[cartazio]

on a meta note: i was concerned about how certain folks would react to my input on that ticket, i'm disappointed to be exactly correct about the type of abuse certain people hurl on the twitters

[flip111]

Summarized and paraphrased information so far.

---

Using components (parts of a package): allows for an architecturally sound implementation of Backpack, as the process of instantiating requirements of a Backpack only makes sense for libraries, rather than an entire package (an executable and test suite cannot be instantiated in any meaningful sense).

---

Interaction with Backpack. Backpack needs to perform mix-in linking on components, and then a further expansion step to instantiate components. Mix-in linking occurs as we are expanding the package graph into a component graph, while instantiation happens as a separate step after expansion.

At the moment stack does not have a graph for components. "mix-in" relates to which types are used for a package from which other package. See: #2540 (comment)

---

There are some key library functions for configuring and instantiating packages, and new-build just calls out to them. The two key functions toConfiguredComponent and toLinkedComponent

and a LinkedComponent has all the information to actually build with Backpack (the model is that every source package produces a configured component, which is mix-in linked into a linked component; then you go ahead and instantiate all the linked components getting the final graph of build products you need to build.)

---

how difficult it would be to get Stack to support a per-component build model?

Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:

1. Dependencies vary based on whether --enable-tests or --enable-benchmarks is passed to Cabal, so this needs to be figured out on a package level
2. Avoiding package reconfiguration - best to configure all the components we want at once
   And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'.

Open question: Does stack know how to register multiple libraries produced by a setup script? --> If it can then mechanisms there could be re-used to do components (as one library would be one component)

---

So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package

An approach to start getting a proof of concept? Maybe better to have real components to get "architecturally sound implementation".

---

Best plan for implementation:

1. See point 3 in Backpack for Stack #2540 (comment)
2. Re-use types from https://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/Distribution-Backpack.html
3. Re-use or take inspirations from tests https://github.com/haskell/cabal/tree/master/cabal-testsuite/PackageTests/Backpack and dbaynard/stack@b8d8859...dbaynard:fb61d68
4. Take inspiration from latest attempt master...ezyang:per-component-build It (used to) typecheck ConstructPlan

---

Feel free to correct me :)

@ezyang what did you mean with "register" in your opening post?

[flip111]

@snoyberg can you explain how far this file has the type of component that is needed to support backpack ?? https://github.com/commercialhaskell/stack/commits/master/src/Stack/Types/NamedComponent.hs Is this in principle the same "component" but might need some enhancement to store backpack specific information ?

EDIT: also is it ok for stack to use types like https://github.com/haskell/cabal/blob/1a31242dfe1f0376051057370e25cc846e319ea8/Cabal/Distribution/Types/Component.hs

I'm not sure what parts of cabal can being pulled in or not

[Bodigrim]

Well, for what it's worth, I've been recently discussing with colleagues switching back from stack to cabal as our main build tool because of Backpack support.

Of course, there is not much pressure at the moment, because there are few Backpacked packages, because package authors avoid Backpack, because it cannot be built by one of two main Haskell build tools. It is a pity that packages cannot explore features, merged into GHC four versions ago. Changing this status quo would be a massive shift IMO.

On the other hand, this is open source: people are free to pop in, fix an issue and get away with praise/glory/bounty. It would be unfair to blame maintainers of stack for not catering everyone's needs immediately. Backpack support is no small thing.

[evanrelf]

I agree with @Bodigrim. Very few Haskell packages take advantage of Backpack because a major build tool (stack) doesn't support it. Which means there's little motivation for stack developers to add support. It's a negative feedback loop.

[sjakobi]

Note that there are several GHC bugs related to backpack, although I don't see anything too bad.

[flip111]

@ezyang are the OpenUnitId and DefUnitId from Distribution.Backpack serving the same function as the proposed types

data UnitIdentifier = UnitIdentifier PackageIdentifier [(ModuleName, Module)]
data Module = Module UnitIdentifier ModuleName

My guess would be that these "units" are not the same kind of units, because one includes PackageIdentifier and the other doesn't. Confirmation would be appreciated.

[flip111]

@ezyang

* rather than only build a `PackageIdentifier`, it builds both a package identifier and an instantiation (to be defined shortly). Let's call such an identifier a `UnitIdentifier`

Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ?

[flip111]

On how to blend backpack code with stack code ...

Searching in cabal's codebase for toLinkedComponent brings up the buildComponent function. Even if this function was on top level and being exported on first glance it seems there is too much cabal specific stuff in there.

On the one hand for stack's purposes it seems useful to do the expansion of tasks to build that is needed for backpack as late as possible. So after tasks are made expand them in yet even more tasks.

stack/src/Stack/Build/ConstructPlan.hs

Line 206 in a7a6c35

tasks = M.fromList $ mapMaybe toTask adrs

This prevents having to modify code earlier and thus having to carry changes through all code paths.

On the other hand toConfiguredComponent and toLinkedComponent need information that will only be available earlier than toTask so this information needs to be carried through anyway.

First step seems to be to figure out where the arguments from toConfiguredComponent come from and if they can be constructed (have data constructors exported) and where in the stack constructPlan this information is available. After that the same for toLinkedComponent.

After that see if the return value of toConfiguredComponent correspond to a type Stack has, and wether this return type (LogProgress ConfiguredComponent) can be embedded in a Stack type.

[bbarker]

I agree with @Bodigrim. Very few Haskell packages take advantage of Backpack because a major build tool (stack) doesn't support it. Which means there's little motivation for stack developers to add support. It's a negative feedback loop.

As a very new Haskell user, I was exposed to backpack (while still getting my bearings straight with regard to Haskell build tools), only to realize it would be a no-go if I wanted to use it with Stack. I've since been tempted to use it again, but realized the issue was still open. Had I been a user of backpack all along (which has only been a little over a year), I'm sure I'd have thought of many more places I could have used it. Just my 2c - I understand all too well that development time is limited.

[ezyang]

Sorry about the delay, I saw this in my GH notifications but sat on it for a week.

@flip111

what did you mean with "register" in your opening post?

referring to:

Stack will need to understand how to register multiple libraries produced by a Setup script

My recollection from the last time I looked at the Stack code, was that everything in Stack revolves around installing packages (there's a dep graph of packages, you install them one by one). The comment on private Backpack use is simply pointing out that sometimes installing one package results in multiple libraries getting installed, and as long as Stack can deal with this situation, all is good.

---

are the OpenUnitId and DefUnitId from Distribution.Backpack serving the same function as the proposed types

Yeah, they're not the same, precisely because the proposed type here refers to packages rather than components.

Components are a pretty big deal for usability of Backpack, because a lot of things you want to do involve having to write many small libraries and then recombining them together. If you'd have to make (and distribute) a separate package for each of these libraries, it would be a serious source of friction.

---

Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ?

It's fine to do it after depsolving; that's how Cabal-install does it. Depsolve the package structure, and then elaborate it into a more fine-grained component structure.

---

As for the last few comments, the kit in the Cabal library is what stack can usefully use, the stuff in cabal-install is a useful reference for how to use the APIs but I expect none of it is salvageable. That's OK; I have a "mini" implementation of the Cabal-install logic for Setup.hs and it was mostly fine.

To me, the problem was always how to drag Stack into the component world. That required more time investment learning about Stack that, sadly, I never found time to do in the end.

[cartazio]

@snoyberg is this on the todo queue of y’all ?

[theobat]

Following:

To me, the problem was always how to drag Stack into the component world. That required more time investment learning about Stack that, sadly, I never found time to do in the end.

I think for now, it's better to follow the progress here, for anyone interested:
#4745
Also, given that component-centric builds seem to require significant changes, my guess is backpack support in stack is unlikely to land before some time.