InstalledPackageInfo mega issue

[andreabedini]

There are a bunch of issues and PRs calling for adding more information to InstalledPackageInfo. I thought it would be a good idea to collect them here with a short summary so we can have better visbility and, if needed, discuss the bigger picture.

• Encode flag setting in InstalledPackageInfo? #8702 by @bgamari. The solver does not know what flags have been used to compile an installed package, and it won't recompile an installed package even if we request it with different flags.
• extra-bundled-libraries shouldn't use hsLibraries #8701 by @bgamari. Cabal should use a separate field for extra-bundled-libraries as hsLibraries is deemed inapproriate.
• Track static vs. dynamic dependencies #8624 and Extend the InstalledPackageInfo record with 2 fields for artifacts. #8696 by @bairyn. Similar to the first issue, solver does not know about artifact and could end up choosing an installed package that is missing some files (I admit I don't have a good understanding of this issue, please correct me if I botched this summary).
• Make the cabal-install solver multilibs-aware #6039 by @fgaz. Sublibraries are installed as separate packagedb entries, and the solver has no information to group them back together into a coherent source package configuration. This causes the solver to always reject and recompile installed packages that provide public sublibraries (as the installed package is seen as missing the sublibrary)
• RFC consider including identifying infos from ghc --info in nix-style hash #5116 by @hvr. It suggests adding something to distinguish two GHC HEAD with the same version. Not sure whether this is still relevant.
• Optionally add the revison number to Paths_$pkgid.version #1506 by @23Skidoo. It suggests adding information about cabal file revisions to InstalledPackageInfo, deemed a bad idea by @dcoutts and led to Add a Version_ module which does not depend on install dirs #3909 which was implemeted in Add PackageInfos_ (#3909) #8534. I think this can just be closed.

AFAIU Changing InstalledPackageInfo requires some level of coordination with GHC since ghc-pkg uses Cabal to read the files and distill them into a GHC specific version. What is the process to make changes to InstalledPackageInfo?

Is it a good idea to keep adding cabal specific things to InstalledPackageInfo? Other tools might install packages in packagedb.

Should cabal install have its own database?

Should we have a clear distinction between ghc fields and tool specific fields?

In #8702 @bgamari refers to "non-normative" fields (which I have not found any documentation for) and suggests a syntax for fields ghc-pkg will automatically ignore (x-field).

Ping spamming @ulysses4ever @fgaz @Mikolaj @gbaz @hsyl20 and others (apologies in advance).

[Mikolaj]

x-field sounds fine. Would establishing this convention (with a buy-in from GHC) solve all the other problems?

[andreabedini]

x-field sounds fine. Would establishing this convention (with a buy-in from GHC) solve all the other problems?

Should it be x-cabal- for cabal specific fields?

Package metadata (author, etc) are not used by GHC, should they get the x-cabal- prefix? why are they even needed in packagedb?

Sorry for bombarding with questions, I don't want to led us to analysis-paralisys :-) I want to understand the design space a bit more.

[bgamari]

Thanks for putting together this list, @andreabedini !

An X- field convention sounds quite reasonable to me and reasonably straightforward to implement. Yes, I think it would be best if users prefixed tool/library-specific fields with their name (e.g. X-Cabal).

Package metadata (author, etc) are not used by GHC, should they get the x-cabal- prefix? why are they even needed in packagedb?

Indeed GHC doesn't care about these fields but they are fairly universal and I don't think they do much harm to include. Moreover, some of these fields may be used by tools other than GHC (e.g. license for license compliance checking).

[bairyn]

The Cabal-syntax package is the one inside cabal.git that provides the InstalledPackageInfo record.

data InstalledPackageInfo
	= InstalledPackageInfo {
…
		author            :: !ShortText,
		stability         :: !ShortText,
		homepage          :: !ShortText,
		pkgUrl            :: !ShortText,
		synopsis          :: !ShortText,
…
		depends           :: [UnitId],
		abiDepends        :: [AbiDependency],
		ccOptions         :: [String],
…
	}
	deriving (Eq, Generic, Typeable, Read, Show)

In ghc.git, there is a utility application called ‘ghc-pkg’ whose source is largely in ghc.git/utils/ghc-pkg/Main.hs. ghc-pkg depends on Cabal-syntax to have access to Cabal-syntax's InstalledPackageInfo. cabal-install also depends on Cabal-syntax.

I think x- sounds fine at least if RFC 6648's warning is accommodated. We could add a list of external/extensible ‘x-’ fields as a new field in InstalledPackageInfo, and, secondly, any new field inside InstalledPackageInfo could optionally itself get an ‘x-’ prefix if it isn't wanted to be added without ‘x-’. It technically wouldn't be conformant with that RFC, for what it's worth, since it has the opinion that an ‘x-’ convention makes future standardization of the name difficult. But I think we can still allow for future standardization, by adding a Cabal-syntax version field to the IPI record and by using the Haskell PVP. Then if support for older Cabal-syntax versions and their IPI records is desired (e.g. in particular if ‘x-’ is removed for an experimental (rather than external/extensible field) once a new major version is released), newer versions could look at what the IPI version is, to make adjustments similar to the various migrations in e.g. cabal-install:D.C.Setup.filterConfigureFlags, which adjusts flags depending on the cabal version.

Perhaps the main solution is adding a version field, then, enabling the IPI record serialized into the package's .conf file to be tied to the Cabal-syntax version that specifies it. This would make any ‘x-’ experimental standardization easier in the future. Then perhaps a list of x- fields can be added, for external users of Cabal-syntax to use to extend IPI. Cabal-syntax would store these key value pairs uninterpreted in InstalledPackageInfo, for external users to manage.

Does this sound reasonable?

[andreabedini]

@bairyn Thanks for chiming in and also for mentioning RFC 6648.

I am actually reluctant to go down the x- route since I believe it will keep masking a natural separation of concerns. If you look at ghc-pkg, you will see that only a small part of IPI is actually used by GHC. Which again brings the question of why we have packagedb, who owns it and who uses it.

You could say it's a Cabal thing that GHC reads, but then why ghc-pkg can modify it?
You could say it's a GHC thing that Cabal writes, but then why IPI is defined in Cabal-syntax and why it includes details irrelevant to GHC?
You could say it's a common component both Cabal and GHC depends on and use to communicate; then it should be moved to a separate library along with (most of) ghc-pkg and it should take into account being used by other tools too.

TBH I am still confused by the whole situation.