PROPOSAL: Publish haddock documentation for all CHaP packages.

[jonathanknowles]

Proposal

Publish versioned Haddock documentation for all CHaP packages, in the style of Hackage.

Terminology

In the context of this proposal, the terms "documentation" and "API documentation" mean "Haddock API documentation for a Haskell package", as built with the Haddock tool.

Motivation

Lower the barrier of entry for developers wishing to:

• consume packages in the Cardano Haskell ecosystem.
• author packages for the Cardano Haskell ecosystem.

In particular, a single, integrated place for API documentation would mean that:

1. Developers wishing to understand existing packages can read API documentation:

   • without having to hunt through source code;
   • without having to download packages manually;
   • without having to build documentation manually.

2. Developers wishing to understand dependencies of existing packages can reach and browse the API documentation for those packages just by following ordinary hyperlinks.

3. Developers wishing to author or maintain packages can publish their API documentation:

   • without having to independently create and maintain their own infrastructure for building and hosting documentation.

The last point is of particular interest to those who wish to author libraries for the Cardano Haskell ecosystem. Under this proposal, a library author could upload their package to CHaP, and then take advantage of the shared documentation publication facilities that CHaP provides, saving a potentially large amount of duplicated effort.

Proposed acceptance criteria

☑️ 1. Publication of documentation for new package versions

After a user adds a new version v of package p to CHaP, then CHaP should:

1. Attempt to compile the documentation for version v of package p;

2. Assuming compilation is successful, publish it at the following URL:

   https://chap.intersectmbo.org/package/p-v

   The above URL scheme is similar to the one used by Hackage:

   https://hackage.haskell.org/package/p-v

Documentation should be published within a reasonable time frame, where "reasonable" can be interpreted as "similar to the length of time that Hackage would require to build and publish documentation".

💡 Example

Alice adds version 1.2.3.4 of package cardano-moon-launcher to CHaP.

CHaP then publishes the documentation for this package at the following URL:

https://chap.intersectmbo.org/package/cardano-moon-launcher-1.2.3.4

☑️ 2. Preservation of documentation for historical package versions

CHaP should preserve all documentation that is already published for historical versions of packages.

In particular, if CHaP already has published documentation for version v1 of package p, then uploading version v2 of the same package should not cause the documentation for v1 to become unavailable.

This is similar to Hackage, which preserves documentation for historical versions of packages.

💡 Example

Bob adds version 1.0.0.0 of package cardano-moon-base to CHaP.

CHaP publishes documentation at the following URL:

• https://chap.intersectmbo.org/package/cardano-moon-base-1.0.0.0

Later on:

Carol adds version 2.0.0.0 of package cardano-moon-base to CHaP.

CHaP publishes documentation at the following URL:

• https://chap.intersectmbo.org/package/cardano-moon-base-2.0.0.0

Documentation for version 1.0.0.0 is still available at the original URL.

☑️ 3. Hyperlinks between CHaP packages

Assuming:

• p1 and p2 are both packages published on CHaP;
• the public interface of p2 includes one or more usages of a symbol s exported from the public interface of p1;
• symbol s is an entity that Haddock can reference with a hyperlink;

Then usages of s within the published documentation for p2 should have hyperlinks that resolve to the definition of s within the published documentation for p1.

💡 Example

David publishes package cardano-tx to CHaP. This package exports a data type called Tx:

module Cardano.Tx
    ( Tx )
    where

data Tx = ...

Documentation for this package becomes available at the following location:

• https://chap.intersectmbo.org/package/cardano-tx

Later on, Emily publishes package cardano-tx-merge to CHaP. This package exports the function mergeTxs, whose type signature depends on the Tx data type:

module Cardano.Tx.Merge
    ( mergeTxs )
    where

import Cardano.Tx
    ( Tx )

data TxMergeError = ...

mergeTxs :: Tx -> Tx -> Either TxMergeError Tx
mergeTxs = ...

Documentation for this package becomes available at the following location:

• https://chap.intersectmbo.org/package/cardano-tx-merge

When readers view the published documentation for cardano-tx-merge, they are able to follow hyperlinks from usages of Tx in cardano-tx-merge to the definition of Tx in the published documentation for cardano-tx.

☑️ 4. Hyperlinks from CHaP packages to Hackage packages

Assuming:

• p1 is a package published on Hackage;
• p2 is a package published on CHaP;
• the public interface of p2 includes one or more usages of a symbol s exported from the public interface of p1;
• symbol s is an entity that Haddock can reference with a hyperlink;

Then usages of s within the published documentation for p2 on CHaP should have hyperlinks that point to the definition of s within the published documentation for p1 on Hackage.

💡 Example

Fred creates a package called cardano-utxo that depends on the containers package from Hackage.

The cardano-utxo packages exports functions whose type signatures include the Map data type:

module Cardano.UTxO
    ( UTxO, fromMap, toMap )
    where

import Data.Map.Strict
    ( Map )

fromMap :: Map TxIn TxOut -> UTxO
fromMap = ...

toMap :: UTxO -> Map TxIn TxOut
toMap = ...

Fred publishes cardano-utxo to CHaP, which publishes documentation at the following location:

• https://chap.intersectmbo.org/package/cardano-utxo

When readers view the published documentation for the Cardano.UTxO module, they are able to follow hyperlinks from usages of Map to its definition in the published documentation for containers.

[michaelpj]

Note that foliage already generates a static site. So it would be nice to incorporate it into that.

Our CI already builds (most) packages, so I don't think it would be too expensive to do 1 or 2.

The hard part is building it, i.e. 3&4. I don't know how hackage-server does it, but ideally we'd just do what they do. With Nix. Or something.

[michaelpj]

Also this is obviously massively desirable, it's just quite non-trivial.

[angerman]

I always wondered if we could use some sphinx cross ref logic. That should allow some federated links between different hackages as well. It might be a bit far out though.

[michaelpj]

I actually wrote some stuff for this, it worked quite well: https://github.com/michaelpj/sphinxcontrib-haddock

[andreabedini]

Hi @jonathanknowles, thanks for writing this up. I agree that everything you list here is desirable. Indeed we have already been building haddocks for all packages in our hydra ci.

https://ci.iog.io/jobset/input-output-hk-cardano-haskell-packages/main/latest-eval?filter=haddock

The problem that has always stopped me from pushing this further is this: haddock generated documentation depends on the exact build configuration. This include ghc version, platform, dependencies, pkg-config, etc.

What we do in hydra at the moment is building haddock idependently for each single package component. This complicates cross linking: the lastest version of pkg-a might not be able to use the latest version of pkg-b; so the links from pkg-a must be to pkg-b-$(prior-version) rather than to pkg-b-$(latest-version). Since we build and keep all version we could find a way to make this work, but platform dependence could give us more headaches.

If you want to work on this, I would be very happy to support your efforts. The URL schema you can use is

https://ci.iog.io/job/input-output-hk-cardano-haskell-packages/main/${system}.${compiler-nix-name}.${package-name}."${version}".${qualified-component-name}:haddock/latest/download-by-type/doc/haddock

e.g.

https://ci.iog.io/job/input-output-hk-cardano-haskell-packages/main/x86_64-linux.ghc96.cardano-api.%228.36.1.1%22.cardano-api:lib:cardano-api:haddock/latest/download/1

Building the documentation for the whole package rather than by components would be a good first step. Perhaps it's trivial, I admit I haven't looked at this for a while.

[michaelpj]

Yes, this is one reason why I wonder how hackage-server does it. They must surely have some similar issues...

[coot]

Ad 3. & 4. I recently stumbled on a similar problem with cabal haddock-project --hackage (see haskell/cabal#9852). When building the documentation one needs to know whether a given dependency is on hackage or CHaP (in the haddock-project is whether its a local or hackage package); nix might know this, and then it will need to pass the right --read-interface options to haddock.