all: dividing go-cloud into more than one module

[jba]

Currently, this repo consists of a single module. At some point, we will want to take some packages beta (that is, v1), while leaving others at v0. That implies that we'll have at least two modules.

This issue tracks the decisions involved in using more than one module. It assumes we have a "vanity" import path, as indeed we soon will (gocloud.dev/...). With a vanity path, we have the option using more than one repo. For example, the import path gocloud.dev/blob could point to a repo just for blob (say, github.com/google/go-cloud-blob) while the import path gocloud.dev/pubsub could continue to refer to github.com/google/go-cloud/pubsub.

The disadvantages of using more than one repo are:

• Administrative overhead to obtain each additional repo (a one-time cost)
• Development overhead in maintaining multiple repos
• The inability to make commits atomically across repos

The advantage is that we would avoid the problems with multi-module repos. There are two such problems:

1. Developing code in a multi-module repo is tricky. The main problem is that git works in terms of commits—your working directory is always "at" some single commit—but in your program, each module in your repo may be at a different version (and therefore at a different commit). At present, there is no clean, supported way to manage this. See cmd/go: support simultaneous edits of interdependent modules golang/go#27542 for discussion.

2. If one module is in a subdirectory of another, a "requires" cycle is needed. That complicates testing, and also, go mod tidy currently removes the cycle. See cmd/go: ensure that 'go mod tidy' and go get -u do not introduce ambiguous imports golang/go#27899.

Note that the first of these is a problem even if the modules are siblings.

We should monitor those two issues for improvements to the situation, and we should defer any decision on multiple modules as long as possible in the hope that things will get better.

[tv42]

Another argument would be having fewer dependencies pulled in. At least with Go modules on go1.12rc1, the list of packages that had their metadata downloaded was very long, and the delay on my current LTE internet connection was long enough that I seriously considered not using this library and hardcoding the exact blob provider I plan to use.

[eliben]

I will take a stab at evaluating this, seeing if we can split off some of the heaviest dependencies first.

[tamalsaha]

In the process, if you can divide this into multiple repos, that will be really great! For example, I am primarily interested in the blob store pkg. That will make it simpler for me or anyone else, I suppose.

[eliben]

@tamalsaha I'm curious to hear about your use case. Why would multiple repos simplify things for you? Are you worried about the dependencies pulled when go get-ing the CDK, or something else?

[tamalsaha]

Are you worried about the dependencies pulled when go get-ing the CDK, or something else?

• Yes.

[eliben]

Are you worried about the dependencies pulled when go get-ing the CDK, or something else?

* Yes.

This is precisely what splitting into multiple modules attempts to address. AFAIU you shouldn't much care if it comes from the same github repo, as long as you only pull the dependencies you want.

[tamalsaha]

Yeah. Though I read this from @FiloSottile https://twitter.com/FiloSottile/status/1096165600890707973

[FiloSottile]

My opinions on multi-module repositories are mostly informed by the opinions of @ianthehat, @bcmills and @jayconrod. Multi-repo (what's proposed here) is probably stricly better than multi-module repos.

Another argument would be having fewer dependencies pulled in.

This part will be mostly solved by proxies in Go 1.13, because only the go.mod file will be fetched for dependencies that you don't actually need to build.

[eliben]

Fixed with #2189

[vangent]

\o/ thanks for all the hard work on this Eli!