-
Notifications
You must be signed in to change notification settings - Fork 20
Notes on migrating to a Go mono repo
Paul Jolly edited this page Jul 2, 2018
·
31 revisions
Ultimately this wiki will become a blog post, but maintaining here for now until things settle.
- Ultimately to migrate all
myitcv.io/...
packages to a mono-repo, housed at https://github.com/myitcv/x - This mono-repo should support
vgo
and, critically, be backwards compatible with "old"go get
- https://godoc.org/ should work as expected
- Have multiple modules within the mono-repo,
myitcv.io/react
being one of them. This will mean that we have submodules within the outermyitcv.io
module (although it's not strictly required to have a root module, I will use it as a catch-all for any packages belowmyitcv.io/
that are not submodules) - Drop any existing
_vendor
directories that were previously used to "cache" (dev) dependencies, e.g.myitcv.io/react/_vendor
- categorise issues/PRs in much the same way as the Go project does, e.g.
myitcv.io/react
issues would be prefixed byreact
,react/cmd/reactGen
etc
- Less overhead in terms of maintaining multiple repos, issue lists, CI setups etc
- ...
- It's a change from where we are today; loss of continuity, history etc (although the history of the old repos will remain intact in the original repo)
- ...
- The custom import paths for all
myitcv.io/...
packages will continue to be driven by https://myitcv.io, which is an extremely basic (and hacky) Google App Engine Go App (source code https://github.com/myitcv/myitcv.io). Once everything is migrated to the mono-repo we can remove the explicit package listing and tidy things up considerably - The
go-import
git
meta tag for allmyitcv.io/...
packages needs to be the root of all packages,myitcv.io
in this case:
$ curl https://myitcv.io/gogenerate?go-get=1
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="myitcv.io git https://github.com/myitcv/x">
<meta name="go-source" content="myitcv.io https://github.com/myitcv/x/wiki https://github.com/myitcv/x/tree/master{/dir} https://github.com/myitcv/x/blob/master{/dir}/{file}#L{line}">
<meta http-equiv="refresh" content="0; url=https://godoc.org/myitcv.io/gogenerate">
</head>
<body>
Redirecting to docs at <a href="https://godoc.org/myitcv.io/gogenerate">godoc.org/myitcv.io/gogenerate</a>...
</body>
</html>
-
vgo
works without needing to publish modules (see "Publishing" section https://research.swtch.com/vgo-module).However, per https://github.com/golang/go/issues/24687#issuecomment-379056477 there appear to be some issues within submodules when resolved from VCS for custom import paths.I am also publishing pre-release modules using https://github.com/myitcv/pubx. Thego-import
mod
meta tag is then used to refer to these published packages:
$ curl https://myitcv.io/blah2?go-get=1
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="myitcv.io git https://github.com/myitcv/x">
<meta name="go-import" content="myitcv.io/blah2 mod https://raw.githubusercontent.com/myitcv/pubx/master">
<meta name="go-source" content="myitcv.io https://github.com/myitcv/x/wiki https://github.com/myitcv/x/tree/master{/dir} https://github.com/myitcv/x/blob/master{/dir}/{file}#L{line}">
<meta http-equiv="refresh" content="0; url=https://godoc.org/myitcv.io/blah2">
</head>
<body>
Redirecting to docs at <a href="https://godoc.org/myitcv.io/blah2">godoc.org/myitcv.io/blah2</a>...
</body>
</html>
It appears however that the oldgo get
does not like there being twogo-import
meta tags which satisfy amyitcv.io/...
package. Raised as https://github.com/golang/go/issues/24751The issue with the dual-meta tags also affects https://godoc.org/, because it (https://godoc.org/) usesgo get
in order to access code. Otherwise, however, things appear to work well: https://godoc.org/myitcv.io/blah. This has been raised as https://github.com/golang/gddo/issues/558- It seems my approach to migrating package-by-package is somewhat flawed. The "issue" is me creating a mono-repo that has as its effective import path
myitcv.io
. Because unless I move everything over in one go, thenvgo
orgo
both have to handle a situation where there is one repo that is rooted atmyitcv.io
and others that are rooted one level deeper. Andvgo|go
understandably can't handle that- This actually has the consequence that a package in the mono-repo that depends on another package in the mono-repo has to do so via a
replace
directive until the "flip" is complete. e.g.immutable
- Also, once the "flip" is complete I will need to drop the exceptions from this
.gitignore
, thereby ignoring everythingmyitcv.io/...
- This actually has the consequence that a package in the mono-repo that depends on another package in the mono-repo has to do so via a
-
vgo test ./...
at the root modifies the rootgo.mod
; related to https://github.com/golang/go/issues/26048 -
vgo install pkg
installs toGOBIN
using the module version suffix; https://github.com/golang/go/issues/24667
- Much like the original motivation for committing
_vendor
to my repos, I want my CI to behave in a deterministic way. But I don't want to have to commit all those dependencies to my mono-repo (which would massively burden users of my code). Hence I am maintaining https://github.com/myitcv/cachex. The CI then uses https://github.com/myitcv/cachex augmented with https://github.com/myitcv/pubx formyitcv.io/...
packages as itsGOPROXY
. This means if a dependency is missing fromcachex
orpubx
then the CI build will fail. It is worth noting that that both https://github.com/myitcv/cachex and https://github.com/myitcv/pubx are in effect append-only repositories (although I don't currently have anything that enforces this - see TODO) - To simplify the process of publishing modules I have developed
myitcv.io/cmd/modpub
. This takes a clone of agit
repo and a target directory and "publishes" modules within thegit
repo into the target directory. This target directory is then pushed to Github as https://github.com/myitcv/pubx, per above - Much like others, I needed to use a Personal Access Token to avoid Github rate limits. For more details see the Github help article (no scopes required) and the appropriate line in my test runner
All issues listed above have been raised in the Go repo
In addition, I link here to a number of interesting issues/proposals:
-
Proposal: cmd/go: allow "go run" without arguments - https://github.com/golang/go/issues/22726Would work particularly well withvgo
And would be particularly useful ingo:generate
directives- Effectively parking because of https://github.com/golang/go/issues/25416 - i.e.
go run pkg
will never be as fast as running ago install
-ed program. Hence we will writegr
to wrapgo list -export
to determine whether we need to build a new program binary, then cache it and use the cache entry ID as effectively the cache key - Also raised https://github.com/golang/go/issues/25922 to discuss the question of tooling, which happens to be very related
- Not required, but useful, the ability to list all available versions of a module https://github.com/golang/go/issues/25656
- x/vgo: integrate with guru, goimports, etc - https://github.com/golang/go/issues/24661
- Would allow vetters/linters/tools that require type information to access that from the build cache
- Therefore applies to
gocode
et al. Per @rsc, for Go 1.10 it suffices to usevgo install
of the packages - Per this golang-dev thread,
vet
already has these powers. Other tools need these powers in avgo
world - An initial cut of this a solution has been created as
myitcv.io/vgoimporter
- Until a
go/types
-dependent tool is converted to bevgo
-aware however it will fail. e.g.stringer
. More specifically,go/importer
(which is typically used bygo/types
-dependent tools) is notvgo
-aware, and hence when it tries to resolve an import via the old$GOPATH/pkg
route it will fail becausevgo
uses the build cache. And this is whatmyitcv.io/vgoimporter
does; uses the relevant build files for import-resolution. - The ultimate solution here from an importer perspective is being discussed here https://github.com/golang/go/issues/14120#issuecomment-383994980
- Getting module import information from
vgo list -json
- https://github.com/golang/go/issues/24042- Just generally useful
- Use of
internal
packages/modules: https://github.com/golang/go/issues/25164
- Do we need to have a
go.mod
file at the repo root? i.e. https://github.com/myitcv/x/blob/master/go.mod
- Enforce https://github.com/myitcv/cachex and https://github.com/myitcv/pubx being append-only repositories