-
Notifications
You must be signed in to change notification settings - Fork 20
Notes on migrating to a Go mono repo
Paul Jolly edited this page Apr 10, 2018
·
31 revisions
Ultimately this wiki will become a blog post, but maintaining here for now until things settle.
- Ultimately to migrate all
myitcv.io/...
packages to a mono-repo, housed at https://github.com/myitcv/x - This mono-repo should support
vgo
and, critically, be backwards compatible with "old"go get
- https://godoc.org/ should work as expected
- Have multiple modules within the mono-repo,
myitcv.io/react
being one of them. This will mean that we have submodules within the outermyitcv.io
module - Drop any existing
_vendor
directories that were previously used to "cache" (dev) dependencies, e.g.myitcv.io/react/_vendor
- categorise issues/PRs in much the same way as the Go project does, e.g.
myitcv.io/react
issues would be prefixed byreact
,react/cmd/reactGen
etc
- Less overhead in terms of maintaining multiple repos, issue lists, CI setups etc
- ...
- It's a change from where we are today; loss of continuity, history etc (although the history of the old repos will remain intact in the original repo)
- ...
- The custom import paths for all
myitcv.io/...
packages will continue to be driven by https://myitcv.io, which is an extremely basic (and hacky) Google App Engine Go App (source code https://github.com/myitcv/myitcv.io). Once everything is migrated to the mono-repo we can remove the explicit package listing and tidy things up considerably - The
go-import
git
meta tag for allmyitcv.io/...
packages needs to be the root of all packages,myitcv.io
in this case:
$ curl https://myitcv.io/gogenerate?go-get=1
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="myitcv.io git https://github.com/myitcv/x">
<meta name="go-source" content="myitcv.io https://github.com/myitcv/x/wiki https://github.com/myitcv/x/tree/master{/dir} https://github.com/myitcv/x/blob/master{/dir}/{file}#L{line}">
<meta http-equiv="refresh" content="0; url=https://godoc.org/myitcv.io/gogenerate">
</head>
<body>
Redirecting to docs at <a href="https://godoc.org/myitcv.io/gogenerate">godoc.org/myitcv.io/gogenerate</a>...
</body>
</html>
-
vgo
should work without needing to publish modules (see "Publishing" section https://research.swtch.com/vgo-module). However, per https://github.com/golang/go/issues/24687#issuecomment-379056477 there appear to be some issues within submodules when resolved from VCS. Hence in the meantime (and indeed for the longer term) I am publishing pre-release modules via https://github.com/myitcv/pubx. Thego-import
mod
meta tag is then used to refer to these published packages:
$ curl https://myitcv.io/blah2?go-get=1
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="myitcv.io git https://github.com/myitcv/x">
<meta name="go-import" content="myitcv.io/blah2 mod https://raw.githubusercontent.com/myitcv/pubx/master">
<meta name="go-source" content="myitcv.io https://github.com/myitcv/x/wiki https://github.com/myitcv/x/tree/master{/dir} https://github.com/myitcv/x/blob/master{/dir}/{file}#L{line}">
<meta http-equiv="refresh" content="0; url=https://godoc.org/myitcv.io/blah2">
</head>
<body>
Redirecting to docs at <a href="https://godoc.org/myitcv.io/blah2">godoc.org/myitcv.io/blah2</a>...
</body>
</html>
- It appears however that the old
go get
does not like there being twogo-import
meta tags which satisfy amyitcv.io/...
package. Raised as https://github.com/golang/go/issues/24751 - The issue with the dual-meta tags also affects https://godoc.org/, because it (https://godoc.org/) uses
go get
in order to access code. Otherwise, however, things appear to work well: https://godoc.org/myitcv.io/blah
- Much like the original motivation for committing
_vendor
to my repos, I want my CI to behave in a deterministic way. But I don't want to have to commit all those dependencies to my mono-repo (which would massively burden users of my code). Hence I am maintaining https://github.com/myitcv/cachex. The CI then uses https://github.com/myitcv/cachex augmented with https://github.com/myitcv/pubx formyitcv.io/...
packages as itsGOPROXY
. This means if a dependency is missing fromcachex
orpubx
then the CI build will fail. It is worth noting that that both https://github.com/myitcv/cachex and https://github.com/myitcv/pubx are in effect append-only repositories (although I don't currently have anything that enforces this - see TODO) - To simplify the process of publishing modules I have developed
myitcv.io/cmd/modpub
. This takes a clone of agit
repo and a target directory and "publishes" modules within thegit
repo into the target directory. This target directory is then pushed to Github as https://github.com/myitcv/pubx, per above - Much like others, I needed to use a Personal Access Token to avoid Github rate limits. For more details see the Github help article (no scopes required) and the appropriate line in my test runner
All issues listed above have been raised in the Go repo
In addition, I link here to a number of interesting issues/proposals:
- Proposal: cmd/go: allow "go run" without arguments - https://github.com/golang/go/issues/22726
- Would work particularly well with
vgo
- Would work particularly well with
- x/vgo: integrate with guru, goimports, etc - https://github.com/golang/go/issues/24661
- Would allow vetters/linters/tools that require type information to access that from the build cache
- Therefore applies to
gocode
et al. Per @rsc, for Go 1.10 it suffices to usevgo install
of the packages - Per this golang-dev thread,
vet
already has these powers. Other tools need these powers in avgo
world - An initial cut of this a solution is being worked on in https://github.com/myitcv/x/pull/10
- Getting module import information from
vgo list -json
- https://github.com/golang/go/issues/24042- Just generally useful
- Do we need to have a
go.mod
file at the repo root? i.e. https://github.com/myitcv/x/blob/master/go.mod
- Enforce https://github.com/myitcv/cachex and https://github.com/myitcv/pubx being append-only repositories