cmd/go: rough UX for developing multiple modules in one repository

[vllry]

What version of Go are you using (go version)?

$ go version
go version go1.14.1 windows/amd64

Does this issue reproduce with the latest release?

Untested.

What operating system and processor architecture are you using (go env)?

Windows 10, amd64.

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\wcecs\AppData\Local\go-build
set GOENV=C:\Users\wcecs\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\wcecs\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\wcecs\AppData\Local\Temp\go-build982679850=/tmp/go-build -gno-record-gcc-switches

What did you do?

I encountered a frustrating scenario with Go Modules, while helping another Go programmer. I’m filing an issue because we feel it exposed some rough edges in Go Modules behaviour.

We have a private repository containing a Go gRPC server (“X”) with proto code, and a Go gRPC client (“Y”).

The relevant directory structure is as follows:

/x
	/proto
/y

X contains a sub-package, proto, with the proto file and autogenerated Go proto code. Both X and Y’s main packages need to import proto.

We were able to build X without issue. We found that Y failed to build, due to a failure to fetch x/proto from Github. As far as I can tell, this is expected Go Modules behaviour: external packages are fetched from source.

This raised issues around how to handle private repository with Go Modules infrastructure. It also raised issues with how to easily develop and prototype, as even if the repository was public, the newly-written X was not yet pushed to upstream.

We attempted to use the replace directive, to explicitly point to the local copy of proto. This proved challenging, as Go Modules requires a git-based version specified in the dependency. We worked around this issue by committing /x, and manually writing the v0.0.0-time-sha version. This workaround allowed us to import /x, but not /x/proto. Error messaging led us to experiment with defining a separate go.mod in /x/proto, but we were unable to make that approach work.

We resolved he issue by moving proto out of the server, into its own top-level module:

/x
/proto
/y

X and Y were both able to import proto successfully.

What did you expect to see?

We expected to be able to import the local copy of /x/proto, in both /x and /y codebases

What did you see instead?

Overall, this was a frustrating time waste, especially for a new programmer. To summarize the main issues:

• Fetching the upstream version of modules, from the same repository, is unintuitive and often undesirable.
• It was unclear how, if at all, we could import the local copy of a sub-package in the same repository (fortunately in this case, moving proto to a top-level package was perfectly acceptable).
• There doesn’t appear to be a deliberately unstable way to fetch the latest version of a module (i.e. “the current version on disk”).

[vllry]

CC @violetoctagon @Sajmani

[thepudds]

Hi @vllry

First, I would agree there are some rough spots in editing multiple modules simultaneously.

That said, I might not have fully understood the overall scenario you are encountering.

Regarding this:

We attempted to use the replace directive, to explicitly point to the local copy of proto. This proved challenging, as Go Modules requires a git-based version specified in the dependency. We worked around this issue by committing /x, and manually writing the v0.0.0-time-sha versio

and:

There doesn’t appear to be a deliberately unstable way to fetch the latest version of a module (i.e. “the current version on disk”).

One question is -- did you try a directory path based replace directive to point to the current version on disk? If the path on the right side of => in a replace directive is an absolute or relative path, it uses that as the local file path to the replacement module's root directory (which must contain a go.mod file).

There is a little more here in this FAQ on the replace directive:

https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive

Perhaps that helps your scenario, or perhaps not, but if it doesn't help, perhaps you could say a few more words about how and why that does not help, or why it might seem burdensome, or any other related comments?

Regarding this:

[...] Y failed to build, due to a failure to fetch x/proto from Github. As far as I can tell, this is expected Go Modules behaviour: external packages are fetched from source.

and:

[...] Go Modules requires a git-based version specified in the dependency.

Just to clarify, you can work with modules entirely outside of any VCS. Here is another FAQ with a small example of doing that:

https://github.com/golang/go/wiki/Modules#can-i-work-entirely-outside-of-vcs-on-my-local-filesystem

[bcmills]

See also #27542 and #28835.

That said, I think the real solution here is probably #36460, in that you shouldn't need to split the repo into separate client and server modules to begin with. (Today, there is a slight benefit to doing so: splitting out the client module prunes away unnecessary server dependencies for users of the client package. Under lazy loading, those users would not need to load those unnecessary dependencies in the first place.)

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)