x/vgo: proposal: support for companies using monorepos

[akavel]

Background

In our company, we're using a (git) monorepo to host all our code and dependencies. Currently, we're keeping it in a _vendor/ subdirectory, which was created before vendor/ became canonical. We kept it so mainly because of #19090, and because we had our in-house vendoring tool (created long ago) which serves us well enough. Since #19090 was closed, we were open to migrate (our vendoring tool) to the canonical vendor/ path (and maybe even to some more popular tool), and at this point the only thing stopping us was inertia and not enough internal resources.

Being in this position, I'm quite interested in following the vgo story for our company. It seems to have many characteristics which sound attractive in our use case, esp. the conservative approach to version upgrading, and stated interest in build reproducibility.

However, after reading all the relevant articles from rsc published till now, I don't see how we could use vgo in its current shape in our monorepo scenario. Please note we still have code from code.google.com in our repository; I remember that lesson (from even before leftpad became a meme) very lively. I understand that the vgo articles hint at some proxy/caching solution for this concern. I've seen mention of the GOPROXY=file://... feature. I understand from this that we could potentially store .zip files + some metadata in our monorepo, and with the above GOPROXY feature, should be able to use our pinned copies of our dependencies. However, at this point it feels very weird to me to not be able to easily inspect the source code of the libraries, given that they'd have to be stored in zips, IIUC.

Proposal

So at this point, after I've tried to explain my background and situation, I'd like to ask and propose: could you consider going one step further, and supporting GOPROXY-like feature that could be pointed at "unzipped" source code of dependency packages/modules, e.g. in case they're stored in a monorepo? In other words, something more or less like: GOPROXY=file://$GOPATH/vendor (in our particular case, more like GOPROXY=file://$GOPATH/our/package/root/_vendor/src). And by the way, isn't this in fact more or less how third-party dependencies are stored in a monorepo in Google? I'd imagine it would be good for vgo to support a Google-like scenario?

[akavel]

More relevant details are available in my message in a related thread on golang-nuts. In particular, the tool we've developed in-house was actually open-sourced by us a few years ago at: https://github.com/zpas-lab/vendo/; notably, use-cases.md might be interesting. Unfortunately, we didn't have enough resources to even write a proper readme, or advertise it well enough on golang-nuts.

[kardianos]

@rsc has responded to these concerns here, requesting feedback:

https://groups.google.com/forum/m/#!topic/golang-dev/FTMScX1fsYk

[akavel]

@kardianos Yep, and I've already responded, and linked to the thread and my response from my comment above.

[rsc]

@akavel, we've decided to keep support for vendor in the top-level directory of the overall module being built (not in dependencies). If your company monorepo stays a mono-module, then you'll need to mv _vendor vendor, but otherwise it should work. Probably you should save vendor for clean copies of available external dependencies, though, and move the code.google.com stuff to somewhere else in your tree (maybe a top-level external directory, with appropriate import path changes).

Since we are keeping vendor at the top level of the tree, I'll mark this as closed.

[akavel]

I hope and assume, that this should let us have fully reproducible builds, by setting GOPROXY to some nonexistent address, and thus disabling downloading of any new packages, leaving vendor as the only source for packages. That makes it sound usable for us, thanks!

At this point, a few things are still not clear to me, however:

1. @rsc Why the suggestion to move code.google.com stuff somewhere else? I don't grasp reasoning behind this, can you please try to give some more details, so I could better understand? For now, the main advantage and goal of the vendor directory in my eyes was to be able to keep reproducibility by safely storing a pinned copy of dependencies, protected from external circumstances which could try to influence them. It's not clear to me what changes are planned which could interfere with this understanding?
2. In a "mono-module", do I understand correctly, that I can still build multiple executables from its multiple subpaths, as long as they share a common root with a common go.mod?
3. One point I didn't express in the above issue, but I did in the linked thread on golang-nuts, is a hope for a tool which could automatically manage this vendor directory for us, based on the new module-based ecosystem, in a conservative and controlled way. I didn't find any communication in this regard yet; is there a chance for this to be part of official vgo? Or is it maybe expected to be delegated to some proxy tool? Or was it forgotten? Is it enough that I mentioned this on golang-nuts, or should I better open a new issue to make sure this concern won't get lost?

[rsc]

Hi @akavel,

Yes, there will be some way to say "no network allowed".

1. Unless you want to maintain the vendor directory entirely by hand (you really don't), the code.google.com packages are going to confuse every tool that tries to help you. In particular "vgo vendor" is going to assume it can just rm -rf vendor and start over, and when it does that it won't be able to re-fetch the code.google.com packages.

2. Yes.

3. Yes, I think "vgo vendor" will do this. But again that will depend on your vendor dir not diverging from what's actually on the network (and code.google.com isn't, anymore).

[akavel]

@rsc Thanks for the reply! And thanks for not letting this slip. I hope you'll see my follow-up below too.

Re 1. & 3.: Hmm, then this concerns me with regards to our use case. As I see it, this will put a worrying burden on anyone in our org who'd want to add a new dependency to the repo, making this a punishing operation, which I see as bad incentive. I'll try to explain:

First of all, just to repeat for background, the main value I see in vendor directory is that of reliable reproducibility. As you recently communicated, this stays, and that makes me super happy. In other words: if we vendor gopkg.in/foobar today, and gopkg.in closes doors next week, we can still build our app from our repo even 1 year away from now. Reproducibility achieved, we can sleep safe!

However, let's take a look now what this means for Joe the Programmer working at our company in one particular use case. Joe writes a new app in our module, and wants to use influxdb, so he adds a dependency on influxdb_client package. We want to vendor it, so Joe runs "vgo vendor". Assuming it can rm -rf vendor and start over, there are suddenly some unwelcome things that can happen:

• a) gopkg.in (alias code.google.com) closed doors last week, and suddenly Joe is blocked from adding completely unrelated package influxdb_client! As long as he doesn't add the new dependency and run "vgo vendor", everything works and builds OK. But if he does try, he's punished for this with having to take burden of resolving a completely unrelated external problem/downtime. In other words, the cost of coping with Internet being unstable is "amortized" on the first unlucky Joe who needs to do "vgo vendor".
• b) in the meantime, because of Reasons, the author of gopkg.in/foobar may have decided to move the repo's v1 tag to a different commit. Suddenly, although go.mod stays the same, the vendor/gopkg.in/foobar directory shows some diffs in our repo, and Joe has to investigate. Again, this is something completely unrelated to influxdb_client.

Arguably, (b) is somewhat similar to a situation when MVS would kick in and upgrade some package. But there's a subtle difference, in that for MVS this is justifiable as a consequence of adding influxdb_client, so the burden is acceptable as reasonable. In case (b), it's accidental, unexpected and "unjust punishment". But regardless of (b), the (a) is what worries me most.

For now, one workaround I imagine could help bear with (a) and (b) would be to write a proxy and try to implement the logic we want in there (probably something like "copy our vendor to tmp, then serve any packages first from there, or from net if not found"). This looks like it could mitigate the issue to some extent. However, I still see this as an extra cost required of Joe whenever he wants to add a new dependency, and thus needs to run "vgo vendor" — that he now needs to remember to start the proxy before running "vgo vendor". FWIW, I imagine we could at least script this for him, so that instead of "vgo vendor" he runs "vendorize.sh", which does "setup proxy; VGOPROXY=localhost:1234 vgo vendor; tear down proxy". (And most probably actually: "setup proxy; for each GOOS, GOARCH in [...]; do GOOS=... GOARCH=... VGOPROXY=... vgo vendor; done; tear down proxy"; I hope multi-GOOS operation gets reasonably supported.)