Big Kahuna: Do cache fills synchronously

[arschles]

We've had some discussion about whether to do cache fills synchronously or asynchronously. We've discussed how we can make a vgo get call still succeed if we do cache fills asynchronously (and return a 404 to the user), and created golang/go#26334 to discuss a possibility.

I've mostly argued for asynchronously filling caches until now, but I'm changing my mind because I think doing so makes the CLI (vgo, soon to be Go modules in 1.11), the architecture, and the conceptual model of both the registry and the proxy easier to grasp.

Here's exactly what I'm suggesting on the servers:

• For a cache miss on a GET /list request, we return only the tags from the upstream. No fetching go.mod or other assets
  • In the registry, "the upstream" means the VCS
  • In the proxy, "the upstream" means the VCS or the registry, or just 404-ing to the client, depending on exclude rules
• For a cache miss on a /vX.Y.Z.info, vX.Y.Z.info or vX.Y.Z.zip request, we fetch and store all three assets and return the appropriate one

With this machinery in place, the CLI won't need to consider whether to "back off" to the VCS. If it's not in the proxy/registry, it doesn't exist, and that invariant makes golang/go#26334 no longer necessary.

[marwan-at-work]

@arschles if you haven't started working on it already, I've already written some of the code for how you described /latest. I can push that as a separate PR.

[arschles]

@marwan-at-work I'd love that!

I'm gonna work in increments starting with moving the fetcher logic in #291 and then to putting synchronous fetches into each endpoint, so I'll skip the /list endpoint if you've pushed it up already

[arschles]

We're breaking this up into a few pieces:

• Cleanup: refactor the fetcher logic (Moving the fetch logic into 'pkg/module' #291)
  • Follow-up to that: Removing ./pkg/repo #316
• Cleanup: remove the proxy fetch handler (Removing the fetch handler & gomod parser #314)
  • Requires Moving the fetch logic into 'pkg/module' #291 and Removing ./pkg/repo #316
• Install synchronous fetcher logic in the various endpoints as necessary
• Implement the /list endpoint (Download: get /list from upstream #300)

Edit #291 is in, but #291, #314 and #316 are closed out, in favor of doing all the cleanup work in #335

[arschles]

Ok right now I have a ton of PRs up that attack this problem, and I've opened some against other branches. Here's how it all looks:

• The root of everything is Moving the fetch logic into 'pkg/module' #291. That's the thing that needs reviewing first 😄
• Doing synchronous cache filling for the .info endpoint #319, Removing ./pkg/repo #316 and Removing the fetch handler & gomod parser #314 are opened against Moving the fetch logic into 'pkg/module' #291 and they need Moving the fetch logic into 'pkg/module' #291 merged before I can continue with them

Edit: see #290 (comment)