pre-fetch all dependencies for build envs without network

[vrothberg]

This is a follow-up discussion from the newly introduced gvisor project: google/gvisor#31

The problem context is that most Linux distributions (in my case openSUSE/SUSE) disable network connections when building a package (e.g., with https://openbuildservice.org/). Fetching external dependencies at build-time, however, is an integral part of Bazel, which, in my humble opinion, may explain why many distributions do not yet package Bazel. For some projects that are build with Bazel, we could find workarounds, but the more dependencies a project has the more complicated is the process to untangle the deps.

The problem I am trying to solve to package gvisor is to pre-fetch all dependencies of gvisor into a specific folder in order to subsequently compress them into tar ball, which can then be used in the build context (no network). In order to do so, I made use of the --repository_cache option (for both, fetch and build), but did not succeed as Bazel was still missing some dependencies. Hence the question: Is this a supported use-case, and if, how can I make it work?

[aehlig]

First of all, I'm well aware that bazel's story for external dependencies is not in good shape. (Historically, bazel was used within google where all dependencies are in the one single repository, so external dependencies have been added as an after-thought.) We're working on it, but it is a long way to go, given all the quick hacks that have been added before a systematic approach was attempt. Also, the tendency of a lot projects to insist on building all dependencies themselves is unfortunate, but probably also motivated by that historical background.

Concerning your question, a simply bazel build of your target for sure will fetch all needed dependencies (for that architecture). Note, however, that a lookup in the --repository_cache will only happen

• for downloads of files (not, e.g., external git repositories), and
• only if a sha256 checksum is specified.

So, most probably a patching of the WORKSPACE file is necessary. (I don't know in detail how go_repositories work and how they can be made fetching from a cache.)

Besides using --repository_cache there are also the following other options that might be interesting for distributions.

• Using --override_repository or patching the WORKSPACE file to use local_repository, you can make bazel look at a local directory. Having a local checkout of dependencies might be nicer for a source-package than some hash-sum indexed cache directory.
• The option --experimental_distdir allows to specify a directory to look for files to download first (by name, but again, only taken if a checksum is specified and matches). In this way, the file names are a bit more meaningful, and it might easier to share common dependencies between projects.

But, in summary: no, there is no supported use case of "just fetch everything and cache it". An unsupported way is to build and then look into the external subdirectory of the directory given by bazel info execution_root. There, at least, you have all the external sources that were used for that build.

Given the even more urgent problems we currently have with external dependencies, I also doubt that a satisfying solution will happen soon. Sorry for not having any better news. I hope you can get along with one of the work-arounds described.

[vrothberg]

@AheliG, thanks a lot for your detailed reply. I will look into it again and provide feedback here.

[aehlig]

https://bazel-review.googlesource.com/c/bazel/+/56490 shows how a set of dependencies can be fetched and put into an archive. d703a5e added a new option --experimental_repository_resolved_file that records all the invocations of (Skylark) repository rules that actually did happen. That, together, provides everything to fetch and pack all the needed files (not convenient, but at least possible); making go rules use that distdir is still a different task. (Note that d703a5e will only be available in bazel 0.15.0, so the release after the one currently in flight; but you can already try it by building bazel at head.)

Also, it doesn't help the underlying problem that projects like to bundle random snapshots of random other projects; but it is not the task of a build system to change upstream's development model.

[vrothberg]

According to google/gvisor#31 (comment) the issue has been resolved. Thank you very much!

[SoftwareApe]

@vrothberg According to google/gvisor#31 (comment) the issue is being worked on, with a reference to this issue here. So I think it's not resolved, at least that doesn't follow from that comment.