Recursive WORKSPACE file loading

[kchodorow]

Design doc: https://bazel-review.googlesource.com/c/6131

[jart]

Drive by review. I love watching the amazing work you do @kchodorow. This seems like a potential solution for #1550.

You're tackling a very difficult problem. Many people have tried to solve this problem and only introduced new ones. One of Bazel's virtues is that it's held off on trying to solve this so far until it can come up with a solution it knows is going to be awesome. I'd like to help you find that solution.

[Diamonds]

The PNG files seems to be broken in https://github.com/bazelbuild/bazel/blob/master/site/designs/_posts/2016-09-19-recursive-ws-parsing.md so it's not clear to me yet how this is going to work.

We always download/link every external dependency before a build can happen.

I want to only pay for the dependencies I use. If I depend on @foo//bar/some/tiny:library, then I don't want to have to download the externals required to build @foo//.... That could include testing dependencies like web browsers, documentation generators, and all this other stuff that could be gigabytes in size, when all I actually wanted was maybe a tiny jar file.

A lot of people have opinions on this topic, as you know. I'm sure you hear user complaints all the time about why Bazel can't solve all their problems without them having to do additional work. Allow me to counteract some of those opinions by saying this. I don't care if Bazel is convenient. I don't care if it's easy. I just care that it's performant, minimal, deterministic, reproducible, and correct (PMDRC). Even if that means I'm configuring my build on the hardest difficulty setting. I like difficult things, as long as I'm reaping significant benefits from that difficulty. This level of high precision professionalism is what sets Bazel apart from many other build systems, which play fast and loose with dependencies. I also believe that once we figure out how to make our builds PMDRC, then we can add abstractions later that make PMDRC easy.

[kchodorow]

Sorry, missed this comment! Thanks for the props, commit in progress to fix the images.

Many people have tried to solve this problem and only introduced new ones.

I've read the internal doc on it :)

I'm not really trying to solve the problem, fwiw. The tl;dr of that design doc is:

1. Use unambiguous transitive external dependencies.
2. Error out fast if there's an ambiguity.
3. Allow overriding dependency conflicts in the top-level WS file.

There's also some cool strictness stuff we'll be doing, like only allowing external dependencies that have been declared in that WS (so @x can't use @y unless it's declared in x's WS file).

We always download/link every external dependency before a build can happen.

We're not doing that!

They'll be downloaded as-needed, as it is today.

[ittaiz]

@kchodorow is this still being targeted for 0.6?

[kchodorow]

It depends when 0.6 is getting cut, I'd like it to be!

[jgavris]

Also eagerly awaiting this, as we have bazel as a submodule of our monorepo and would like to put the WORKSPACE at the root.

[ittaiz]

Will this enable me to have a nested WORKSPACE which has (for example) a java_binary and then in the "main" WORKSPACE a repository rule which depends on the deploy jar of the java_binary?
IIUC currently that isn't possible Note that remote repositories are executed during the analysis phase and thus cannot depends on a target result (the label should point to a non-generated file).

If the answer is yes then any rough estimates on when this will land in HEAD?

[aehlig]

@aehlig Friendly ping. This seems like a critical feature for any project of moderate size. Are there plans to get this into the 1.0 release?

@gnossen No. When @dslomov decided on the priorities for Bazel 1.0, recursive workspaces were not in scope. So, unfortunately, nobody will be working on recursive workspaces before Bazel 1.0 is out.

[gnossen]

Understood. I'd just like to raise awareness that this gap is causing a fair bit of pain. Related: census-instrumentation/opencensus-cpp#378

[immars]

Hi,

is there any update for this feature? I came across this issue and #1550 . Seems it's really hard to make WORKSPACE modularized without this one.

Since 1.2.0 is out now, could you please consider putting this feature somewhere in the roadmap? thanks.

[UebelAndre]

Friendly ping, this is a highly anticipated feature for my group. Can we get an update on it's status?

[smolkaj]

Friendly ping, this is a big problem for open sourcing Google projects.
Is there a known workaround at least?

[EricCousineau-TRI]

@smolkaj In our project, we try to ensure that our repository rules and toolchains can ultimately be grouped into one macro, and is crafted such that you only need one WORKSPACE-level load() statements in a downstream project (e.g. no inter-dependencies for repository rule load() statements).

The example macro:
https://github.com/RobotLocomotion/drake/blob/2c485822aecda3b2400ce5db4262c28a37caeeeb/tools/workspace/default.bzl#L286-L306

Example usage in downstream Bazel project:
https://github.com/RobotLocomotion/drake-external-examples/blob/d5067ad0395cc20c917866ad6ee3d63b0df90fb0/drake_bazel_external/WORKSPACE#L84-L85

Disclaimer: Not the author of the implementation ;)

Meta: I tried searching the doc for something that talks about the load() statement inter-dependency stuff. Didn't find anything easy. As of 3.1.0, looked at:

• https://docs.bazel.build/versions/3.1.0/glossary.html#analysis-phase
• https://docs.bazel.build/versions/3.1.0/skylark/concepts.html#evaluation-model
• https://docs.bazel.build/versions/3.1.0/build-ref.html#workspace
• https://docs.bazel.build/versions/3.1.0/external.html

Multiple loads here, but does not appear to have inter-dependencies?
#1550 (comment)

[meteorcloudy]

We are developing a new external dependency system for Bazel (codename Bzlmod), which will tackle the fundamental problem this issue tries to solve. Please read more at https://bazel.build/docs/bzlmod