Support nested Source roots

[orsenthil]

Currently pants fails with mysterious error when it encounters targets with overlapping source root. It does not fail fast.

But discussions has led to the idea of supporting overlapping source root. The concept of SourceRoot seems applicable to IDEs and build system should be oblivious to it. Support of overlapping sourceroot would help. Even failing fast with correct error message would be helpful to the user before the support of overlapping source root lands.

[ericzundel]

I want to provide a few details. The issue that I face that led to this report is a maven project with a second maven project nested beneath it.

/foo/pom.XML
/foo/BUILD
/foo/src/main/java
/foo/src/main/proto/
/foo/src/test/java

/foo/bar/pom.XML
/foo/bar/BUILD
/foo/bar/src/main/java
/foo/bar/src/main/proto
/foo/bar/src/test/java

The source roots are not really nested, but the project BUILD files where I defined java_library and java_protobuf_library are nested. The logic in sources.py assumes that given a target, you can find a single source root from it (before analyzing the source file list.) I took a quick stab at trying to infer the correct source root from the list of source files but hit a chicken and egg issue in some constructor calling SourceRoot.find()

[benjyw]

I think that patrick's currently in-flight refactor of BUILD file parsing
and the target hierarchy also re-does source roots. For example, it gets
rid of the shady logic we currently have where if it can't find an
appropriate source root when it parses a target it lazily evaluates BUILD
files by climbing up the directory ancestry until it finds a source_root()
that works.

After his refactor the source_roots won't be lazily discovered, they'll
have to be declared up front in BUILD.bootstrap or wherever.

So I expect that he'll be able to support this kind of thing pretty easily,
basically your source root will be whichever one is "closest" to you in the
ancestry. I think this is what you'd expect, and also supports your use
case?

On Thu, Apr 3, 2014 at 6:01 AM, Eric Ayers [email protected] wrote:

I want to provide a few details. The issue that I face that led to this
report is a maven project with a second maven project nested beneath it.

/foo/pom.XML
/foo/BUILD
/foo/src/main/java
/foo/src/main/proto/
/foo/src/test/java

/foo/bar/pom.XML
/foo/bar/BUILD
/foo/bar/src/main/java
/foo/bar/src/main/proto
/foo/bar/src/test/java

The source roots are not really nested, but the project BUILD files where
I defined java_library and java_protobuf_library are nested. The logic in
sources.py assumes that given a target, you can find a single source root
from it (before analyzing the source file list.) I took a quick stab at
trying to infer the correct source root from the list of source files but
hit a chicken and egg issue in some constructor calling SourceRoot.find()

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-39447696
.

[ericzundel]

I'll have to look at the code. The problem as I saw it was not from the lazy evaluation when the BUILD file is underneath two prospective source roots.

source_root('/foo/src/main/proto/src', java_protobuf_library)
source_root('/foo/bar/src/main/proto/src', java_protobuf_library)

If you had a target foo/BUILD that contained a java_protobuf_library(name="proto", ...), how would it choose between foo/src/main/proto/src and foo/bar/src/main/proto/src as the "source root" for the target?

This is currently important for figuring out the appropriate --proto-path arguments to protoc.

[benjyw]

Where are the sources of that java_protobuf_library?

On Thu, Apr 3, 2014 at 11:54 AM, Eric Ayers [email protected]:

I'll have to look at the code. The problem as I saw it was not from the
lazy evaluation when the BUILD file is underneath two prospective source
roots.

source_root('/foo/src/main/proto/src', java_protobuf_library)
source_root('/foo/bar/src/main/proto/src', java_protobuf_library)

If you had a target foo/BUILD that contained a
java_protobuf_library(name="proto", ...), how would it choose between
foo/src/main/proto/src and foo/bar/src/main/proto/src as the "source root"
for the target?

This is currently important for figuring out the appropriate --proto-path
arguments to protoc.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-39490411
.

[ericzundel]

The sources would be where ever you defined them to be in sources[], but in maven, they would be under src/main/proto beneath the pom.xml file:

foo/src/main/proto/com/squareup/foo/foo.proto contains

import "squareup/com/foo/bar/bar.proto"
...

foo/pom.xml # contains maven deps for the foo/ project
foo/BUILD:

java_protobuf_library(
name = 'proto',
sources = [
'./src/main/proto/com/squareup/foo/foo.proto',
],
dependencies = [
'foo/bar:proto',
],
)

/foo/bar/src/main/proto/com/squareup/foo/bar/bar.proto

foo/bar/pom.xml # contains maven deps for the foo/bar project
foo/bar/BUILD:

java_protobuf_library(
name = 'proto',
sources = [
'./src/main/proto/com/squareup/foo/bar/bar.proto',
],
)

[benjyw]

What I mean is, the answer to your question "how would it choose between
foo/src/main/proto/src and foo/bar/src/main/proto/src" should be that it'll
take whichever one is the closest ancestor to all of the sources.

Technically source_root should be a property of the payload, not of the
target.

On Thu, Apr 3, 2014 at 11:54 AM, Eric Ayers [email protected]:

I'll have to look at the code. The problem as I saw it was not from the
lazy evaluation when the BUILD file is underneath two prospective source
roots.

source_root('/foo/src/main/proto/src', java_protobuf_library)
source_root('/foo/bar/src/main/proto/src', java_protobuf_library)

If you had a target foo/BUILD that contained a
java_protobuf_library(name="proto", ...), how would it choose between
foo/src/main/proto/src and foo/bar/src/main/proto/src as the "source root"
for the target?

This is currently important for figuring out the appropriate --proto-path
arguments to protoc.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-39490411
.

[jsirois]

Assigning to @benjyw to close, but I think this can now be closed. Source roots are discovered unambiguously in overlapping cases now IIUC.

[benjyw]

Yes, source roots are discovered unambiguously, and the way they do so is configured via options. Bootstrap BUILD files are gone, as is the problematic logic in SourceRoots.find().

[gmalmquist]

Hooray!