`@` and `@workspacename` considered different repositories #3115

jayconrod · 2017-06-02T17:28:12Z

Description of the problem / feature request / question:

Bazel evaluates the same Skylark file multiple times if it is loaded with a local path and an absolute path. This creates problems for rules, since everything gets defined twice. It is particularly problematic for named providers, since the duplicate definitions are neither equivalent nor compatible.

We ran into this issue when trying to migrate rules_go to use named providers. We load our .bzl files with local labels in our own repository, but repositories that depend on ours load using absolute labels, which causes errors.

Bazel should not evaluate the same .bzl file in the same repository more than once.

If possible, provide a minimal example to reproduce the problem:

Create an empty workspace with these files.

# WORKSPACE
workspace(name = "scratch")

# BUILD
load("//:def.bzl", "a")
load("@scratch//:def.bzl", "a")

# def.bzl
print("def.bzl evaluated")
a = 10

Run bazel query //... to evaluate the files. The message is printed twice.

$ bazel query //...
____Loading package: 
WARNING: /usr/local/google/home/jayconrod/Code/scratch/def.bzl:1:1: def.bzl evaluated.
WARNING: /usr/local/google/home/jayconrod/.cache/bazel/_bazel_jayconrod/1df8d8ad4cb35ee83522d81953b4d69f/external/scratch/def.bzl:1:1: def.bzl evaluated.
____Empty results

Environment info

Operating System: Linux
Bazel version (output of bazel info release): 0.5.0

cc @ianthehat

The text was updated successfully, but these errors were encountered:

Change all internal load statements to absolute paths otherwise we end up with duplicate instances of the scripts, with thing like providers of the same name that don't match. This is a workaround for bazelbuild/bazel#3115

jwnimmer-tri · 2017-11-09T00:59:50Z

This is acutely annoying for providers, since it both breaks things, and gives bad diagnostics ("does not have mandatory providers Foo" even though it has "Foo", just a different foo). If it is difficult to fix, it might be worth at least failing-fast in the meantime, even if only for providers.

ittaiz · 2018-05-16T05:11:58Z

@laurentlb this P1 will celebrate anniversary soon.
Is there an update here? I'm not hitting this exact issue but I think I have an adjacent problem (same target gets evaluated twice, once as external and once as internal)

laurentlb · 2018-05-16T09:54:26Z

Klaus, are you aware of this issue?
I don't know exactly at which level we need to fix this (so assigning to both Jon and Klaus).

brandjon · 2018-05-25T12:31:59Z

FYI if it's a local and absolute path within the same repo ("//" vs ":") it doesn't reproduce, so I believe it has to be a repository-specific issue.

jayconrod · 2018-05-25T13:37:29Z

Labels that include the repository name are still treated as different though, i.e., in the example above, @scratch//:def.bzl is treated as different from //:def.bzl, even they they refer to the same file.

This is an issue with rules and named providers. It's easy to get two incompatible copies of something with the same name.

ittaiz · 2018-05-25T13:58:56Z

The question is then what to do with a bzl file in repo A which is needed in both repositories A and B which *additionally* has a dependency on a label inside A (for example a tool). We get invoked twice with different strict deps outputs...

…

On Fri, May 25, 2018 at 4:37 PM Jay Conrod ***@***.***> wrote: Labels that include the repository name are still treated as different though, i.e., in the example above, @scratch//:def.bzl is treated as different from //:def.bzl, even they they refer to the same file. This is an issue with rules and named providers. It's easy to get two incompatible copies of something with the same name. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3115 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF3sH2dscaqMbqtWwjqChNwDBlL8pks5t2AkggaJpZM4NukQk> .

jayconrod · 2018-05-25T14:18:17Z

@ittaiz In rules_go, we work around this by always loading .bzl files using absolute labels that include the repo name. So we always load @io_bazel_rules_go//go:def.bzl even from inside the repository.

ittaiz · 2018-05-25T14:23:06Z

We also did that and it didn't solve the problem. I think since the label that the bzl is dependent on gets loaded twice.

…

On Fri, May 25, 2018 at 5:18 PM Jay Conrod ***@***.***> wrote: @ittaiz <https://github.com/ittaiz> In rules_go, we work around this by always loading .bzl files using absolute labels that include the repo name. So we always load @io_bazel_rules_go//go:def.bzl even from inside the repository. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3115 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFxO9iMz2V9Il37skQ3OBavye6aBMks5t2BK0gaJpZM4NukQk> .

Seems like this is a standard workaround: https://github.com/bazelbuild/bazel/issues/3115\#issuecomment-392072493

* BUILD: use absolute load statements Seems like this is a standard workaround: https://github.com/bazelbuild/bazel/issues/3115\#issuecomment-392072493

aehlig · 2018-06-05T09:34:54Z

Two remarks.

The problem is the @scratch. Loading @//:def.bzl and //:def.bzl evaluates the file only once. So it seems that @ and @scratch are treated as different repositories.
That also fits with the debug message: the direct location of the file, and in bazel's output base under external/scratch---as if @scratch where an external repository (of type local_repository) located at the current working directory.

Added @dkelmer, as this has to be considered as part of the label renaming project as well.

aehlig · 2018-06-05T10:18:50Z

A better way to demonstrate that @scratch is really treated as a different local_repository is shown below. Note that not only another action is carried out when building @scratch//:it after having built @//:it, but it also has a different output file.

/memfs/tmp/work>cat WORKSPACE 
workspace(name="scratch")
/memfs/tmp/work>cat BUILD 
genrule(
  name = "it",
  outs = ["it.txt"],
  cmd = "echo hello > $@",
)
/memfs/tmp/work>bazel clean --expunge
Starting local Bazel server and connecting to it...
........
(12:13:55) INFO: Starting clean.
/memfs/tmp/work>bazel build -s @//...
Starting local Bazel server and connecting to it...
........
(12:14:10) INFO: Current date is 2018-06-05
(12:14:10) INFO: Analysed target //:it (8 packages loaded).
(12:14:10) INFO: Found 1 target...
(12:14:11) SUBCOMMAND: # //:it [action 'Executing genrule //:it']
(cd /crypto/general/aehlig/.cache/bazel/_bazel_aehlig/ea3a7c72d5e8b43a0d4e9c691a291e8f/execroot/scratch && \
  exec env - \
    PATH=/home/aehlig/bin:/usr/local/bin:/home/aehlig/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/aehlig/bin:/sbin:/usr/sbin:/sbin:/usr/sbin \
  /usr/local/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; echo hello > bazel-out/freebsd-fastbuild/genfiles/it.txt')
Target //:it up-to-date:
  /crypto/general/aehlig/.cache/bazel/_bazel_aehlig/ea3a7c72d5e8b43a0d4e9c691a291e8f/execroot/scratch/bazel-out/freebsd-fastbuild/genfiles/it.txt
(12:14:11) INFO: Elapsed time: 0.959s, Critical Path: 0.07s
(12:14:11) INFO: 1 process: 1 processwrapper-sandbox.
(12:14:11) INFO: Build completed successfully, 2 total actions
/memfs/tmp/work>bazel build -s @scratch//...
(12:14:27) INFO: Current date is 2018-06-05
(12:14:27) INFO: Analysed target @scratch//:it (1 packages loaded).
(12:14:27) INFO: Found 1 target...
(12:14:27) SUBCOMMAND: # @scratch//:it [action 'Executing genrule @scratch//:it']
(cd /crypto/general/aehlig/.cache/bazel/_bazel_aehlig/ea3a7c72d5e8b43a0d4e9c691a291e8f/execroot/scratch && \
  exec env - \
    PATH=/home/aehlig/bin:/usr/local/bin:/home/aehlig/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/aehlig/bin:/sbin:/usr/sbin:/sbin:/usr/sbin \
  /usr/local/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; echo hello > bazel-out/freebsd-fastbuild/genfiles/external/scratch/it.txt')
Target @scratch//:it up-to-date:
  /crypto/general/aehlig/.cache/bazel/_bazel_aehlig/ea3a7c72d5e8b43a0d4e9c691a291e8f/execroot/scratch/bazel-out/freebsd-fastbuild/genfiles/external/scratch/it.txt
(12:14:27) INFO: Elapsed time: 0.199s, Critical Path: 0.04s
(12:14:27) INFO: 1 process: 1 processwrapper-sandbox.
(12:14:27) INFO: Build completed successfully, 2 total actions
/memfs/tmp/work>

Adds a `provides` field to haskell_library and haskell_binary to tell users about our custom providers (and possibly throw better error messages). As a related change, the `load` targets for `providers.bzl` are made absolute to work around bazelbuild/bazel#3115

sergiocampama · 2019-05-09T17:56:52Z

I'm still seeing these issues in Bazel 0.25.1. Was the --incompatible_remap_main_repo flag made the default?

dhalperi · 2019-05-09T18:11:00Z

@sergiocampama - yes, experimental_enable_repo_mapping has been the default behavior for a while, and the flag has even been removed in 0.24.0: c32c334

sergiocampama · 2019-05-09T18:15:27Z

what about

bazel/src/main/java/com/google/devtools/build/lib/packages/StarlarkSemanticsOptions.java

Line 504 in 9a005f7

defaultValue = "false",

? It is also a noop? Or was the experimental flag migrated to incompatible and never flipped?

sergiocampama · 2019-05-09T18:16:24Z

I'm seeing the exact provider missing error described in this issue in https://buildkite.com/bazel/rules-apple-darwin/builds/827#5ef4767f-56d7-48b6-a07e-992c9fa26db9

katre · 2019-05-09T18:19:07Z

The flag is now --incompatible_remap_main_repo: https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/packages/StarlarkSemanticsOptions.java;l=503?q=main_repo

It was scheduled to be flipped but there was missing functionality, and it was not made default in 0.26.

See the tracking issue: #7130.

sergiocampama · 2019-05-09T18:54:07Z

Shouldn't this issue be open then until the incompatible flag is flipped?

…kspace names (#613) bazelbuild/bazel#3115 GitOrigin-RevId: 32e685aa3cd32eae15938e9c0f6180ef4a19175c

@bazel_skylib. This works around not having flipped the --incompatible_remap_main_repo flag. If that gets done we can unwind this change. The exact problem we are working around is namespace resolution for providers which are global. Without the main repo remapping, we see the equivalent of '//StarlarkLibraryInfo' from within this library, but Stardoc sees '@bazel_skylib//StarlarkLibraryInfo'. bazelbuild/bazel#3115 bazelbuild/bazel#7130

aehlig · 2019-08-23T12:12:22Z

Interestingly, the issue is not fully solved by (the current implementation of) --incompatible_remap_main_repo (#7130). Consider the following test cases

The first test passes, whereas the second fails in the build with remapping; interestingly enough, that command passes if we build //:y instead of @foo//:y.

...by taking the applicable repository mapping into account, instead of simply assuming it is empty. Related to #7130, #3115. Change-Id: I7f3274b7e238ec0a9ad00643b738fbb12b84a872 PiperOrigin-RevId: 271133642

aehlig · 2019-12-03T11:59:02Z

#7130 is flipped now.

ittaiz · 2020-07-20T03:42:10Z

@aehlig if there are test cases that are failing with the flag on, shouldn't they be fixed?

katre · 2020-07-20T12:34:16Z

@ittaiz Klaus is no longer working at Google or on Bazel. As far as I know there are open issues for a few related problems, if there are any more, can you file them separately?

ittaiz · 2020-07-20T16:54:43Z

Didn't know that. Will do, thanks.

The use of absolute labels in `load()` calls was a workaround for a Bazel bug[0] that was fixed in 2018, before the Bazel v1.0 release. Using either absolute or relative labels as appropriate makes it easier to understand whether a given load is expected to happen in the `@rules_m4` repository or in the root workspace. [0] bazelbuild/bazel#3115 GitOrigin-RevId: 7e5120adf89230c32569c6d9eb281123f1d48b54

jayconrod mentioned this issue Jun 2, 2017

Absoulte paths for load statements bazel-contrib/rules_go#503

Merged

katre assigned laurentlb Jun 7, 2017

katre added the category: extensibility > skylark label Jun 7, 2017

laurentlb added the P1 I'll work on this now. (Assignee required) label Nov 9, 2017

laurentlb assigned laurentlb and damienmg and unassigned laurentlb Nov 9, 2017

damienmg removed their assignment Dec 14, 2017

jwnimmer-tri mentioned this issue Apr 16, 2018

Add drake_cc_package_library and library_lint RobotLocomotion/drake#8582

Merged

jwnimmer-tri mentioned this issue Apr 27, 2018

common/proto/BUILD.bazel install rules are sensitive to whether they use //tools:drake.bzl vs. //tools/skylark:drake_cc.bzl RobotLocomotion/drake#8700

Closed

laurentlb assigned brandjon and aehlig May 16, 2018

brandjon removed their assignment May 25, 2018

dhalperi referenced this issue in batfish/batfish May 30, 2018

BUILD: use absolute load statements

8ef105c

Seems like this is a standard workaround: https://github.com/bazelbuild/bazel/issues/3115\#issuecomment-392072493

dhalperi mentioned this issue May 30, 2018

BUILD: use absolute load statements batfish/batfish#1530

Merged

dhalperi referenced this issue in batfish/batfish May 31, 2018

BUILD: use absolute load statements (#1530)

5b16054

* BUILD: use absolute load statements Seems like this is a standard workaround: https://github.com/bazelbuild/bazel/issues/3115\#issuecomment-392072493

aehlig assigned dkelmer Jun 5, 2018

aehlig changed the title ~~Bazel evaluates the same Skylark file multiple times~~ @ and @workspacename considered different repositories Jun 5, 2018

Profpatsch mentioned this issue Nov 23, 2018

Linkstatic shell tests tweag/rules_haskell#488

Merged

clintharrison mentioned this issue Dec 21, 2018

--output_base in workspace causes infinite symlink when WORKSPACE uses load() #6989

Open

clintharrison mentioned this issue Jan 18, 2019

Visibility not working with @my_repository references #7180

Closed

aiuto added team-Starlark and removed z-category: extensibility > skylark labels Feb 4, 2019

sergiocampama reopened this May 9, 2019

sergiocampama mentioned this issue May 9, 2019

incompatible_remap_main_repo: treat @main_repo//, @//, and // labels as equivalent. #7130

Closed

natansil pushed a commit to wix-incubator/exodus that referenced this issue May 16, 2019

added --experimental_remap_main_repo=true to solve non cannonical wor…

47bf98e

…kspace names (#613) bazelbuild/bazel#3115 GitOrigin-RevId: 32e685aa3cd32eae15938e9c0f6180ef4a19175c

aiuto mentioned this issue Aug 21, 2019

Change all loads of bzl_library to have the absolute repository bazelbuild/bazel-skylib#184

Closed

This was referenced Nov 18, 2019

Bazel CI: Flip --incompatible_remap_main_repo broke many projects in downstream #10246

Closed

rules_rust broken by remapping of the main repository bazelbuild/rules_rust#271

Closed

aehlig closed this as completed Dec 3, 2019

jwnimmer-tri mentioned this issue Sep 20, 2023

Fix install with new bazel 0.5.3 RobotLocomotion/drake#6736

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`@` and `@workspacename` considered different repositories #3115

`@` and `@workspacename` considered different repositories #3115

jayconrod commented Jun 2, 2017

jwnimmer-tri commented Nov 9, 2017

ittaiz commented May 16, 2018

laurentlb commented May 16, 2018

brandjon commented May 25, 2018

jayconrod commented May 25, 2018

ittaiz commented May 25, 2018 via email

jayconrod commented May 25, 2018

ittaiz commented May 25, 2018 via email

aehlig commented Jun 5, 2018

aehlig commented Jun 5, 2018

sergiocampama commented May 9, 2019

dhalperi commented May 9, 2019

sergiocampama commented May 9, 2019

sergiocampama commented May 9, 2019

katre commented May 9, 2019

sergiocampama commented May 9, 2019

aehlig commented Aug 23, 2019 •

edited

Loading

aehlig commented Dec 3, 2019

ittaiz commented Jul 20, 2020

katre commented Jul 20, 2020

ittaiz commented Jul 20, 2020

@ and @workspacename considered different repositories #3115

@ and @workspacename considered different repositories #3115

Comments

jayconrod commented Jun 2, 2017

Description of the problem / feature request / question:

If possible, provide a minimal example to reproduce the problem:

Environment info

jwnimmer-tri commented Nov 9, 2017

ittaiz commented May 16, 2018

laurentlb commented May 16, 2018

brandjon commented May 25, 2018

jayconrod commented May 25, 2018

ittaiz commented May 25, 2018 via email

jayconrod commented May 25, 2018

ittaiz commented May 25, 2018 via email

aehlig commented Jun 5, 2018

aehlig commented Jun 5, 2018

sergiocampama commented May 9, 2019

dhalperi commented May 9, 2019

sergiocampama commented May 9, 2019

sergiocampama commented May 9, 2019

katre commented May 9, 2019

sergiocampama commented May 9, 2019

aehlig commented Aug 23, 2019 • edited Loading

aehlig commented Dec 3, 2019

ittaiz commented Jul 20, 2020

katre commented Jul 20, 2020

ittaiz commented Jul 20, 2020

`@` and `@workspacename` considered different repositories #3115

`@` and `@workspacename` considered different repositories #3115

aehlig commented Aug 23, 2019 •

edited

Loading