…5/Running-Google-Machine-Learning-Library-Tensorflow-On-ARM-64-bit-Platform.html

Fixes bazelbuild#5272

Change-Id: Ia49c084f1fe036ff0e5b18fea096f31642cf8b5b

Closes bazelbuild#5280.

Change-Id: Ia49c084f1fe036ff0e5b18fea096f31642cf8b5b
PiperOrigin-RevId: 200005785

- use an internal event wrapper to unify handling
- rewrite the test to be more explicit about event names and ordering

This a part split out of unknown commit.

PiperOrigin-RevId: 200015904

Instead, add some simple APIs to ExecutorBuilder and inline all the previous
subclasses into their corresponding modules. This removes a bunch of
boilerplate.

PiperOrigin-RevId: 200017162

Convert most `COMPILER_MSVC` to `_WIN32` (as they apply to Windows platform, not MSVC compiler). Only `src/tools/singlejar/zip_headers.h` and `src/main/cpp/util/md5.h` actually need `_MSC_VER`.

`COMPILER_MSVC` in `third_party/protobuf` are not removed. They can be fixed by updating dependency to newer version.

/cc @meteorcloudy

Closes bazelbuild#5350.

Change-Id: Ibc131abfaf34a0cb2bd338549983ea9d28eaabfe
PiperOrigin-RevId: 200019793

This is the first step on the way to a proper action graph query command.

RELNOTES: None
PiperOrigin-RevId: 200026440

RELNOTES:none
PiperOrigin-RevId: 200031466

Rolling forward bazelbuild@6c87715. When I first tried to roll forward this CL I missed one place where PyCcLinkParamsProvider.TO_LINK_PARAMS should have been called.

The target //production/midas/config:client_config_pb builds fine now.

RELNOTES:none
PiperOrigin-RevId: 200032805

- migrate all startTask/completeTask pairs to the new API

PiperOrigin-RevId: 200038703

This also gets rid of some boilerplate. The ExecutionTool.addActionContext
method has been around for a while, but is underused.

There are still a few ActionContextProvider implementations left, which are
implementing other functionality besides adding action contexts.

As a side effect, this change reduces null build time with a hot server on
linux by about a quarter. We were running the linux sandbox twice on every
build, which takes about 70ms each (on my machine), with the total null build
time around 300ms.

PiperOrigin-RevId: 200045145

…h by using a symlink.

This is enabled with --test_env=TEST_SHORT_EXEC_PATH=true.

Passing CI run: https://buildkite.com/bazel/google-bazel-presubmit/builds/3576

RELNOTES: N/A
PiperOrigin-RevId: 200050318

LcovMerger is a tool that merges all the intermediate lcov tracefiles (with .dat extension) found under a coverage directory and prints the merged tracefile to a given output file.

A custom implementation for merging lcov tracefiles is needed because the merging functionality of lcov itself is very slow.

LcovMerger is required to get a single coverage report (lcov tracefile) from a bazel coverage command that executes multiple tests.

ATM LcovMerger is only invoked by tools/test/collect_coverage.sh that collects and merges the tracefiles from a single test invocation. It will also be used from a CoverageReportAction.

Progress on bazelbuild#5246.

PiperOrigin-RevId: 200054506

- Updates the embedded JDK to Azul Zulu 9.0.7
- All integration tests use Bazel with the embedded JDK

Also updated: http://storage.googleapis.com/bazel-mirror/openjdk/index.html

Closes bazelbuild#5312, bazelbuild#5314, bazelbuild#5315

PiperOrigin-RevId: 200055008

PiperOrigin-RevId: 200060887

Switch statements were poorly formatted. Fixing it in a separate commit so
that it doesn't clutter the diff.

RELNOTES: None.
PiperOrigin-RevId: 200062930

- move the save method to an inner class
- don't use a timer, use a blocking queue instead
- add a format enum (in anticipation of adding a json output format)
- update the test to use an in memory buffer, and avoid FoundationTestCase

PiperOrigin-RevId: 200065404

…bstractFileSystemWithCustomStat.

RELNOTES: None
PiperOrigin-RevId: 200071031

Mostly 'debug' level logs that won't appear by default.

PiperOrigin-RevId: 200072597

Also remove the old ConfiguredRuleClassProvider.addSkylarkModule() method, as it, after this change, has no callers.

RELNOTES: None.
PiperOrigin-RevId: 200078816

The on-disk execRoot/blaze-out is now off limits to ActionFS.

RELNOTES: None
PiperOrigin-RevId: 200080287

…vider

RELNOTES: None.
PiperOrigin-RevId: 200096226

…pping is set. Repository rules should throw an attribute not found error if repo_mapping is used but the flag isn't set, otherwise it silently fails.

RELNOTES: None
PiperOrigin-RevId: 200097695

RELNOTES: None.
PiperOrigin-RevId: 200100871

*** Reason for rollback ***

Breaks internal performance tests.

PiperOrigin-RevId: 200103033

…ent.

Pulls out a ThreadPausedState message, containing the previously
separate 'isPaused' and 'locationIfPaused' information, as well as
the stack frames and the reason for pausing, which is useful for
debugging clients (e.g. IDEs might change focus when a breakpoint is
hit, but not when all threads are paused).

TAG_CHANGE_OK=This proto has never yet been used
TYPE_CHANGE_OK=This proto has never yet been used
PiperOrigin-RevId: 200109927

This enables, e.g., TEST rules to be created.

RELNOTES: None.
PiperOrigin-RevId: 200138491

RELNOTES: None.
PiperOrigin-RevId: 200144845

Fixes bazelbuild#5358

Change-Id: I0c9eca59349069a41e3613600df25ed2b678b34c

Closes bazelbuild#5363.

Change-Id: I0c9eca59349069a41e3613600df25ed2b678b34c
PiperOrigin-RevId: 200182247

Fixes bazelbuild#5362

Change-Id: I41d7b78e8d31d4a08ba0aba35c81723cbb315c40

Closes bazelbuild#5364.

Change-Id: I41d7b78e8d31d4a08ba0aba35c81723cbb315c40
PiperOrigin-RevId: 200182384

- the TargetPatternPreloader is still used for query in all its forms
- the remaining TargetPatternEvaluator part is no longer used except in tests
- also make both implementations stateless and pass the offset to the methods
  instead; note that they both modify the underlying skyframe graph, so there
  are side effects to the calls even if there's no direct state anymore

The intent is to migrate the relevant tests to LoadingPhaseRunnerTest (which
could also now be renamed since it's not doing a loading phase), and then
delete the TargetPatternEvaluator interface.

This depends on the previous commit that removed the last direct use of TPE
from an internal command.

PiperOrigin-RevId: 200198067

Even for tests we should not download random files from the internet
without ensuring that we get what we expect.

Change-Id: I4b9ef3f89c1ea12a4b883d833fb598196f9a7db3
PiperOrigin-RevId: 200201636

PiperOrigin-RevId: 200210007

…uested, in order to properly choose the execution platform from the available execution platforms.

Change-Id: I05dc84403e0db765865e9b91c4222894fa867cd9
PiperOrigin-RevId: 200211635

- move the save method to an inner class
- don't use a timer, use a blocking queue instead
- add a format enum (in anticipation of adding a json output format)
- update the test to use an in memory buffer, and avoid FoundationTestCase

Compared to the original bazelbuild@15b8c25, it contains these changes:
- Make it so we don't create a queue if we are not going to write any
  data! The queue is now owned by the writer, and if there is no writer, there
  is no queue.

  This was causing a memory regression because slowest task profiling is
  enabled by default, in which case the profiler is started with no output
  file. In that case, there's no thread that is emptying the queue, but the
  queue was still created by default.

- add additional tests for slowest task and histogram handling; these also
  provide coverage for the case where the profiler is started without an output
  stream
- move all the writer thread handling into the inner class
- make writer access thread-safe
- add a bunch of documentation

PiperOrigin-RevId: 200212978

RELNOTES: None
PiperOrigin-RevId: 200213204

For languages that use launchers, there are some requirements about files other than the main executable being co-located with the executable:

1. For Python, the .zip file must be alongside the executable.
2. For Shell, the shell file must be alongside the exe.

In addition, a .exe suffix is needed for launchers.

Tested that cc_test (non-launcher), java_test, py_test, sh_test (all launchers) work remotely when using the TEST_SHORT_EXEC_PATH env var.

RELNOTES: N/A
PiperOrigin-RevId: 200216308

PiperOrigin-RevId: 200218884

[Documentation fix]

In recent versions of xdot, piping the output of `bazel query` no
longer displays the visualization due to an unrelated change in xdot.
So, use redirection instead of pipe.

Closes bazelbuild#5316.

PiperOrigin-RevId: 200220180

Temporary workaround for bazelbuild#5328.

PiperOrigin-RevId: 200224317

…epository name is remapped.

For example if main/WORKSPACE contains:
local_repository(
    name = "a",
    path = "../a",
    repo_mapping = {"@x" : "@y"},
)

a/BUILD
load("@x//:sample.bzl", "sample")

Then the load in a/BUILD will be resolved as "@y//:sample.bzl"

RELNOTES: None
PiperOrigin-RevId: 200227431

Environment$Continuation#caller is most definitely nullable in practice.

I suspect it's a bug in skylark itself, but for now, just properly mark
it nullable and handle it in the debugger.

Stack trace from NPE:

Caused by: java.lang.NullPointerException
	at com.google.devtools.build.lib.syntax.Environment.listFrames(Environment.java:1197)
	at com.google.devtools.build.lib.syntax.Environment.listFrames(Environment.java:81)
	at com.google.devtools.build.lib.skylarkdebug.server.ThreadHandler.listFrames(ThreadHandler.java:236)
	at com.google.devtools.build.lib.skylarkdebug.server.ThreadHandler.getThreadProto(ThreadHandler.java:345)
	at com.google.devtools.build.lib.skylarkdebug.server.ThreadHandler.pauseCurrentThread(ThreadHandler.java:289)
	at com.google.devtools.build.lib.skylarkdebug.server.ThreadHandler.pauseIfNecessary(ThreadHandler.java:203)
	at com.google.devtools.build.lib.skylarkdebug.server.SkylarkDebugServer.pauseIfNecessary(SkylarkDebugServer.java:158)
	at com.google.devtools.build.lib.skylarkdebug.server.SkylarkDebugServer$DebugAwareEval.exec(SkylarkDebugServer.java:262)
	at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:91)
	at com.google.devtools.build.lib.syntax.BaseFunction.callWithArgArray(BaseFunction.java:462)
	at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:440)
	at com.google.devtools.build.lib.analysis.skylark.SkylarkRuleConfiguredTargetUtil.lambda$buildRule$1(SkylarkRuleConfiguredTargetUtil.java:105)
	at com.google.devtools.build.lib.skylarkdebug.server.SkylarkDebugServer.runWithDebugging(SkylarkDebugServer.java:142)
	at com.google.devtools.build.lib.syntax.DebugServerUtils.runWithDebuggingIfEnabled(DebugServerUtils.java:70)
	at com.google.devtools.build.lib.analysis.skylark.SkylarkRuleConfiguredTargetUtil.buildRule(SkylarkRuleConfiguredTargetUtil.java:100)

PiperOrigin-RevId: 200229036

…packages

RELNOTES: None.
PiperOrigin-RevId: 200246780

PiperOrigin-RevId: 200247872

*** Reason for rollback ***

PY3 does not support Python proto1.

*** Original change description ***

Automated rollback of commit 4c72a82.

*** Reason for rollback ***

Breaks, at least, //ads/aswan/tools:format_mr_results.

*** Original change description ***

Remove python3/ prefix to generated .pyc files.

That makes:
(a) merging PY2 and PY3 .runfiles impossible (which is incorrect anyway) and
(b) generated .py source files incompatible with 2to3 (src_version=PY2) - that's OK as we deprecate 2to3.

RELNOTES: n/a
PiperOrigin-RevId: 200256210

Add a --experimental_generate_json_trace_profile option that puts a file into
the output base (or uses --profile if set).

There are still a lot of problems with this.
- unexplained holes
- way too many threads
- nonsensical event titles
- too many detail events, too little overview
- it may also cause unnecessary load
- it silently overwrites the existing file on subsequent invocations

The format is documented here: goo.gl/oMZPLh

PiperOrigin-RevId: 200259431

When bazel calls wrapped_clang, it single-quotes all arguments. However
it passes flags with arguments quoted as a whole. That is, wrapped_clang
will be called with arguments like these:

  wrapped_clang '-isysroot /a/path/with spaces' '/a/file with spaces.m'

Before this commit, wrapped_clang was blindly splitting on space and
calling clang with invalid arguments. Now it only splits on the _first_
space, and only if the argument starts with '-'.

Closes bazelbuild#5147.

PiperOrigin-RevId: 200259496

…k objects

RELNOTES: None.
PiperOrigin-RevId: 200260766

…lly invalidate the BUILD_INFO_KEY node on --workspace_status_command and related flag changes. Instead, the action has a supplier that allows it to retrieve the correct values at execution time.

This does not sacrifice correctness because the action executes unconditionally on every build, so it will never have stale data.

PiperOrigin-RevId: 200265375

When this flag is turned on, and the user's working directory is beneath the
workspace (but is not the workspace itself), the workspace's absolute path is
printed as a prefix to the convenience symlink.

With this flag off, the displayed convenience symlink path is always relative
to the user's workspace, even if that isn't the current working directory.
(This is the current behavior.)

The new (flag-on) behavior will become default soon, and then this flag will
be removed.

RELNOTES: None.
PiperOrigin-RevId: 200278355

…n fails.

If this exception is thrown, performance no longer matters - we're detonating
the place and riding the explosion out to stderr. So we might as well just
dump everything we know.

RELNOTES: None.
PiperOrigin-RevId: 200290439

…text and thread it through to graph traversal functions. Some other light refactorings as well.

PiperOrigin-RevId: 200292556

…resources to true.

RELNOTES[NEW]: android_local_test now takes advantage of Robolectric's binary resource processing which allows for faster tests.

PiperOrigin-RevId: 200296572

…add unit test coverage for this.

Note that ActionFS is not generic enough to make use of FileSystemTest.

RELNOTES: None
PiperOrigin-RevId: 200304871

On Windows, extracting file symlink in an archive would be performed as copy. To ensure the copy will be successful, we defer all symlink creation after all regular files are extracted.

Fix bazelbuild#5367

RELNOTES: None.
PiperOrigin-RevId: 200345463

RELNOTES: None.
PiperOrigin-RevId: 200351084

…>.gen.def

This helps avoid name conflicts when users want to generate DEF files by themselves. For example, when using a genrule to do that, people naturally name the def file as <target name>.def.

Fixed bazelbuild#5357

RELNOTES: None
PiperOrigin-RevId: 200354303

PiperOrigin-RevId: 200363345

…ForRuntime.

RELNOTES:none
PiperOrigin-RevId: 200366616

PathFragments are not well supported in Skylark, quite the opposite, the Skylark
team tries hard to not use path fragments if possible. All existing users I
looked into had to convert the PathFragment to string using str() anyway.
Because of that this cl should not break anybody.

RELNOTES: None.
PiperOrigin-RevId: 200372447

Instead of hard-coding "-p0", allow the arguments for that patch tool
to be overridden. In particular, this supports the use case of patches
generated with `git format-patch` which are to be read as `-p1`.

Improves on bazelbuild#5379.
Closes bazelbuild#4974 as superseded.

Change-Id: I809fde14beab21d8a755ba4f1706b602bae3c1bb
PiperOrigin-RevId: 200373909

The patch(1) utility usually gives error messages on stdout. So it is
not useful to report only stderr in case a patch failed. Report both.

Fixes bazelbuild#5379.

Change-Id: Ief198849e29ca989dfdefe2fadf495a0b0949972
PiperOrigin-RevId: 200377306

…() syntax.

RELNOTES: None.
PiperOrigin-RevId: 200379060

Currently the selection of C++ toolchain configuration depends on 2 different things:
1. CROSSTOOL file: a proto text file that contains multiple CrosstoolConfig.CToolchain messages, out of which we want to select the appropriate CToolchain;
2. cc_toolchain_suite rule, specified by --crosstool_top option: this rule contains "toolchains" map attribute. The map's keys are of type <cpu>|<compiler>, or just <cpu>, and the values are labels of cc_toolchain rules (which contain additional C++ information).

If there is an entry in cc_toolchain_suite.toolchains that corresponds to the --cpu and --compiler options, we use it, otherwise we loop through all the CToolchains in the CROSSTOOL file, select the one that corresponds to the --cpu and --compiler options, and then get the cc_toolchain label from cc_toolchain_suite.toolchains[toolchain.targetCpu|toolchain.compiler].
In both cases we read the CROSSTOOL file and pass all its information forward, to be used in creation of CppConfiguration and CcToolchainProvider creation.

As part of the efforts of rewriting CROSSTOOL in Skylark, we need to make obtaining the cc_toolchain label independent of the CROSSTOOL file and toolchain selection. As a step towards that goal, we add a new "toolchain_identifier" attribute to cc_toolchain, which uniquely identifies a CToolchain in the CROSSTOOL file.

Now the process of getting the CToolchain goes as follows:
Check for existence of cc_toolchain_suite.toolchains[<cpu>|<compiler>], if --compiler is specified, otherwise check for cc_toolchain_suite.toolchains[<cpu>].

1. if a value is found, load the cc_toolchain rule and look for the toolchain_identifier attribute.
    1.a if the attribute exists, loop through all the CToolchains in CROSSTOOL and select the one with the matching toolchain identifier.
    1.b otherwise fall back to selecting the CToolchain from CROSSTOOL by matching the --cpu and --compiler values.

2. If a value is not found, select the CToolchain from CROSSTOOL by matching the --cpu and --compiler values, and construct the key as follows: <toolchain.cpu>|<toolchain.compiler>.

In the future we will get rid of 2. by making sure that cc_toolchain_suite.toolchains[<cpu>|<compiler>] and cc_toolchain_suite.toolchains[<cpu>] are always defined.
1.b will be removed by making the cc_toolchain.toolchain_identifier attribute mandatory
After this deriving the cc_toolchain label will be independent of the CROSSTOOL file

1.a will ultimately be replaced by an attribute that points to a skylark_crosstool rule, that will contain all the information that CROSSTOOL contains, allowing us to remove CROSSTOOL altogether.

Work towards issue bazelbuild#5380
RELNOTES: None.
PiperOrigin-RevId: 200388550

Do the work in beforeCommand instead.

PiperOrigin-RevId: 200391086

This is in preparation for dismantling BuildView and merging the relevant
parts into AnalysisPhaseRunner.

PiperOrigin-RevId: 200391088

Bazel on Windows is now consistent with Bazel on
Unixes, by setting the mtimes of embedded binaries
to 10 years in the future.

Before this change, on Windows, Bazel used to set
these mtimes to CURRENT_YEAR + 10, January 1st.

This meant that if a user ran Bazel on 2017/12/29,
then on Unix Bazel set the mtimes to 2027/12/29
but on Windows it set them to 2027/01/01.

If the user then ran Bazel in the same workspace
on 2018/01/02, on Unixes it worked fine, but on
Windows it detected that the embedded binaries'
mtime is older than 2018/01/01, and reported a
"corrupt installation" error.

Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

Closes bazelbuild#5385.

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd
PiperOrigin-RevId: 200395493

RELNOTES: None.
PiperOrigin-RevId: 200399094

…rated

files are going to be beneath the target that generates them. In this case,
don't duplicated the package's path.

RELNOTES: None.
PiperOrigin-RevId: 200400464

Onto bazelbuild#5328

PiperOrigin-RevId: 200410170

PiperOrigin-RevId: 200410790

PiperOrigin-RevId: 200410988

…ist when replaying.

We stash the size in the same field as the order using bit packing. The small additional cost of masking out the int should be less than the resizing when replaying large nested sets.

The upper size bound on nested sets will be 512M objects as a result. If Java had unsigned ints we could push it to 1G.

RELNOTES: None
PiperOrigin-RevId: 200417105

While the original intent was to provide better messaging, the truth is things
like ibtool and actool can error because of the state of CoreServices when
targeting anything other than macOS. It appears this attempt to validate
the sdk argument ends up succeeding in some cases, meaning it doesn't
fail quickly, it does a fair mount of work, which we promptly throw away
and still return the original error result.

Rather then trying to expand this into some even more completely retry logic
it seems better to back away from this attempted messaging instead.

Looking at at history the better messaging stems from why it was recommended
that folks also pass the SDK version along with the Xcode version, but we've
gone back to just saying specific the Xcode version, so need for this
improved messaging has also been reduced.

PiperOrigin-RevId: 200422952

…emoval of bazel info crosstool-configuration

RELNOTES: None.
PiperOrigin-RevId: 200437121

…locking on that future.

This allows NestedSet deserialization not to block on storage reads - in-progress deserializations are simply made a member of the NestedSets they produce.

PiperOrigin-RevId: 200440131

RELNOTES: None
PiperOrigin-RevId: 200446926

…box feature.

PiperOrigin-RevId: 200451804

package lookups to get Targets.

PiperOrigin-RevId: 200452642

…viders and output groups.

* Make ctx a keyword argument so that we can more easily add more parameters in the future and eventually remove ctx.

PiperOrigin-RevId: 200453550

A minor serialization improvement. I'm planning to refactor this code
significantly, but wanted to get this in before I forgot.

PiperOrigin-RevId: 200453925

RELNOTES: None
PiperOrigin-RevId: 200455297

…'s stdout/stderr.

RELNOTES: None
PiperOrigin-RevId: 200459354

…ses.

Tag some static members with @AutoCodec.
Replace some lambdas with explicit functions or classes.

PiperOrigin-RevId: 200467500

RELNOTES: None.

PiperOrigin-RevId: 200471197

…s equality.

If the output from composeTransitions is one of the input PatchTransitions, return it.
If it's a composed transition, make sure to wrap it in a class which respects equality.

Without this change, running the same RuleTransitionFactory on the same attribute and
target twice will produce two non-equal Dependencies which can't be deduplicated by
OrderedSetMultimap - no lambda is equal to any other instance of the same lambda.

RELNOTES: None.
PiperOrigin-RevId: 200472777

… are not serialized. Tag TestCompletionValue and any ActionExecutionValue coming from a NotifyOnActionCacheHit (i.e., tests) like that. To make such values really not shared, request the ActionExecutionValue from TestCompletionFunction as opposed to the ArtifactValue (propagating the unshareable bit up seemed like too much fuss, and I have a dream of getting rid of ArtifactValue anyway).

PiperOrigin-RevId: 200504358

… static constant.

This allows us to continue using lambdas in its definition.

This is a partial rollback of bazelbuild@ed1e759, since the work done to class-ify things there is unnecessary once every instance is @AutoCodec-ed.

PiperOrigin-RevId: 200504678

We could conceivably do some monkey-patching at server startup to make all lambdas act Serializable, but one step at a time.

PiperOrigin-RevId: 200509321

PiperOrigin-RevId: 200516863

PiperOrigin-RevId: 200518973

RELNOTES:none
PiperOrigin-RevId: 200525022

…form constraints.

Closes bazelbuild#5341.

Change-Id: Ib74e59fec48102469a5039e045e3f3d0e0d86d8c
PiperOrigin-RevId: 200526448

Move the testing class to the tests tree. This is in preparation for
dismantling BuildView and merging the relevant parts into AnalysisPhaseRunner.

PiperOrigin-RevId: 200532794

Link currently finds no issues. This seems to be because the proper tag is "platform:windows" rather than the documented "multi-platform > windows".

The fix merely updates the text and link to "platform:windows".

Closes bazelbuild#5393.

PiperOrigin-RevId: 200533655