--incompatible_remote_build_event_upload_respect_no_cache + combined cache (remote + disk cache) causes BES-only uploads to disk cache

[brentleyjones]

Description of the problem / feature request:

When using the new --incompatible_remote_build_event_upload_respect_no_cache flag, along with a combined remote+disk cache, blobs that are only for BES upload are uploaded to the disk cache (in addition to the remote cache). This has a couple negative side effects including:

• Using disk space for blobs that will never be referenced locally
• Performing expensive FindMissingBlobs calls on the disk cache
• Causing --bes_upload_mode=fully_async to block for disk IO

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Perform a build with the following flags, two times:

build --bes_upload_mode=fully_async
build --remote_cache=grpcs://remote.buildbuddy.io
build --bes_results_url=https://app.buildbuddy.io/invocation/
build --bes_backend=grpcs://remote.buildbuddy.io
build --disk_cache=/tmp/bazel_disk_cache
build --incompatible_remote_build_event_upload_respect_no_cache
build --noremote_upload_local_results

If the build has lots of things that will be BES-only uploads (one of my examples was 20k+ blobs), the second build will stall at the end as it performs disk cache related operations.

What operating system are you running Bazel on?

macOS 12.0.1

What's the output of bazel info release?

release 5.0.0rc3

Have you found anything relevant by searching the web?

#14338

[brentleyjones]

cc: @coeuvre

Also, since this is for a new flag that is shipping with 5.0, it might be good to get into another RC (if there is one).

[brentleyjones]

The biggest issue for us (out of the negative side effects listed) is the potentially very expensive FindMissingBlobs on the disk cache for all BES referenced blobs.

[coeuvre]

Do we still have the stall at the end of the build if --incompatible_remote_build_event_upload_respect_no_cache is not set?

[brentleyjones]

Yes, it is just longer, as more blobs are considered. I only related the two, as without --incompatible_remote_build_event_upload_respect_no_cache, we would set --experimental_build_event_upload_strategy=local instead, which would get around this issue (as nothing BES related would be uploaded).