GoStdLib is slow with remote caching / execution

[mariusgrigoriu]

What version of rules_go are you using?

Upgrading from 0.19.1 to 0.19.2+

What version of gazelle are you using?

0.18.1

What version of Bazel are you using?

0.28.1

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux EC2 instance c5.large

Any other potentially useful information about your toolchain?

rules_docker 0.7.0
Remote caching to Google storage bucket turned on

What did you do?

Using go_image to build the following:

go_image(
    name = "image",
    embed = ["@com_github_traum_ferienwohnungen_conntrack_stats_exporter//:go_default_library"],
    base = ":base",
    goarch = "amd64",
    goos = "linux",
    pure = "on",
)

results in a build time approaching 20 minutes
[55 / 78] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg; 1166s remote-cache, linux-sandbox
but only when using rules_go to higher than 0.19.1. Using 0.19.1, stdlib takes only about 10s.

What changed between releases that could cause this?

[jayconrod]

Regrettably it's a known issue that the GoStdLib action is slow for remote execution. That action involves sending and receiving thousands of small files.

The only thing that changed between those versions is the default Go version, so perhaps that simply invalidated the cache? Nothing about the implementation changed.

[pcj]

Would pre-packaging the inputs for GoStdlib as a tarball help? It would be unnecessary for local builds but likely not too noticeable.

[pcj]

Just checked: 6148 files

[bjacklyn]

To avoid uploading/downloading this to/from the remote cache you can try:
echo "build --modify_execution_info='GoStdlib.*=+no-remote-cache'" >> .bazelrc

[derekperkins]

I tried using the --modify_execution_info='GoStdlib.*=+no-remote-cache' flag on Google Cloud Build and didn't see any improvements.

I've also tried go_register_toolchains(version = "host") without any measurable improvements, which made me wonder if the stdlib was the problem or not.

bazel: 3.7.0
rules_go: 0.25.0

[prestonvanloon]

@derekperkins I've also noticed that remote caching with rules_go in a containerized environment usually increases build times by x2 to x5. Even without caching of gostdlib. It seems remote caching is far from being production ready.