Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stdlib not being cached on remote server #1531

Closed
kwiesmueller opened this issue Jun 7, 2018 · 13 comments
Closed

stdlib not being cached on remote server #1531

kwiesmueller opened this issue Jun 7, 2018 · 13 comments

Comments

@kwiesmueller
Copy link

We finally got our project to build successfully using bazel, go and docker on cloudbuild but we are still seing very high build times on all cloud builds (up to 6mins) for all kinds of changes (changing just a comment or the readme).

This seems to be caused by the GoStdlib being built every time.
We are using this command to build:

bazel run \
	--remote_http_cache=https://storage.googleapis.com/cache-xxx \
	--google_credentials=$(GOOGLE_KEYFILE) \
	--spawn_strategy=remote --genrule_strategy=remote \
	--workspace_status_command=./helpers/status.sh //:push

This are the logs:

Finished Step #1
Step #1: INFO: Build completed successfully, 118 total actions
Step #1: INFO: Build completed successfully, 118 total actions
Step #1: INFO: 88 processes: 64 remote cache hit, 24 local.
Step #1: INFO: Elapsed time: 224.905s, Critical Path: 77.52s
Step #1: [91 / 101] GoCompile vendor/go.uber.org/zap/internal/color/linux_amd64_stripped/go_default_library~/bitbucket.org/seibert-media/events/vendor/go.uber.org/zap/internal/color.a; 0s remote
Step #1: [83 / 101] GoStdlib external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/pkg; 34s remote
Step #1: [83 / 101] GoStdlib external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/pkg; 10s remote
Step #1: [48 / 50] [Sched] SHA256 cmd/listeners/reseller/image-layer.tar.sha256
Step #1: [18 / 44] [-----] GoCompile external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile~/go/tools/builders/compile.a [for host]
Step #1: [17 / 44] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib~/pkg; 40s remote
Step #1: [17 / 44] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib~/pkg; 16s remote
Step #1: [17 / 44] GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib~/pkg; 8s remote
Step #1: [16 / 44] [-----] GoLink external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib [for host]
Step #1: [10 / 20] [-----] Writing file external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib~/ROOT
Step #1: [0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
Step #1: INFO: Found 30 targets...
Step #1: INFO: Analysed 30 targets (61 packages loaded).
Step #1: Analyzing: 30 targets (58 packages loaded)
Step #1: Analyzing: 30 targets (58 packages loaded)
Step #1: Analyzing: 30 targets (58 packages loaded)
Step #1: Analyzing: 30 targets (58 packages loaded)
Step #1: Analyzing: 30 targets (34 packages loaded)
Step #1: Analyzing: 30 targets (33 packages loaded)
Step #1: INFO: SHA256 (https://github.com/google/containerregistry/archive/v0.0.25.tar.gz) = 64886684c60bb6f19f73b0e802cdbbf051d10c3803cd11e9ab06d7da8d011ce4
Step #1: Analyzing: 30 targets (29 packages loaded)
Step #1: Analyzing: 30 targets (24 packages loaded)
Step #1: Analyzing: 30 targets (24 packages loaded)
Step #1: Analyzing: 30 targets (23 packages loaded)
Step #1: Loading: 0 packages loaded
Step #1: Loading: 0 packages loaded
Step #1: Loading: 0 packages loaded
Step #1: INFO: SHA256 (https://github.com/bazelbuild/rules_docker/archive/v0.4.0.tar.gz) = 6dede2c65ce86289969b907f343a1382d33c14fbce5e30dd17bb59bb55bb6593
Step #1: Loading: 0 packages loaded
Step #1: INFO: SHA256 (https://github.com/bazelbuild/rules_go/archive/e4d0254fb249a09fb01f052b23d3baddae1b70ec.tar.gz) = e903b171d962467c4e0a425ea3d1bb022e9247886c8272f78b4793473b183777
Step #1: Loading: 0 packages loaded
Step #1: Loading: 0 packages loaded
Step #1: Loading: 
Step #1: .............................
Step #1: Starting local Bazel server and connecting to it...
Step #1: Extracting Bazel installation...
Step #1: Already have image (with digest): gcr.io/cloud-builders/bazel
Starting Step #1
Finished Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/gcloud
Starting Step #0
BUILD
HEAD is now at a93cfbb show default value in help
* branch a93cfbb2bcdf7170496180f82dbde1bd31c5291d -> FETCH_HEAD
From https://source.developers.google.com/p/xxx
Initialized empty Git repository in /workspace/.git/
FETCHSOURCE

starting build "b21188f2-7b7b-41f8-a145-543ba7b124ee"

Any ideas why the files won't get cached?
@jayconrod
Copy link
Contributor

This might be related to the fact that the standard library is a directory output, not regular file outputs. Buildfarm, for example, doesn't work because of this.

I don't have a setup to debug this with cloudbuild right now, but I'm working with that team to get something online. Will let you know as soon as I have more information.

@kwiesmueller
Copy link
Author

kwiesmueller commented Jun 7, 2018 via email •
edited
Loading

@justinsb justinsb mentioned this issue Jun 13, 2018
Closed
@kwiesmueller
Copy link
Author

Is it possible, that this issue also occurs when using the google api packages?
When doing this, those builds never get cached remotly and those buids take forever.

@jayconrod
Copy link
Contributor

Unfortunately, I haven't had time to investigate this yet.

I've been primarily working on unblocking #1548, which is causing conflicts for people mixing pre-generated proto code with code generated at build time. That may affect you as well, since we don't have a standard set of rules yet for googleapis. I expect that to be done by the end of next week. I hope to look into this issue before then though.

Off the top of my head, I don't know what would cause the standard library not to be cached. We generated the standard library as a directory artifact rather than individual files, which is a little bit unusual, but I've been told that's fully supported. We use it in our toolchains; toolchains and remote execution are both relatively new, and it's possible there's some functionality in there that isn't fully integrated yet.

How are you building the google api packages? Are you building from github.com/googleapis/googleapis with custom rules or are you building pre-generated code out of google.golang.org/genproto?

@jayconrod jayconrod mentioned this issue Jun 21, 2018
Closed
@kwiesmueller
Copy link
Author

kwiesmueller commented Jun 21, 2018 •
edited
Loading

Okay, good to hear. Take your time and feel free to involve me if I can help. Happy to learn more about Bazel.

Maybe we are using all of this completely wrong...
So after following the docs I found online, we are generating everything as described in bazel-contrib/bazel-gazelle#227
(yes they are related).
Our code imports those packages:

"cloud.google.com/go/pubsub"
"golang.org/x/oauth2/google"
"golang.org/x/oauth2/jwt"
"google.golang.org/api/option"
"google.golang.org/api/reseller/v1"

All our dependencies are managed using deps and are stored in /vendor and are checked in as well (we made this decission some time ago).

Our WORKSPACE looks like this:

workspace(name = "events")

# Imports basic Go rules for Bazel (e.g. go_binary)
git_repository(
    name = "io_bazel_rules_go",
    remote = "https://github.com/bazelbuild/rules_go.git",
    commit = "e4d0254fb249a09fb01f052b23d3baddae1b70ec",
)

# Imports the Gazelle tool for Go/Bazel
git_repository(
    name = "bazel_gazelle",
    remote = "https://github.com/bazelbuild/bazel-gazelle",
    commit = "644ec7202aa352b78d65bc66abc2c0616d76cc84",
)

# Imports Docker rules for Bazel (e.g. docker_image)
git_repository(
    name = "io_bazel_rules_docker",
    remote = "https://github.com/bazelbuild/rules_docker.git",
    tag = "v0.4.0",
)

# Loads Go rules for Bazel
load("@io_bazel_rules_go//go:def.bzl", "go_rules_dependencies", "go_register_toolchains")

go_rules_dependencies()

go_register_toolchains(
    go_version = "1.10.1",
)

# Loads Docker rules for Bazel
load(
    "@io_bazel_rules_docker//go:image.bzl",
    _go_image_repos = "repositories",
)

_go_image_repos()

# Loads Gazelle tool
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository")

gazelle_dependencies()

I saw in one docs file, that defining external dependencies in WORKSPACE or BUILD is possible, but as we are using external = "vendored", with gazelle i thought this is not required.

I get, that building all those deps is necessary somehow, but doing this with go build is much faster and doesn't take 10min+ at the first time, so I guess we are doing it wrong...

@kubaj
Copy link

kubaj commented Aug 8, 2018

I am using Google Cloud Builder and I've tried to setup remote caching on Google Cloud Storage so my step looks like:

- name: gcr.io/cloud-builders/bazel
  args:
  - 'run'
  - '--remote_http_cache=https://storage.googleapis.com/bzl'
  - '--google_default_credentials'
  - '//:gazelle'

Build with this configuration always times out, and step is stuck on the line:

[17 / 49] GoStdlib external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/pkg [for host]; 356s linux-sandbox
[17 / 49] GoStdlib external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/pkg [for host]; 416s linux-sandbox
[17 / 49] GoStdlib external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/pkg [for host]; 485s linux-sandbox

@prestonvanloon
Copy link
Contributor

prestonvanloon commented Oct 5, 2018 •
edited
Loading

+1 to @kubaj's comment. For some reason, building stdlib when remote cache is enabled takes significantly longer (or it's not built at all for non remote cache builds). Any idea why?

Edit: the build takes ~15 minutes for stdlib in a 2 CPU 3.75Mb ram cloud machine with remote cache, but a negigilble amount of time without remote cache.

@prestonvanloon prestonvanloon mentioned this issue Oct 5, 2018
Merged
@kwiesmueller
Copy link
Author

I get way better performance with the new RBE but still caching for std and for example the Google API Client seems broken.
I can not count how often I had to rebuild all google_protobuf targets without ever changing dependencies and of course the stdlib.

@prestonvanloon prestonvanloon mentioned this issue Nov 12, 2018
Open
@rohansingh
Copy link

For some reason, building stdlib when remote cache is enabled takes significantly longer (or it's not built at all for non remote cache builds).

This is still an issue. Does anyone have insight on where to start debugging?

@rohansingh
Copy link

rohansingh commented Mar 8, 2019 •
edited
Loading

Alright, looks like the underlying issue is bazelbuild/bazel#6091. Lack of parallel uploads means that each file from a target is uploaded sequentially, which takes forever for stdlib. That explains the slow-down.

Another thing I've noticed when running targets with large numbers of files is that occasionally my cache server might fail a single request, resulting in a file missing from the cache. I'm speculating that in this case, the entire target would need to be rebuilt. That could explain why it appears that the stdlib sometimes doesn't seem to get cached at all.

@mostynb mostynb mentioned this issue Jan 24, 2020
Merged
@achew22
Copy link
Member

achew22 commented Nov 17, 2020

With the release of 3.7.0 (which includes bazelbuild/bazel@159c3fe which parallelized this codepath), has anyone experienced better build times on this?

@mariusgrigoriu
Copy link

My gut says no, or at least that it's still slow. I wasn't paying close attention since I didn't see this in the 3.7.0 release notes, but I think I recently experienced this when updating the go version.

@mrmeku mrmeku mentioned this issue Nov 18, 2020
Closed
@jayconrod
Copy link
Contributor

Closing old unlabelled issues.

#2188 is probably the right general issue to follow for GoStdLib being slow with remote execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use use_default_shell_env whenever possible mrmeku/rules_go
7 participants
@achew22 @jayconrod @mariusgrigoriu @rohansingh @prestonvanloon @kubaj @kwiesmueller