Regression in v0.34.0 breaks go_image on M1 macs

[stabai]

What version of rules_go are you using?

v0.34.0

What version of gazelle are you using?

v0.25.0

What version of Bazel are you using?

Bazelisk version: v1.11.0
Build label: 5.2.0
Build target: bazel-out/darwin_arm64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 7 16:11:09 2022 (1654618269)
Build timestamp: 1654618269
Build timestamp as int: 1654618269

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

OS: macOS Monterey v12.5
Architecture: Apple M1 (arm64) Max
Model: MacBook Pro (16-inch, 2021)

Any other potentially useful information about your toolchain?

N/A

What did you do?

After upgrading to rules_go v0.34.0, I tried to build a docker image for my go application by running:

bazel run //my/foo/bar:server_image -- --norun

What did you expect to see?

A successful build of the docker image, followed by loading and tagging the image.

What did you see instead?

I get the error below when building the docker image with rules_go v0.34.0.

My go_binary targets still work, but my go_image targets (using rules_docker) give me an error.

Note: Switching back to v0.33.0 (or any of the previous 5 releases) makes the error go away. The issue only presents when I use rules_go v0.34.0. I switched nothing else in the entire workspace between testing different versions.

ERROR: /private/var/tmp/_bazel_stabai/9072a78e88c27b50b7aa1da1fa2eec2e/external/io_bazel_rules_go/BUILD.bazel:86:17: While resolving toolchains for target @io_bazel_rules_go//:cgo_context_data: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
ERROR: Analysis of target '//my/foo/bar:server_image' failed; build aborted:
INFO: Elapsed time: 34.179s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (157 packages loaded, 9786 targets configured)
FAILED: Build did NOT complete successfully (157 packages loaded, 9786 targets configured)
    Fetching @org_golang_google_protobuf; fetching
    Fetching @com_github_desertbit_timer; fetching
    Fetching @org_golang_x_net; fetching
    Fetching @com_github_cenkalti_backoff_v4; fetching
    Fetching @io_nhooyr_websocket; fetching
    Fetching @com_github_golang_protobuf; fetching
    Fetching @go_googleapis; Restarting.

[fmeum]

Do you have a reproducer I can run on my Mac? The output with --toolchain_resolution_debug=@bazel_tools//tools/cpp:toolchain_type would be helpful. Also, which architecture/OS has your image?

The correct future solution for these kind of issues seems to be to make the C++ toolchain requirement optional. This is only possible on Bazel HEAD though.

[stabai]

I pulled out an old template project to use for a reproducer, and it's behaving a little differently. 😕 That project works with any version of rules_go.

However, in both repos, it will only ever work if my go_image has the following parameters set:

goos = "linux",
goarch = "amd64",

If those are omitted, I get the same error I mentioned above (in either repo, regardless of rules_go version):

private main project:

goos/goarch set	rules_go version	result
true	v0.34.0	error
true	v0.33.0	success
false	v0.34.0	error
false	v0.33.0	error

public reproducer project (instructions below):

goos/goarch set	rules_go version	result
true	v0.34.0	success
true	v0.33.0	success
false	v0.34.0	error
false	v0.33.0	error

I can't share the main project I'm working on, and I'm not yet sure what the difference between the projects is that causes my main project to fail on v0.34.0 even when these parameters are set.

---

Reproducing

The template project repo is https://github.com/stabai/readysetgo/tree/repro/rules_go/3269 and I will leave its repro/rules_go/3269 branch intact while this issue is open.

To reproduce the issue, clone the repo, switch to the repro/rules_go/3269 branch, and then execute bazel run //:tidy to initialize the workspace. Then, you can build the following targets to see a working one (with goos and goarch set) and a non-working one (with those parameters omitted).

bazel build src/example/service:example-go-grpc-image         # succeeds
bazel build src/example/service:example-go-grpc-image-broken  # fails with error

Since I can't share the main project I'm working on, I don't currently have a way for you to reproduce the issue that fails on v0.34.0 and succeeds on v0.33.0. However, tell me if that's an important part of debugging, and I can figure out what the difference is to make a sharable version of that. The error message and toolchain resolution debug output for that project when using v0.34.0 is exactly the same as what happens in this demo repo's broken target.

---

Toolchain resolution debug output

INFO: Build option --toolchain_resolution_debug has changed, discarding analysis cache.
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: execution @local_config_platform//:host: Selected toolchain @local_config_cc//:cc-compiler-darwin_arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-darwin_x86_64; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-darwin_x86_64; mismatching values: x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64e; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64e; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_armv7; mismatching values: ios, armv7
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_armv7; mismatching values: ios, armv7
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_i386; mismatching values: ios, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_i386; mismatching values: ios, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_sim_arm64; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_sim_arm64; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_x86_64; mismatching values: ios, x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-ios_x86_64; mismatching values: ios, x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-tvos_arm64; mismatching values: tvos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-tvos_arm64; mismatching values: tvos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-tvos_sim_arm64; mismatching values: tvos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-tvos_sim_arm64; mismatching values: tvos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-tvos_x86_64; mismatching values: tvos, x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-tvos_x86_64; mismatching values: tvos, x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64; mismatching values: watchos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64; mismatching values: watchos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64_32; mismatching values: watchos, arm64_32
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64_32; mismatching values: watchos, arm64_32
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_armv7k; mismatching values: watchos, armv7k
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_armv7k; mismatching values: watchos, armv7k
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_i386; mismatching values: watchos, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_i386; mismatching values: watchos, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_x86_64; mismatching values: watchos, x86_64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-watchos_x86_64; mismatching values: watchos, x86_64
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-darwin_arm64, type @bazel_tools//tools/python:toolchain_type -> toolchain @bazel_tools//tools/python:_autodetecting_py_runtime_pair
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-darwin_arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-darwin_arm64; mismatching values: osx, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-darwin_arm64; mismatching values: osx, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-darwin_arm64e; mismatching values: osx, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-darwin_arm64e; mismatching values: osx, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-darwin_x86_64; mismatching values: osx
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-darwin_x86_64; mismatching values: osx
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64; mismatching values: ios, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64; mismatching values: ios, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64e; mismatching values: ios, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_arm64e; mismatching values: ios, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_armv7; mismatching values: ios, armv7
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_armv7; mismatching values: ios, armv7
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_i386; mismatching values: ios, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_i386; mismatching values: ios, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_sim_arm64; mismatching values: ios, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_sim_arm64; mismatching values: ios, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_x86_64; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-ios_x86_64; mismatching values: ios
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-tvos_arm64; mismatching values: tvos, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-tvos_arm64; mismatching values: tvos, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-tvos_sim_arm64; mismatching values: tvos, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-tvos_sim_arm64; mismatching values: tvos, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-tvos_x86_64; mismatching values: tvos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-tvos_x86_64; mismatching values: tvos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64; mismatching values: watchos, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64; mismatching values: watchos, arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64_32; mismatching values: watchos, arm64_32
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_arm64_32; mismatching values: watchos, arm64_32
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_armv7k; mismatching values: watchos, armv7k
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_armv7k; mismatching values: watchos, armv7k
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_i386; mismatching values: watchos, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_i386; mismatching values: watchos, i386
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_x86_64; mismatching values: watchos
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: Rejected toolchain @local_config_cc//:cc-compiler-watchos_x86_64; mismatching values: watchos
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @io_bazel_rules_docker//platforms:image_transition: No toolchains found.

[thefallentree]

this is not really rules_go's issue, it is rules_docker 's issue

you can work around this by upgrading to rules_docker latest then

bazel build --@io_bazel_rules_docker//transitions:enable=false ... shuld work

and you can add this to .bazelrc file

[michaeljs1990]

Same issue even with bazel build --@io_bazel_rules_docker//transitions:enable=false set. Changing back to v33 fixes the issue and I can see causes the toolchain to find a match.