amd64 v3 platform is broken in recent releases

[fishy]

What version of rules_go are you using?

0.50.1

What version of gazelle are you using?

0.39.0

What version of Bazel are you using?

7.3.1

Does this issue reproduce with the latest releases of all the above?

Those are the latest releases

What operating system and processor architecture are you using?

linux/amd64

Any other potentially useful information about your toolchain?

What did you do?

We recently upgraded the bzlmod dependencies. Before:

bazel_dep(name = "rules_go", version = "0.48.1")
bazel_dep(name = "gazelle", version = "0.38.0")
bazel_dep(name = "rules_oci", version = "1.7.6")
# other dependencies I don't think it's important

After

bazel_dep(name = "rules_go", version = "0.50.1")
bazel_dep(name = "gazelle", version = "0.39.0")
bazel_dep(name = "rules_oci", version = "2.0.0")
# other dependencies

We have amd64_v3 platform defined in top level BUILD.bazel as:

platform(
    name = "amd64_v3",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
        "@rules_go//go/constraints/amd64:v3",
    ],
)

And this defined in .bazelrc:

build:docker --@rules_go//go/config:pure --platforms=//:amd64_v3
run:docker --@rules_go//go/config:pure --platforms=//:amd64_v3

And for docker images, we have these rules:

go_binary(
    name = "foo",
    # ...
)

pkg_tar(
    name = "bin",
    srcs = [":foo"],
    mode = "0755",
    package_dir = "/path/to/bin",
)

oci_image(
    name = "docker",
    base = "...",
    tars = [
        ":bin",
        # other tars
    ]
    ...
)

genrule(
    name = "stamped_tag",
    outs = ["stamped_tag.txt"],
    cmd = "grep BUILD_EMBED_LABEL bazel-out/stable-status.txt | cut -d ' ' -f 2 >$@",
    stamp = True,
)

oci_push(
    name = "push",
    image = ":docker",
    repository = "...",
    remote_tags = ":stamped_tag",
)

And we just run this to push the amd64_v3 image:

bazel run --config=docker --embed_label=test //path/to:push

But after the bzlmod updates, it starts to give this error:

system/clock.go:4:2: could not import time (object is [go object linux amd64 go1.23.1 GOAMD64=v3 X:regabiwrappers,regabiargs
] expected [go object linux amd64 go1.23.1 GOAMD64=v1 X:regabiwrappers,regabiargs
])

(the error is a few different source go files with different stdlib imports, this is just one example)

I tried to revert rules_oci back to 1.7.6 but got the same error, so I think this is introduced by new version of rules_go not new version of rules_oci.

What did you expect to see?

What did you see instead?

[fmeum]

Could you share a reproducer that doesn't require rules_oci? That will greatly simplify the effort needed to debug this and can also serve as an integration test for the fix.

[fishy]

@fmeum thanks, here's a minimal example:

• MODULES.bazel:

bazel_dep(name = "rules_go", version = "0.50.1")

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = "1.23.1")

• BUILD.bazel:

load("@rules_go//go:def.bzl", "go_library")

platform(
    name = "amd64_v3",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
        "@rules_go//go/constraints/amd64:v3",
    ],
)

go_library(
    name = "test_lib",
    srcs = [
        "main.go",
    ],
    importpath = "github.com/fishy/test",
    visibility = ["//visibility:private"],
)

• main.go:

package main

import (
        "fmt"
        "os"
        "time"
)

func main() {
        fmt.Println(time.Now())
        os.Exit(0)
}

• go.mod:

module github.com/fishy/test

go 1.23

$ bazel build --platforms=//:amd64_v3 :test_lib
INFO: Invocation ID: f7e93433-accb-4b78-a6fa-44aa11ec8a57
INFO: Analyzed target //:test_lib (0 packages loaded, 0 targets configured).
ERROR: /home/fishy/work/amd64v3/BUILD.bazel:12:11: GoCompilePkg test_lib.a failed: (Exit 1): builder failed: error executing GoCompilePkg command (from target //:test_lib) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_go~~go_sdk~main___download_0/builder_reset/builder compilepkg -sdk external/rules_go~~go_sdk~main___download_0 -goroot ... (remaining 27 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
main.go:4:2: could not import fmt (object is [go object linux amd64 go1.23.1 GOAMD64=v3 X:regabiwrappers,regabiargs
] expected [go object linux amd64 go1.23.1 GOAMD64=v1 X:regabiwrappers,regabiargs
])
main.go:5:2: could not import os (object is [go object linux amd64 go1.23.1 GOAMD64=v3 X:regabiwrappers,regabiargs
] expected [go object linux amd64 go1.23.1 GOAMD64=v1 X:regabiwrappers,regabiargs
])
main.go:6:2: could not import time (object is [go object linux amd64 go1.23.1 GOAMD64=v3 X:regabiwrappers,regabiargs
] expected [go object linux amd64 go1.23.1 GOAMD64=v1 X:regabiwrappers,regabiargs
])
compilepkg: error running subcommand external/rules_go~~go_sdk~main___download_0/pkg/tool/linux_amd64/compile: exit status 2
Target //:test_lib failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.140s, Critical Path: 0.02s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully

$ bazel build :test_lib
INFO: Invocation ID: e2a96395-4463-4109-a308-39dcbf08cb9e
WARNING: Build option --platforms has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //:test_lib (0 packages loaded, 12205 targets configured).
INFO: Found 1 target...
Target //:test_lib up-to-date:
  bazel-bin/test_lib.x
INFO: Elapsed time: 21.004s, Critical Path: 20.30s
INFO: 3 processes: 1 internal, 2 linux-sandbox.
INFO: Build completed successfully, 3 total actions

[fishy]

looks like 0.48.1 is the last version this works. 0.49.0 gives the same error I'm seeing now with 0.50.1.

[ghost]

The GOAMD64 env variable is dropped in the compilepkgs action, the modification below should fix this issue:

diff --git a/go/private/context.bzl b/go/private/context.bzl
index d2e0c653..db28e5c2 100644
--- a/go/private/context.bzl
+++ b/go/private/context.bzl
@@ -508,12 +508,6 @@ def go_context(
         "GODEBUG": "winsymlink=0",
     }
 
-    # Path mapping can't map the values of environment variables, so we pass GOROOT to the action
-    # via an argument instead in builder_args. We need to drop it from the environment to get cache
-    # hits across different configurations since the stdlib path typically contains a Bazel
-    # configuration segment.
-    env_for_path_mapping = {k: v for k, v in env.items() if k != "GOROOT"}
-
     # The level of support is determined by the platform constraints in
     # //go/constraints/amd64.
     # See https://go.dev/wiki/MinimumRequirements#amd64
@@ -525,6 +519,12 @@ def go_context(
     if mode.arm:
         env["GOARM"] = mode.arm
 
+    # Path mapping can't map the values of environment variables, so we pass GOROOT to the action
+    # via an argument instead in builder_args. We need to drop it from the environment to get cache
+    # hits across different configurations since the stdlib path typically contains a Bazel
+    # configuration segment.
+    env_for_path_mapping = {k: v for k, v in env.items() if k != "GOROOT"}
+
     if cgo_context_info:
         env.update(cgo_context_info.env)
         cc_toolchain_files = cgo_context_info.cc_toolchain_files

I'm going to push a PR with this modification, maybe with an update of the _TAG_AFFECTS_STDLIB list to support all new tags for recent versions of Go + all goexperiments.* and amd.* tags. I don't understand why these tags have been explicitly excluded from this list 🤔.