x/build: "unexpected stale targets" on darwin builders #33598

bcmills · 2019-08-12T16:38:56Z

Two flakes in the darwin-amd64-10_11 builders over the past week or so, both with the same symptom: unexpected stale targets failures for internal/cpu, and a build ID mismatch for runtime/internal/sys.

CC @jayconrod @ianlancetaylor @aclements

darwin-amd64-10_11 at 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3

:: Running /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash with args ["/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/make.bash"] and env ["TERM_PROGRAM=Apple_Terminal" "TERM=xterm-256color" "SHELL=/bin/bash" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/" "Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.55fT0Y9EVR/Render" "TERM_PROGRAM_VERSION=361.1" "TERM_SESSION_ID=D33F650A-CE77-4BF0-B95E-A37406DCC122" "USER=gopher" "SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.43nQd98RPC/Listeners" "__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0" "PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" "PWD=/Users/gopher" "LANG=en_US.UTF-8" "XPC_FLAGS=0x0" "XPC_SERVICE_NAME=0" "SHLVL=2" "HOME=/Users/gopher" "LOGNAME=gopher" "SECURITYSESSIONID=186a8" "GO_BUILDER_ENV=macstadium_vm" "GO_STAGE0_NET_DELAY=300ms" "GO_STAGE0_DL_DELAY=0s" "WORKDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir" "GOROOT_BOOTSTRAP=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go1.4" "GO_BUILDER_NAME=darwin-amd64-10_11" "GOROOT_BOOTSTRAP=/Users/gopher/go1.4" "GOBIN=" "TMPDIR=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/tmp" "GOCACHE=/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/gocache"] in dir /var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src

Building Go cmd/dist using /Users/gopher/go1.4.
Building Go toolchain1 using /Users/gopher/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel 0ca4f6be35d2abf54128b57c3aa1a1fcbd5adfe3"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos darwin goarch amd64\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "modinfo \"\"\n"
HASH[build runtime/internal/sys]: "compile PRWJHAzAczrrtWSoyeY4 [] []\n"
HASH[build runtime/internal/sys]: "=\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/arch_amd64.go: d301f8642ec7fcc35d0d39a4b07e0187971afea7c6d0dfd5ddcef1d031670160
HASH[build runtime/internal/sys]: "file arch_amd64.go 0wH4ZC7H_MNdDTmksH4B\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/stubs.go: 23b3e5c631b086fe7a2dec4bf044600e034bf6a8eeb25e0a19efc4ce6311423d
HASH[build runtime/internal/sys]: "file stubs.go I7PlxjGwhv56LexL8ERg\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoarch_amd64.go: 6f2fa4ebe8999a3b7cfe3ead6e24b8356ed842292f23cb1b4f995c0b5b45126b
HASH[build runtime/internal/sys]: "file zgoarch_amd64.go by-k6-iZmjt8_j6tbiS4\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zgoos_darwin.go: 464151a4ab63f8c7aab345f457789cb12c9b68051fc73b8a675717d7886214dc
HASH[build runtime/internal/sys]: "file zgoos_darwin.go RkFRpKtj-Meqs0X0V3ic\n"
HASH /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/src/runtime/internal/sys/zversion.go: 623491756e2adacebf4c53078883e4284349c6159d0404aba9975887d7660d06
HASH[build runtime/internal/sys]: "file zversion.go YjSRdW4q2s6_TFMHiIPk\n"
HASH[build runtime/internal/sys]: cd175367cc0e62c53ed8e20197f8e44b2270f61d3eb7b4f3c1991a5abb06ae16
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir/go/pkg/tool/darwin_amd64/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch

The text was updated successfully, but these errors were encountered:

bcmills · 2019-08-27T12:45:11Z

darwin-amd64-nocgo: https://build.golang.org/log/51522ce78248f3033ac4106c9c8aaed15b169b38

bcmills · 2019-09-11T14:44:34Z

darwin-amd64-10_14: https://build.golang.org/log/b56f0ee52abc686e36bc921be252232bbf7afbd9

bcmills · 2019-10-22T13:20:24Z

Not clear to me whether this is during toolchain3, or if the underlying cause is even related, but there is a similar symptom on the linux-ppc64le-power9osu builder in https://build.golang.org/log/57277f725843345df871d34cfabf1fda58934be0.

laboger · 2019-10-22T13:34:21Z

I see the same type of failure with the stale targets when I try to force -pie to be the default build mode for ppc64le (due to a request). It is noted in #28531. Maybe something wrong with the way that internal/cpu is hashed so that it doesn't get consistent results?

bcmills · 2019-11-21T14:42:17Z

bcmills · 2019-11-21T16:23:07Z

@laboger, the ppc64le failures seem to occur at a different point in the build process. I'll file a separate issue for that.

bcmills · 2020-02-20T13:53:32Z

2020-02-19T21:26:10-7299308/darwin-amd64-10_12

bcmills · 2020-03-18T01:53:46Z

Curiously, the only builder that has failed in this way in the last month or so is the plan9-arm builder.

(There hasn't been another darwin-amd64 failure since February, and we have a bit of circumstantial evidence from linker crashes that that rate of failure may be due to platform filesystem instability.)

2020-03-02T17:40:11-a2bff7c/plan9-arm
2020-02-19T21:26:10-7299308/darwin-amd64-10_12
2020-02-18T19:44:25-a0cf2c8/plan9-arm

bcmills · 2020-03-18T02:08:58Z

Curiously, CL 223755 seems to demonstrate that the darwin-amd64-10_12 builder is receiving a stale snapshot (or not using snapshots and not rebuilding appropriately).

Building Go cmd/dist using /Users/gopher/go1.4. (go1.4-bootstrap-20161024 darwin/amd64)
Building Go toolchain1 using /Users/gopher/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
---
Installed Go for darwin/amd64 in /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go
Installed commands in /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_12/go/bin
…
go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std cmd]:
	STALE cmd/addr2line: build ID mismatch
	STALE cmd/api: build ID mismatch
	STALE cmd/asm: build ID mismatch
	STALE cmd/buildid: build ID mismatch
	STALE cmd/cgo: build ID mismatch
	STALE cmd/compile: build ID mismatch
	STALE cmd/cover: build ID mismatch
	STALE cmd/dist: build ID mismatch
	STALE cmd/doc: build ID mismatch
	STALE cmd/fix: build ID mismatch
	STALE cmd/go: build ID mismatch
	STALE cmd/gofmt: build ID mismatch
	STALE cmd/link: build ID mismatch
	STALE cmd/nm: build ID mismatch
	STALE cmd/objdump: build ID mismatch
	STALE cmd/pack: build ID mismatch
	STALE cmd/pprof: build ID mismatch
	STALE cmd/test2json: build ID mismatch
	STALE cmd/trace: build ID mismatch
	STALE cmd/vet: build ID mismatch

CC @cagedmantis @dmitshur @toothrot

gopherbot · 2020-03-18T18:21:52Z

Change https://golang.org/cl/223925 mentions this issue: cmd/go: rebuild cmd/cgo in test_race_install_cgo instead of asserting that it is not stale

bcmills · 2020-03-18T18:24:56Z

I haven't seen the solaris failure mode in months, so retitling to focus on the remaining darwin failures.

bcmills · 2020-03-18T18:26:16Z

The affected builders seem to be:

darwin-amd64-10_12
darwin-amd64-10_14
darwin-amd64-10_15

Curiously, darwin-386-10_14 and darwin-amd64-nocgo appear to be unaffected. To me, that suggests a problem with the builder configuration rather than a bug in cmd/dist.

bcmills · 2020-03-18T18:26:51Z

(Those builders broke due to unexpected staleness in CL 223746.)

toothrot · 2020-03-18T18:29:14Z

This could be related to #35459 /cc @cagedmantis

Some of the darwin-amd64 builders are providing a stale environment. Let's un-break them while we investigate. Updates #37573 Updates #33598 Change-Id: I8b79778fe4d5aa916557c1ba89fa9c776d130b01 Reviewed-on: https://go-review.googlesource.com/c/go/+/223925 Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Andrew Bonventre <[email protected]> Reviewed-by: Jay Conrod <[email protected]>

gopherbot · 2021-10-01T16:15:49Z

Change https://golang.org/cl/353352 mentions this issue: cmd/go: insert goroot to the hash of build cache when the packages include C files

dmitshur · 2021-10-06T17:47:16Z

I mailed out a CL to fix this, but I think it will require some deployment work to determine whether it fixes the problem.

I think the change in CL 353549 most relevant to the issue in #33598 (comment) is in x/build/cmd/buildlet that runs on builders. Builders are generally configured with a "stage0" that's responsible for updating the buildlet binary whenever a new one becomes uploaded, so we should see results after we do so.

Also consistently deduplicate Env entries, mainly to reduce confusion when reading logs. This change is probably larger than what is strictly needed to fix the issue, but it seemed simpler than trying to figure out which of the many calls to exec.Command in the module are actually relevant. (It also provides some useful case studies for possible additions to the Go standard library.) For golang/go#33598 Change-Id: Ia2bce4549e4a71b56fb497e3df093f79fbcf7f29 Reviewed-on: https://go-review.googlesource.com/c/build/+/353549 Trust: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

dmitshur · 2021-10-08T17:49:13Z

The buildlet binary for darwin-amd64 has been updated (the rest to follow).
@bcmills Can you check whether it's working as expected in CL 223755?

gopherbot · 2021-10-08T17:55:41Z

Change https://golang.org/cl/354752 mentions this issue: dashboard: update host-ios-arm64-corellium-ios to match its previous env

In CL 353549 I refactored environment-handling in the buildlet such that an explicit per-host GOROOT_BOOTSTRAP environment configured in dashboard/builders.go overrides the latent GOROOT_BOOTSTRAP value on the host. (The entry in dashboard/builders.go may omit GOROOT_BOOTSTRAP entirely to defer to the host environment.) That change caught out a previously-unnoticed configuration mismatch for the host-ios-arm64-corellium-ios builder: its entry in dashboard/builders.go indicated "GOROOT_BOOTSTRAP=/var/mobile/go-ios-arm64-bootstrap", but (as shown in previous builder logs¹) it was actually using "GOROOT_BOOTSTRAP=/var/root/go-ios-arm64-bootstrap". The latter empirically exists on the builder, while the former does not. ¹https://build.golang.org/log/4da918256e30f3442be7885019dc2d17bc718e0d For golang/go#33598. Change-Id: I09e5f83b46f23d5a4744c752fb66fcd4b33e4cff Reviewed-on: https://go-review.googlesource.com/c/build/+/354752 Trust: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

bcmills · 2021-10-08T18:23:20Z

Delightful! CL 353549 does seem to have done the trick. We could submit CL 223755 to prevent future regressions, but if I understand correctly it might seriously degrade the performance of certain builders (notably plan9). It was very useful for debugging but I'm indifferent as to whether we actually merge it.

…clude C files This reverts CL 351851, which itself reverted CL 348991. The problem with the original CL, as far as I can tell, was due to a bug in the Go project's builder infrastructure (#33598) and not the change itself. Once the build infrastructure is fixed, this change can be resubmitted. Fixes #48319 Updates #33598 Change-Id: I0fdbcc241eb2bdeb350944aad58bf58774fb591e Reviewed-on: https://go-review.googlesource.com/c/go/+/353352 Trust: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Jay Conrod <[email protected]>

This should be a no-op, but if the sharded builders are for some reason distributing stale snapshots — or testing them with mismatched environments — this should catch them out at a relatively low cost (#24300 notwithstanding). Given the frequently at which (*tester).runPending already checks for staleness, we do not expect the impact of this extra check to be significant for most builders. For #33598 Updates #24300 Change-Id: I197d6a69c72e2eec9e4563b459206de76c89e8a1 Reviewed-on: https://go-review.googlesource.com/c/go/+/223755 Trust: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>

gopherbot · 2022-05-23T20:00:15Z

Change https://go.dev/cl/407881 mentions this issue: [release-branch.go1.17] cmd/dist: consistently set PWD when executing a command in a different directory

… a command in a different directory Fixes #52995 Updates #33598 Change-Id: If0de906ffa2fcc83bb2a90f9e80a5b29d7667398 Reviewed-on: https://go-review.googlesource.com/c/go/+/353449 Trust: Bryan C. Mills <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]> (cherry picked from commit c035d82) Reviewed-on: https://go-review.googlesource.com/c/go/+/407881 Reviewed-by: Heschi Kreinick <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

gopherbot · 2022-07-18T19:44:36Z

Change https://go.dev/cl/367896 mentions this issue: buildlet,cmd/coordinator: clean up and document buildlet path invariants

For golang/go#53957. Updates golang/go#33598. Change-Id: Ie2c47dc0dfd82614b70c983f11462b70bd0703d2 Reviewed-on: https://go-review.googlesource.com/c/build/+/367896 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Bryan Mills <[email protected]>

bcmills added OS-Darwin NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 12, 2019

bcmills added this to the Go1.14 milestone Aug 12, 2019

bcmills changed the title ~~cmd/dist: "unexpected stale targets" for internal/cpu on darwin-amd64-10_11 builder~~ cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin-amd64-10_11 builder Aug 12, 2019

bcmills changed the title ~~cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin-amd64-10_11 builder~~ cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin builders Aug 27, 2019

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

bcmills changed the title ~~cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin builders~~ cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin and solaris builders Nov 21, 2019

bcmills mentioned this issue Nov 21, 2019

x/build: frequent "unexpected stale targets" on linux-arm-arm5spacemonkey after bootstrapping is complete #35740

Closed

bcmills mentioned this issue Nov 21, 2019

cmd/dist: "unexpected stale targets" on linux-ppc64le-power9osu builder #35749

Closed

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 18, 2020

bcmills changed the title ~~cmd/dist: "unexpected stale targets" during toolchain3 bootstrapping on darwin and solaris builders~~ cmd/dist: "unexpected stale targets" on darwin builders Mar 18, 2020

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 18, 2020

bcmills added the Builders x/build issues (builders, bots, dashboards) label Mar 18, 2020

bcmills changed the title ~~cmd/dist: "unexpected stale targets" on darwin builders~~ x/build: "unexpected stale targets" on darwin builders Mar 18, 2020

bcmills modified the milestones: Backlog, Go1.18 Oct 1, 2021

bcmills closed this as completed Oct 8, 2021

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 8, 2021

This was referenced Nov 19, 2021

x/build: "unexpected stale targets" on darwin-arm64-11_0-toothrot #49692

Closed

x/build: "unexpected stale targets" on plan9-arm #49691

Open

bcmills mentioned this issue Jan 13, 2022

os/exec: improved API and default behavior for the PWD env variable #50599

Closed

qmuntal mentioned this issue Jan 14, 2022

crypto: Use openssl instead of boring microsoft/go#364

Merged

bcmills mentioned this issue Jan 28, 2022

x/build/cmd/coordinator: failures are sometimes missing error output #39349

Closed

rsc unassigned bcmills Jun 23, 2022

heschi moved this to Done in Go Release Sep 27, 2022

heschi added this to Go Release Sep 27, 2022

golang locked and limited conversation to collaborators Jul 18, 2023

gopherbot added the FrozenDueToAge label Jul 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: "unexpected stale targets" on darwin builders #33598

x/build: "unexpected stale targets" on darwin builders #33598

bcmills commented Aug 12, 2019

bcmills commented Aug 27, 2019

bcmills commented Sep 11, 2019

bcmills commented Oct 22, 2019

laboger commented Oct 22, 2019

bcmills commented Nov 21, 2019

bcmills commented Nov 21, 2019

bcmills commented Feb 20, 2020

bcmills commented Mar 18, 2020

bcmills commented Mar 18, 2020

gopherbot commented Mar 18, 2020

bcmills commented Mar 18, 2020

bcmills commented Mar 18, 2020

bcmills commented Mar 18, 2020

toothrot commented Mar 18, 2020

gopherbot commented Oct 1, 2021

dmitshur commented Oct 6, 2021

dmitshur commented Oct 8, 2021

gopherbot commented Oct 8, 2021

bcmills commented Oct 8, 2021

gopherbot commented May 23, 2022

gopherbot commented Jul 18, 2022

x/build: "unexpected stale targets" on darwin builders #33598

x/build: "unexpected stale targets" on darwin builders #33598

Comments

bcmills commented Aug 12, 2019

bcmills commented Aug 27, 2019

bcmills commented Sep 11, 2019

bcmills commented Oct 22, 2019

laboger commented Oct 22, 2019

bcmills commented Nov 21, 2019

bcmills commented Nov 21, 2019

bcmills commented Feb 20, 2020

bcmills commented Mar 18, 2020

bcmills commented Mar 18, 2020

gopherbot commented Mar 18, 2020

bcmills commented Mar 18, 2020

bcmills commented Mar 18, 2020

bcmills commented Mar 18, 2020

toothrot commented Mar 18, 2020

gopherbot commented Oct 1, 2021

dmitshur commented Oct 6, 2021

dmitshur commented Oct 8, 2021

gopherbot commented Oct 8, 2021

bcmills commented Oct 8, 2021

gopherbot commented May 23, 2022

gopherbot commented Jul 18, 2022