x/build: frequent "unexpected stale targets" on linux-arm-arm5spacemonkey after bootstrapping is complete #35740
Labels
Builders
x/build issues (builders, bots, dashboards)
FrozenDueToAge
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Comments
bcmills
added
Builders
|
This failure mode is now happening on every build. @esnolte, you are listed as the owner of that builder here. Could you please investigate? I wonder if the problem is somehow that multiple builds are running at the same time. (A similar failure mode in #35749 turned out to be an older builder running concurrently with the current one.) |
|
The builder is in a loop filling the logs and eventually the filesystem with:
Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloading https://storage.googleapis.com/go-builder-data/buildlet.linux-arm-arm5 to ./buildlet.exe ...
Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloaded ./buildlet.exe (12317133 bytes)
Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloaded buildlet in 200ms
Dec 18 19:01:57 localhost logger: 2019/12/18 19:01:57 buildlet starting.
Dec 18 19:01:57 localhost logger: 2019/12/18 19:01:57 unlinkat /home/builder/stage0scratch/workdir/tmp/TestModReplace2195897861/gopath/pkg/mod/rsc.io/[email protected]/glass_test.go: permission denied
Here is the service we're running:
root@go-builder-2:/home/builder# cat /etc/service/stage0/run
#!/bin/bash
set -e
SCRATCH=~builder/stage0scratch
export TMPDIR=${SCRATCH}/tmp
export WORKDIR=${SCRATCH}/workdir
mkdir -p ${TMPDIR}
mkdir -p ${WORKDIR}
chown -R builder:builder ${SCRATCH}
cd ${SCRATCH}
export GO_BUILDER_ENV=linux-arm-arm5spacemonkey
export GO_TEST_TIMEOUT_SCALE=5
export USER=builder
export HOME=/home/builder
exec bash -c "setuidgid builder /usr/local/bin/stage0 2>&1 | logger"k
The builder is a binary installed in 2017:
root@go-builder-2:/usr/local/bin# ls -l
total 4912
-rwxr-xr-x 1 root root 5013776 Aug 29 2017 stage0
root@go-builder-2:/usr/local/bin# file stage0
stage0: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped
root@go-builder-2:/usr/local/bin# sha1sum stage0
74ae93f72bcb193778383fd7113a5310f74c2e7c stage0
Should I delete stage0scratch and see if that helps or am I running an old builder and need to upgrade it?
- Erik
… On 18 Dec 2019, at 10:54, Bryan C. Mills ***@***.***> wrote:
This failure mode is now happening on every build.
@esnolte <https://github.com/esnolte>, you are listed as the owner of that builder here <https://github.com/golang/build/blob/6b2867eab46056b7c8e75fbfadfa40dc43dbdf91/dashboard/builders.go#L217>. Could you please investigate?
I wonder if the problem is somehow that multiple builds are running at the same time. (A similar failure mode in #35749 <#35749> turned out to be an older builder running concurrently with the current one.)
|
|
I deleted stage0scratch and restarted the builder and things look better. I'm not sure when the results will show up on https://build.golang.org/# <https://build.golang.org/#>
… On 18 Dec 2019, at 12:11, Erik Nolte ***@***.***> wrote:
The builder is in a loop filling the logs and eventually the filesystem with:
Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloading https://storage.googleapis.com/go-builder-data/buildlet.linux-arm-arm5 <https://storage.googleapis.com/go-builder-data/buildlet.linux-arm-arm5> to ./buildlet.exe ...
Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloaded ./buildlet.exe (12317133 bytes)
Dec 18 19:01:57 localhost logger: stage0: 2019/12/18 19:01:57 downloaded buildlet in 200ms
Dec 18 19:01:57 localhost logger: 2019/12/18 19:01:57 buildlet starting.
Dec 18 19:01:57 localhost logger: 2019/12/18 19:01:57 unlinkat ***@***.***/glass_test.go: ***@***.***/glass_test.go:> permission denied
Here is the service we're running:
***@***.***:/home/builder# cat /etc/service/stage0/run
#!/bin/bash
set -e
SCRATCH=~builder/stage0scratch
export TMPDIR=${SCRATCH}/tmp
export WORKDIR=${SCRATCH}/workdir
mkdir -p ${TMPDIR}
mkdir -p ${WORKDIR}
chown -R builder:builder ${SCRATCH}
cd ${SCRATCH}
export GO_BUILDER_ENV=linux-arm-arm5spacemonkey
export GO_TEST_TIMEOUT_SCALE=5
export USER=builder
export HOME=/home/builder
exec bash -c "setuidgid builder /usr/local/bin/stage0 2>&1 | logger"k
The builder is a binary installed in 2017:
***@***.***:/usr/local/bin# ls -l
total 4912
-rwxr-xr-x 1 root root 5013776 Aug 29 2017 stage0
***@***.***:/usr/local/bin# file stage0
stage0: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped
***@***.***:/usr/local/bin# sha1sum stage0
74ae93f72bcb193778383fd7113a5310f74c2e7c stage0
Should I delete stage0scratch and see if that helps or am I running an old builder and need to upgrade it?
- Erik
> On 18 Dec 2019, at 10:54, Bryan C. Mills ***@***.*** ***@***.***>> wrote:
>
> This failure mode is now happening on every build.
>
> @esnolte <https://github.com/esnolte>, you are listed as the owner of that builder here <https://github.com/golang/build/blob/6b2867eab46056b7c8e75fbfadfa40dc43dbdf91/dashboard/builders.go#L217>. Could you please investigate?
>
> I wonder if the problem is somehow that multiple builds are running at the same time. (A similar failure mode in #35749 <#35749> turned out to be an older builder running concurrently with the current one.)
>
|
|
That doesn't seem to have fixed the problem. (https://build.golang.org/log/08d3ac2849287e18678df797c542b7329dab3d3e) |
|
I've unplugged a rogue builder we no longer have access to (go-builder-3). It looks like the other 2 builders are passing tests.
… On 19 Dec 2019, at 07:41, Bryan C. Mills ***@***.***> wrote:
That doesn't seem to have fixed the problem. (https://build.golang.org/log/08d3ac2849287e18678df797c542b7329dab3d3e <https://build.golang.org/log/08d3ac2849287e18678df797c542b7329dab3d3e>)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#35740?email_source=notifications&email_token=ADY27VMKXKRTK2HR6ZV35DLQZOB2RA5CNFSM4JQDMVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJZ3AA#issuecomment-567516544>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADY27VMKJ5CK5SQ6OGPDG4TQZOB2RANCNFSM4JQDMVCA>.
|
|
Hmm, now https://farmer.golang.org/ shows 0/2
|
By now, this seems to be accurately reflected under the "Reverse pool by host type (in use / total)" section of https://farmer.golang.org/#pools: I'm not sure why it was showing something else before. |
|
I rebooted everything 30 minutes ago. Maybe that unwedged the builders.
… On 19 Dec 2019, at 11:42, Dmitri Shuralyov ***@***.***> wrote:
I've unplugged a rogue builder we no longer have access to (go-builder-3). It looks like the other 2 builders are passing tests.
By now, this seems to be accurately reflected under the "Reverse pool by host type (in use / total)" section of https://farmer.golang.org/#pools <https://farmer.golang.org/#pools>:
host-linux-arm5spacemonkey: 2/2 (1 missing)
I'm not sure why it was showing something else before.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#35740?email_source=notifications&email_token=ADY27VJ6QHJGU7A5EBVH7P3QZO6C3A5CNFSM4JQDMVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHKRGIA#issuecomment-567612192>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADY27VKLN4B6LLK4MIMUF4TQZO6C3ANCNFSM4JQDMVCA>.
|
|
The original problem is still happening. |
|
IIRC, the arm5 builders start with a cross-compiled make.bash from a linux-amd64 machine. Perhaps the two environments have drifted? |
|
What is the next step? Should I rebuild stage0 with the most recent source?
I've cloned https://github.com/golang/build and tried building build/cmd/buildlet/stage0 but am missing dependencies like golang.org/x/build/internal/{httpdl,untar} and cloud.google.com/go/compute/metadata.
Are there directions for building stage0?
… On 27 Dec 2019, at 14:45, Brad Fitzpatrick ***@***.***> wrote:
IIRC, the arm5 builders start with a cross-compiled make.bash from a linux-amd64 machine. Perhaps the two environments have drifted?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Change https://golang.org/cl/346530 mentions this issue: |
gopherbot
pushed a commit
to golang/build
that referenced
this issue
Aug 31, 2021
This removes the linux-arm-arm5spacemonkey builders. They have been missing for an extended period of time and there is no timeline for fixing them. Updates golang/go#35740 Fixes golang/go#41027 Change-Id: I4a7bdbf890f4a35201a826d2049693dd2eec6524 Reviewed-on: https://go-review.googlesource.com/c/build/+/346530 Trust: Carlos Amedee <[email protected]> Run-TryBot: Carlos Amedee <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Go Bot <[email protected]>
|
Closing this issue as the associated builder has been removed as part of #41027. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2019-11-20T22:38:34-39a9cb4/linux-arm-arm5spacemonkey
2019-11-20T22:37:23-be04718/linux-arm-arm5spacemonkey
2019-11-19T06:10:03-8cf5293/linux-arm-arm5spacemonkey
2019-11-19T06:08:56-1046a9c/linux-arm-arm5spacemonkey
2019-11-16T20:31:45-6ba0be1/linux-arm-arm5spacemonkey
2019-11-16T02:06:39-c20b71e/linux-arm-arm5spacemonkey
2019-11-16T00:02:02-5042317/linux-arm-arm5spacemonkey
2019-11-15T22:47:41-72f333a/linux-arm-arm5spacemonkey
See also #33598, but the failure mode here is somewhat different: in that issue the failure always occurs during toolchain3 bootstrapping, whereas here it always occurs after
make.bashis complete.CC @jayconrod @cagedmantis @toothrot @dmitshur @bradfitz
The text was updated successfully, but these errors were encountered: