x/build: illumos-amd64 builder consistently failing

[bcmills]

The illumos-amd64 builder has timed out testing cmd/go in every run since CL 473697.

Since the code involved in that change has literally no effect on illumos-amd64, I suspect that something has changed in either the builder itself or its configuration in cmd/coordinator to cause the test to become much slower.

[bcmills]

(attn @jclulow; CC @golang/illumos @golang/release)

[bcmills]

Looking at a few of the failures, it's not just cmd/go — there have also been timeouts in at least runtime and testing.

[jclulow]

As an initial note on this: nothing about the builder has changed to the best of my knowledge, unless AWS have moved it onto a defective or noisy host somehow.

[bcmills]

The failure mode has changed: it is now consistently failing with no space left on device errors during bootstrapping.

[jclulow]

I will investigate!

[jclulow]

I believe disk space exhaustion was caused by the build-up of files in $HOME/.cache/gopls...

ncdu 1.15.1 ~ Use the arrow keys to navigate, press ? for help
--- /home/gobuild/.cache -------------------------------------------------------
                         /..
   18.1 GiB [##########] /gopls
  133.7 MiB [          ] /staticcheck
   40.8 MiB [          ] /go-build
   17.0 KiB [          ] /screentest

 Total disk usage:  18.3 GiB  Apparent size:   6.0 GiB  Items: 1406542

Should these files be deleted automatically after the job completes? Can I delete them now?

[jclulow]

NB: in the interim I have activated the warm standby VM here which obviously doesn't (yet!) have a full disk, and appears to be processing jobs OK.

[bcmills]

@jclulow, it appears that the illumos-amd64 builder is completely disconnected from the coordinator at the moment.

[jclulow]

@jclulow, it appears that the illumos-amd64 builder is completely disconnected from the coordinator at the moment.

Oh no!

I'm looking at it now, and the issue appears to be that the buildlet user has consumed all 10G of its quota. (I set the quota to less than the VM disk size on this one, after the last builder filled the entire disk in the machine making it harder to manage.)

I'm running ncdu at the moment, which is taking a while because there are a lot of tiny files. They are once again concentrated in /home/gobuild/.cache/gopls like last time.

@bcmills What is responsible for managing the contents of $HOME/.cache/gopls? Should these files be deleted automatically after the job completes? Can I delete them now?

[heschi]

Yeah, you can delete them whenever you need to. Maybe a cron job? :-/

@golang/tools-team if you can find a way to make this stop happening that'd be great. I remain unconvinced that a temporary directory that persists across test runs is the right approach.

[findleyr]

@heschi I thought we already did this in https://go.dev/cl/494297? Does that miss something?

[heschi]

Hm. Forgot about that. @jclulow, does this machine auto-update the buildlet?

[jclulow]

Hm. Forgot about that. @jclulow, does this machine auto-update the buildlet?

We run the stage0 binary under our service manager, so, if that auto-updates the buildlet then it should do that yes. Indeed it was, when I looked at it, trying to do that but the disk was full so it was unable. It's also been offline for a while, it seems, so maybe that fix happened after the outage started?

I'm currently clearing out .cache/gopls to make room and I'll kick it off after that and we'll see!

[jclulow]

I have re-enabled the thing and it obtained a new buildlet and is ostensibly doing work now.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)