Pi3 troubles

[rvagg]

Since I redid the Pi boot disk arrangement a few weeks ago, the Pi 3's have been having problems staying online. The Pi 1's and 2's seem to be just fine, all of them are online and none seem to have needed fixing. The 1's are used less, but the 2's should be used as much as the 3's. I think the git-update-nodesource job might be pinned to the 3's but that should be the main difference. Yet twice now I've had to deal with more than half of them being unresponsive. Right now, all but 1, 6 and 9 (ordered by the numbers they appear in inventory) were offline. I've started them all up again and am doing an update on just the 3's just in case there's something in there that helps out.

I'm suspecting OS problems that are impacting just this generation of device. It could also be related to Docker, which is a continual source of trouble on ARM. As yet I have no evidence of what might be locking them up and am just making guesses.

Please help keep an eye on them, I might need reminding if too many disappear. It'll also be interesting to note which ones are impacted, there may be a pattern.

[BethGriggs]

I'm not sure if this has been raised elsewhere (I couldn't spot a recent build issue for it), but it seems we're down to 2 Pis @ https://ci.nodejs.org/label/pi3-raspbian-stretch/

[richardlau]

I mentioned it in the build Slack channel yesterday. I wasn't able to ssh into any of the offline pi3's.

[rvagg]

I'm on it. The Pi3's have been very unstable and I haven't been able to make much of a dent on their stability. I'll try another update today to see if something new comes in to solve this.

In the meantime, if this bottleneck becomes chronic, disable the arm-fanned job entirely to remove it from consideration, it's not the biggest loss and I can usually get ontop of it within 24 hours if notified.

[BethGriggs]

It seems like the a number of Pis have recently dropped off (also reported in Slack by @Trott).

We seem to be down to one Pi 3 device in CI again? Or at least I see only one working on anything in the node-test-binary-arm-12+ jobs.

I think it's starting to backlog (trend is showing a few 4h+ builds). Would someone be able to disable the arm-fanned job temporarily?

[richardlau]

It seems like the a number of Pis have recently dropped off (also reported in Slack by @Trott).

We seem to be down to one Pi 3 device in CI again? Or at least I see only one working on anything in the node-test-binary-arm-12+ jobs.

I think it's starting to backlog (trend is showing a few 4h+ builds). Would someone be able to disable the arm-fanned job temporarily?

I've disabled node-test-commit-arm-fanned for now. I'm unable to ssh into the offline pi3's.

cc @nodejs/build-infra

[richardlau]

Copying over from Rod's comment in #2510 (comment):

Also did some quick maintenance on the Pi cluster and they are back online again and I've reenabled arm-fanned. I'm running some jobs through it now to fill up the workspaces I just emptied. there's 5 Pi1's marked offline and 4 Pi2's marked offline, nothing wrong with them (afaik) but I've done this to stop the workspaces from filling up unnecessarily. Especially with the Pi1's that are hardly ever run now. They're essentially "in reserve" for when other Pi's get knocked off. Pi 3's are all online because we have so much trouble keeping them up that I don't have much hope for the full suite staying afloat for long.

[rvagg]

https://ci.nodejs.org/label/pi3-docker/ 3 are already offline, sigh, I'll do my best to keep them alive but I might need to be reminded now and again

[Trott]

I disabled the arm-fanned job yesterday. https://node-js.slack.com/archives/C16SCB5JQ/p1610575369213600

[targos]

I opened #2523 yesterday about that.

[rvagg]

I've re-enabled arm-fanned, it's working and I got greens on v15.x, but I think master might be in a poor state as far as the Pi's are concerned, I don't know if things have slipped since it's been disabled or if we just have a large build-up of flakeys but it's not looking so awesome.

Feel free to disable again if there's something that's genuinely a problem - like holding up a release or consistently taking >1h to complete. Otherwise it should be treated as a valid test environment that needs to be taken care of when things break.

btw: I think it would be nice if it were disabled @ https://ci.nodejs.org/job/node-test-commit-arm-fanned/ rather than within node-test-commit itself, it's a bit tucked away in there but is a nice big "Disable" and "Enable" button in the job itself and it does the same thing.

[richardlau]

Linking #2583.

[Trott]

I put this in Slack/IRC too: Down to a single pi3 host on Jenkins again. https://ci.nodejs.org/label/pi3-raspbian-stretch/ This is an annoying-but-tolerable bottleneck on the weekend. It's going to be pretty bad come Monday and is probably not great for the security release that's supposed to go out on Tuesday. I'll see if I can find out what to do from the build repo beyond disabling Pi testing altogether.

[rvagg]

Brought a bunch back online, a few still that won't come up. I'm going to try and find some time to play with getting these upgraded to the newer raspbian (now "Raspberry Pi OS"), I think this is a kernel issue that isn't being fixed because the OS is too old.

[Trott]

@rvagg It looks like all Pi devices are failing everything now. NFS problem? I'm seeing things like this:

21:18:43 os.unlink() [Errno 30] Read-only file system: '/tmp/tmpHLUubm'
21:18:43 os.unlink() [Errno 30] Read-only file system: '/tmp/tmpRXPop4'

...and this:

21:18:42 error: unable to create file deps/v8/src/execution/runtime-profiler.h: File exists
21:18:42 error: unable to unlink old 'deps/v8/src/execution/s390/frame-constants-s390.cc': Read-only file system
21:18:42 error: unable to unlink old 'deps/v8/src/execution/s390/frame-constants-s390.h': Read-only file system
21:18:43 error: unable to unlink old 'deps/v8/src/execution/s390/simulator-s390.cc': Stale file handle
21:18:43 warning: unable to access 'deps/v8/src/execution/s390/.gitattributes': Stale file handle
21:18:43 error: unable to create file deps/v8/src/execution/s390/simulator-s390.h: Stale file handle
21:18:43 error: unable to create file deps/v8/src/execution/shared-mutex-guard-if-off-thread.h: Stale file handle
21:18:43 error: unable to create file deps/v8/src/execution/simulator-base.h: Stale file handle

[rvagg]

yeah, nfs server crash, getting it all unblocked now @Trott

[Trott]

It's a Pi2 and not a Pi3, but https://ci.nodejs.org/computer/test-requireio_rvagg-debian10-armv7l_pi2-1/ has been build-failing consistently so I marked it offline. As far as I can tell, everything else is in working order at the moment.

[richardlau]

I've restarted the agent on https://ci.nodejs.org/computer/test-requireio_rvagg-debian10-armv7l_pi2-1/.

[Trott]

Looks like we've been down to just two working/online Raspberry PI 3 devices for the past 24 hours or so. https://ci.nodejs.org/label/pi3-raspbian-stretch/

[rvagg]

my big firmware and in-place upgrades didn't do the trick
But the most interesting thing right now is that one of the devices that hasn't gone down is the one I did a full reinstall of, using a fresh version of RasperryPi OS (the new "Raspbian"), which really shouldn't be different from what the others got in-place upgraded to. But maybe there's something in the replace, or something else I did along the way, that made a difference. So I suppose I need to go through the rest and give them the same treatment. In the meantime I'll get the others restarted and get some more back in the cluster.

[richardlau]

Back to two Pi 3's again and those two are failing to remove a file (nfs):

10:36:31 warning: failed to remove out/Release/.nfs00000000013e2db900001996: Device or resource busy

https://ci.nodejs.org/job/node-test-binary-arm-12+/10387/RUN_SUBSET=0,label=pi3-docker/console

[richardlau]

I removed the workspace on the two working Pi 3's 🤞.

[rvagg]

we have more back online but they probably won't last!

[richardlau]

FWIW (this is a Pi 2 so not sure if it belongs here) I noticed that test-requireio_rvagg-debian10-armv7l_pi2-1 was marked offline for "repeated build failures". I logged into the machine and saw lots of running node processes that were preventing the workspace from being cleaned:

pi@test-requireio--rvagg-debian10-armv7l--pi2-1:~ $ ps -ef | grep "\/node"
iojs     10888   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10894   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10895   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10896   506  0 Apr25 ?        00:00:03 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10898   506  0 Apr25 ?        00:00:03 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10904   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10909   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10921   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10928   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10929   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10951   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10969   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10970   506  0 Apr25 ?        00:00:03 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10971   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10987   506  0 Apr25 ?        00:00:03 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10989   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10990   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     10992   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11005   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11006   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11019   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11020   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11025   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11042   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11043   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11048   506  0 Apr25 ?        00:00:03 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11061   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11064   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11065   506  0 Apr25 ?        00:00:03 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11067   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11069   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11075   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11099   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11113   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11119   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11134   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11140   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11147   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11149   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11150   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11160   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11170   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11171   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11188   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11190   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11197   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11202   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11203   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11215   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11222   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11234   506  0 Apr25 ?        00:00:05 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11237   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11243   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11244   506  0 Apr25 ?        00:00:04 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11246   506  0 Apr25 ?        00:00:05 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11257   506  0 Apr25 ?        00:00:05 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11261   506  0 Apr25 ?        00:00:05 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11262   506  0 Apr25 ?        00:00:03 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11266   506  0 Apr25 ?        00:00:06 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11268   506  0 Apr25 ?        00:00:05 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11296   506  0 Apr25 ?        00:00:05 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11303   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11321   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11327   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11328   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11332   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11333   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11336   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11349   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11350   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11354   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11357   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11359   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11362   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11363   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11364   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11366   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11368   506  0 Apr25 ?        00:00:02 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11369   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11374   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11377   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
iojs     11451   506  0 Apr25 ?        00:00:01 /home/iojs/build/workspace/node-test-binary-arm/out/Release/node -e process.chdir("/home/iojs/build/workspace/node-test-binary-arm/test/pummel");         for (const file of fs.readdirSync('.'))           fs.watch(file, () => {});
pi       18018 17910  0 18:02 pts/0    00:00:00 grep --color=auto \/node
pi@test-requireio--rvagg-debian10-armv7l--pi2-1:~ $

(looks related to pummel?). I've cleaned up the processes and brought the host back online.

ps -ef | grep "Release\/node" | awk '{print $2}' | xargs -rl kill

[richardlau]

I've added

# Kill any running node processes, so they don't lock files in the workspace
ps -ef | grep "Release\/node" | awk '{print $2}' | xargs -rl kill || true

after the existing

# kill any [node] <defunct> zombies, so they don't lock files in the workspace
ps -ef | grep "\[node\] <defunct>" | awk '{print $2}' | xargs -rl kill || true

to kill any running node processes as well as any defunct zombies. This should hopefully reduce the number of build failures when git clean is unable to clean the workspace due to a leftover running process from a previous run.

[Trott]

It appears that we're down to 3 online Pi 3 devices again. I wonder if it's the same 3 every time. That does seem to be the magic number.

[rvagg]

I think it might be. For the record, 1, 3 and 6 (according to my numbering which is different to the public naming .. sorry) are still online. I remember the last time (last week?) I did this, 1 and 3 were the ones still alive (can't remember which one was the third one still alive), and 3 was the one I reprovisioned from scratch recently. Will give the rest a bump right now.

[rvagg]

status update in #2661, I think we can close this but I'll continue to monitor the situation