runtime: TestPingPongHog failures with ratios out of range

[bcmills]

--- FAIL: TestPingPongHog (0.10s)
    proc_test.go:486: want hogCount/lightCount in [0.2, 5]; got 24000000/133792000 = 0.1793829227457546
FAIL
FAIL	runtime	155.091s

See previously #38266, #20494.

greplogs --dashboard -md -l -e 'FAIL: TestPingPongHog .*(?:\n .*)*want hogCount/lightCount in ' --since=2021-01-01

2022-04-06T20:46:47-81ae993/linux-386-longtest
2022-03-10T21:10:30-914195c/windows-amd64-longtest
2021-11-12T22:20:50-9150c16/linux-386-longtest
2021-10-29T17:12:47-3aecb3a/linux-386-longtest

(CC @golang/runtime)

[bcmills]

Marking as release-blocker for Go 1.19 because windows/386 and windows/amd64 are first class ports.

(This at least needs to be triaged by the runtime team before the release, and they can either fix the failure or add a skip based on the outcome of that triage.)

[heschi]

Ping: any progress on this?

[bcmills]

greplogs --dashboard -md -l -e 'FAIL: TestPingPongHog .*(?:\n .*)*want hogCount/lightCount in ' --since=2022-04-07
2022-04-30T04:04:40-fd6c556/windows-amd64-longtest
2022-04-30T02:35:50-0ebe224/windows-amd64-longtest
2022-04-25T23:36:50-a5d61be/windows-amd64-longtest

[bcmills]

greplogs -l -e 'FAIL: TestPingPongHog .*(?:\n .*)*want hogCount/lightCount in ' --since=2022-05-01
2022-05-03T14:36:37-d0cda4d/linux-386-longtest

[bcmills]

greplogs -l -e 'FAIL: TestPingPongHog .*(?:\n .*)*want hogCount/lightCount in ' --since=2022-05-04
2022-05-06T15:32:04-7f71e1f/windows-amd64-longtest
2022-05-06T04:22:04-3ea3bc0/windows-amd64-longtest
2022-05-06T02:40:50-7dd9884/windows-amd64-longtest
2022-05-04T04:21:29-7cf3268/windows-amd64-longtest

and https://storage.googleapis.com/go-build-log/84130882/linux-amd64-longtest_b0bacea2.log (a SlowBot)

@golang/runtime: can this test be skipped until someone has the bandwidth to fix it? It's gotten really noisy.

[cherrymui]

The code comment says

// We can use a
// fairly large factor here to make this robust: if the
// scheduler isn't working right, the gap should be ~1000X.

All failures above are within a factor of 10 (the current factor is 5). Perhaps we can just increase the factor, to (say) 20 or 50?

[mknyszek]

I agree with Cherry. It sounds like this just needs a lot more slack. However, I think this test might just be fundamentally flaky because it's not robust to OS scheduling delays (I'm thinking about #52433 here). I wonder if we could prove that this is the case with perf on the linux/386 builders.

This is definitely a test that, if we had, for instance, scheduler instrumentation for testing, we could construct a situation where we make sure that a goroutine gets scheduled within N calls to the scheduler on average (or something), instead of bounding it by real timings.

[cherrymui]

Yeah, scheduler instrumentation would be much better for testing this kind of things.

Perhaps for short term we could increase the threshold to reduce the noise on builders.

[gopherbot]

Change https://go.dev/cl/407134 mentions this issue: runtime: relax the threshold for TestPingPongHog

[gopherbot]

Change https://go.dev/cl/407415 mentions this issue: runtime: fix overflow in PingPongHog test