tests seem to hang on Mac on GitHub Actions

[mvdan]

For example: https://github.com/ipld/go-ipld-prime/pull/77/checks?check_run_id=1083075042

This seems to happen rather frequently; we've seen six of these in just a week, where twenty or so workflows were run.

GitHub confirmed this doesn't seem like a bug on their end, because the process is still running. But how can that be, since a test package execution should timeout after ten minutes by default? I'm pretty sure that's enforced via kill signals, too.

I'm opening this issue to investigate.

[alepauly]

GitHub confirmed this doesn't seem like a bug on their end, because the process is still running.

@mvdan Just to clarify, we haven't ruled out an issue on our end but the problem here seems to be something is holding on to the runner for the full 6 hours at which time the service times it out. We'll continue digging on the service side but if you can help us by confirming on the workflow side that nothing is keeping the agent process busy, that would be great. Please let us know what you find and I'll update here as soon as we know more. Thanks!

[mvdan]

Thanks! The PR #77 seems to make the hang far more likely, because it makes some of the heavier tests run in parallel. I'm going to push some extra stuff in that PR to try to see what's happening when it hangs.

[mvdan]

We never found out the "why", but I assume it has something to do with all the concurrent builds that get triggered, which honestly aren't a great idea to begin with. For some reason, a bunch of go build calls were hanging or crashing in some way that confused go test.

In any case, though, the fact that go test didn't obey its default 10m timeout is a Go bug. See golang/go#24050 (comment).

[BigLep]

2022-04-05 conversation: @rvagg will talk to @mvdan about this, and confirming that this is the only thing blocking unified CI.

[rvagg]

Going to close this since #400 is merged to deal with the consequences of this problem, although not really the cause - there's TODOs in the code now to point to it.