Parallel execution is not functioning correctly with the 64-core arm runner.

[shusakurainosh]

Select Topic Area

Question

Body

detail:
We are currently performing end-to-end testing using 16 cores and 20 processes with Playwright (Chromium only) in headless mode. To expedite this process, we have configured a runner with 64 cores (ARM) and 80 processes. However, this setup is encountering issues: while the test itself executes correctly, it times out. This problem is likely due to a network issue with GitHub Actions, as running with 64 processes does not resolve the problem.

We have verified that this issue is not related to our server or other hardware specifications.

Could you please advise on how to address this problem?

[MiguelCardoso19]

You should try to decrease the total number in parallel way of working from eighty to a smaller figure, for instance forty or fifty; it is possible that this will help prevent overloading either the resources available for running it or its network, which are probably responsible for such prolonged periods with no response (timeouts). Continue adjusting the processes until you reach an optimal balance between them without having any more of these errors.

[shusakurainosh]

Thank you very much for your insights. You are absolutely correct that reducing the number of processes will lead to less latency. However, our expectation was that the runner’s performance would be quadrupled (along with the fee), which would reduce the execution time to one-quarter.

[shusakurainosh]

Are there any other approaches we might consider? For example, would it be feasible to set up two RUNNERs, each handling 40 processes? Specifically, one RUNNER with 64 cores and 40 processes, and another with 64 cores and 40 processes, resulting in a total of 80 processes. This approach would be applicable only if network performance is influenced by each RUNNER individually.

[MiguelCardoso19]

Yes, it is reasonable to have two runners with 64 cores each that work and process loads per 40 processes This could alleviate some of the congestion and prevent a single runner from tying up an entire network/realm having to run 80 processes. If it's true that network performance is influenced by each runner individually, you should expect to have better executions of your tests without timeouts.

[shusakurainosh]

I will test this idea next week. If it works as expected, I will mark the above response accordingly.

[shusakurainosh]

This approach proved to be effective as anticipated. By dividing the jobs (runners), we were able to allocate the maximum network resources to each job. This allowed us to successfully run 40 processes in parallel across 4 jobs on a 64-core system.

[MiguelCardoso19]

Happy to know that its fixed!

[shusakurainosh]

Thank you!