Worker sometimes fails to send heartbeat after upgrade to urllib3>=1.26.16

[tdadela]

Prerequisites

• I am using the latest version of Locust
• I am reporting a bug, not asking a question

Description

After upgrade of urllib3 (indirect dependency of Locust) to version >=1.26.16 locust started working improperly.
After change (for one of our test, 80 processes on 20 machines, duration: 1 hour) there is ~50% probability of (one or more) worker disconnections "Worker XXX failed to send heartbeat, setting state to missing".
After forking locust to add more logs, it became clear it happens because workers sometimes stop sending heartbeat. (It's not for example network issue. It isn't also high CPU usage).

Changes in urllib3:

• 1.26.16 which is backport of
• 2.X

Issues:

• gevent and weakref.finalize potentially cause non-deterministic deadlock gevent/gevent#1957
• After upgrade to 1.26.16 connection pool blocks at max size during CI tests urllib3/urllib3#3061

Credit to @domik82 for finding them.

After applying this solution, worker disconnections disappeared. (10 performance tests with 80 processes on 20 machines, duration: 1 hour)

Command line

locust -f locustfile.py

Locustfile contents

N/A

Python version

3.11.1

Locust version

2.29.1 (and older)

Operating system

CentOS 7

[cyberw]

Amazing work figuring it out. Will you make a PR, or do you want me to do it? You can title it something like "avoid deadlock in gevent/urllib3 connection pool (fixes worker heartbeat timeouts)"