Quitting master doesnt prevent its heartbeat_worker greenlet from trying to re-establish connections

[cyberw]

[2022-01-14 13:20:02,660] perftest04/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:02,651] master-machine/INFO/locust.runners: Client 'perftest04_1acb085b6d9a4d22b6ba31f0cedb5f0d' quit. Currently 0 clients connected.
[2022-01-14 13:20:02,717] perftest04/INFO/locust.main: Shutting down (exit code 0), bye.
[2022-01-14 13:20:02,717] perftest04/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:02,710] master-machine/INFO/locust.runners: Client 'perftest04_808d502f37ec43a4978766d95fe5ae5e' quit. Currently 0 clients connected.
[2022-01-14 13:20:02,811] master-machine/ERROR/locust.runners: RPCError found when receiving from client: ZMQ network broken
[2022-01-14 13:20:02,812] master-machine/INFO/locust.main: Shutting down (exit code 0), bye.
[2022-01-14 13:20:02,812] master-machine/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:02,812] master-machine/INFO/locust.util.exception_handler: Exception found on retry 1: -- retry after 1s
[2022-01-14 13:20:02,812] master-machine/ERROR/locust.util.exception_handler: ZMQ sent failure
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
[2022-01-14 13:20:03,219] perftest03/INFO/locust.main: Shutting down (exit code 0), bye.
[2022-01-14 13:20:03,219] perftest03/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:03,839] master-machine/INFO/locust.util.exception_handler: Exception found on retry 2: -- retry after 3s
[2022-01-14 13:20:03,839] master-machine/ERROR/locust.util.exception_handler: ZMQ sent failure
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
[2022-01-14 13:20:06,841] master-machine/INFO/locust.util.exception_handler: Exception found on retry 3: -- retry after 5s
[2022-01-14 13:20:06,841] master-machine/ERROR/locust.util.exception_handler: ZMQ sent failure
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
[2022-01-14 13:20:11,846] master-machine/INFO/locust.util.exception_handler: Retry failed after 3 times.
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/locust", line 33, in <module>
    sys.exit(load_entry_point('locust', 'console_scripts', 'locust')())
  File "/Users/myusername/git/locust/locust/main.py", line 483, in main
    shutdown()
  File "/Users/myusername/git/locust/locust/main.py", line 453, in shutdown
    runner.quit()
  File "/Users/myusername/git/locust/locust/runners.py", line 821, in quit
    self.server.send_to_client(Message("quit", None, client.id))
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
2022-01-14:13:20:11,969 INFO [swarm:436] Load gen master process finished (return code 1)

@delulu I think this was introduced in your code a long time ago, but for some reason I started seeing it just now (perhaps after switching from RHEL to Ubuntu for my workers?). I have a fix that I will PR very soon.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity. This does not necessarily mean that the issue is bad, but it most likely means that nobody is willing to take the time to fix it. If you have found Locust useful, then consider contributing a fix yourself!