Unhandled exception: ConnectionResetError, Connection reset by peer (FastHttpUser)

[bendizen]

Describe the bug

I running a test pulling large files (some > 250mb) using the fast client, when connection gets reset ("ConnectionResetError: [Errno 104] Connection reset by peer") on some files it throws an unhandled exception and does not log a failure.

Exception

AttributeError: _cached_content
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/user/task.py", line 284, in run
    self.execute_next_task()
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/user/task.py", line 309, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/user/task.py", line 416, in execute_task
    task(self.user)
  File "/home/mkarlovich/source/locust-plugins/examples/fast_csvreader_cip.py", line 16, in index
    self.client.get(url, headers=headers)
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 283, in get
    return self.request("GET", path, **kwargs)
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 238, in request
    request_meta["content_size"] = len(response.content or "")
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 186, in content
    self._cached_content = self._content()
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 351, in _content
    return super(FastResponse, self)._content()
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 201, in _content
    ret = self._response.read()
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/geventhttpclient/response.py", line 259, in read
    data = self._sock.recv(length or self.block_size)
  File "/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/gevent/_socket3.py", line 450, in recv
    return self._sock.recv(*args)
ConnectionResetError: [Errno 104] Connection reset by peer

Expected behavior

It should handle this exception and report it as a failure.

Actual behavior

Workers throw exception but no failure is recorded

Steps to reproduce

I can't reproduce it except in my test environment but I have a fix that seems to work: Add ConnectionResetError to FAILURE_EXCEPTIONS in fasthttp.py.

# List of exceptions that can be raised by geventhttpclient when sending an HTTP request, 
# and that should result in a Locust failure
FAILURE_EXCEPTIONS = (ConnectionError, ConnectionRefusedError, ConnectionResetError, socket.error, \
                      SSLError, Timeout, HTTPConnectionClosed)

Environment

OS: Ubuntu 16.04 master and workers
Python version: 3.8.2
Locust version: 1.0.3
Locust command line that you ran:
locust -f fast_csvreader_ex.py --master --expect-workers 10 --headless -u 100 -r 1 -t 60m --csv=test100u60m10w
locust -f fast_csvreader_ex.py --worker --master-host=10.66.11.21 & # run 5 per worker machine

Locust file contents (anonymized if necessary):

from locust_plugins.csvreader import CSVReader
from locust import task
from locust.contrib.fasthttp import FastHttpUser
import traceback, logging, sys
import locust.stats
locust.stats.CSV_STATS_INTERVAL_SEC = 5

id_reader = CSVReader("ids.csv")

class MyUser(FastHttpUser):
    @task
    def index(self):
        headers = { "Authorization": "apikey", "Accept": "application/json" }
        feed_id, file_id = next(id_reader)
        url = f'/feeds/{feed_id}/files/{file_id}'
        self.client.get(url, headers=headers)
        
    host = 'https://my-test-sever.com'
    connection_timeout = 6000.0
    network_timeout = 6000.0

[cyberw]

Hi! Can you make a PR?

[bendizen]

Yes will do later today, thanks.

[bendizen]

added PR #1475

[cyberw]

Thx!