Locust starts throwing failures when users > 130 (OS X)

[rafalf]

I got locust installed on Macbook Air
The test is very basic, only one page is tested:

---

from locust import HttpLocust, TaskSet, task

class UserBehavior(TaskSet):
    def on_start(self):
        pass

    @task
    def profile(self):
        self.client.get("/nonprofit/ifrc/articles/125585")

class WebsiteUser(HttpLocust):
    task_set = UserBehavior
    min_wait=5000
    max_wait=15000

---

and the user load is very small obviously.

GET /nonprofit/ifrc/articles/125585 ConnectionError(MaxRetryError("HTTPConnectionPool(host='www.qammado.com', port=80): Max retries exceeded with url: /nonprofit/ifrc/articles/125585 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x10ad788d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))",),)

[rafalf]

image

[heyman]

Hmm, that sounds like a DNS error. What happens if you use the IP-address as host (and set the Host header manually, if needed)?

[rafalf]

DNS is Route 53 and I don't thing that's the issue as I could run 250-500 concurrent users using other tools. I cannot simply use IP address as a host because there's an nginx box in front of it and its serving 3 different sites.

and set the Host header manually, if needed)?
where would I do that with my current setup?

[rafalf]

i might have found something, there is another failure down the log... too many open files

[heyman]

where would I do that with my current setup?

self.client.get("/nonprofit/ifrc/articles/125585", headers={"host":"my-custom-domain"})

[heyman]

i might have found something, there is another failure down the log... too many open files

Ah, that's probably it (http://docs.locust.io/en/latest/installation.html#increasing-maximum-number-of-open-files-limit). You should increase the maximum number of open files. I don't remember exactly how to do it in OSX but it should be easy to google it.

[rafalf]

yes that's the problem ! thanks. Just in case if someone stumble upon the same issue the command is:
ulimit -S -n 1024
there are other ways of setting it up as well
closing it

[rafalf]

@heyman - in fact there were 2 issues and using ip address along with passing the header fixes the original issue reported

ConnectionError(MaxRetryError("HTTPConnectionPool(host='www.qammado.com', port=80): Max retries exceeded with url: /nonprofit/ifrc/articles/125585 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x10ad788d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))",),)

[ghost]

@rafalf typo on it! the correct code is ulimit -S -n 1024

[DataGreed]

I have the same errors on macOS, but when I type ulimit in the terminal t outputs unlimited, so I guess there is something else going on.

Is there any way to fix it? Same config - Route53, arounf 20% of requests failing with

ConnectionError(MaxRetryError("HTTPSConnectionPool(host=\'REDACTED\', port=443): Max retries exceeded with url: /v1/redacted (Caused by NewConnectionError(\'<urllib3.connection.VerifiedHTTPSConnection object at 0x1139ffc18>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known\'))"))'

[DataGreed]

Oops, my bad.

The proper way to check for number of file descriptors on MacOS currently is:

ulimit -n

or you can use this to see all limits:

ulimit -a

Mine was 256, changed it to 4096 and the issue was resolved:

ulimit -n 4096

Note: changes don't persist and last current terminal session only.