Make WORKER_INTERVAL_MS a configurable parameter

[ThomasProctor]

This is something I've found useful for throttling my scrapes to avoid being banned. It just requires 3 changed lines of code.

[leonardiwagner]

throttling my scrapes to avoid being banned

Could you explain how much of time prevents you from being banned? Thank you!

[ThomasProctor]

Right now, I know that the default - 1000 ms - has gotten me banned after scraping around 200 profiles. I've been able to safely scrape with a value much higher than that - 100000 ms or so.

I've been basing my experience off numbers I've gotten from PhantomBuster, which seem to be incredibly conservative. I'm still working on finding exactly what level I can get away with. I'm assuming it depends on other behavior as well too, so it will take some experimentation.