How to run multiple workers?

[adeeb1]

Before I switched to good_job, I used delayed_job, and I ran multiple workers in production with the command:

bundle exec bin/delayed_job -n5 start

With good_job, I couldn't find a way to specify the number of workers. Would increasing the number of max threads be an equivalent? My jobs are being processed too slowly, and I need to allocate more resources to processing jobs from the queue.

[bensheldon]

@adeeb1 GoodJob doesn't have a configurable way to run multiple processes (#150); the assumption is that there is a containerized environment that makes it easy to run multiple isolated processes. But assumptions are made to be challenged!

Can you tell me:

• How may jobs are we talking about? (maybe there is a misconfiguration somewhere else)
• How long do the jobs take to run? And are they mostly Ruby or database/IO? (this will help me tell you whether increasing threads will help)
• What's the platform/service where your running your app?

[adeeb1]

Thanks for the quick reply @bensheldon!

• I've enqueued ~30,000 jobs today for a weekly job that runs
• The jobs take about 1-2 seconds to run, and they are emails that are sent out to users. It pulls a few records and sends out an email, so it's Ruby
• The app is containerized on Docker and runs on AWS EC2

Knowing what you said about containerization, I could easily just spin up more workers running good_job to get through these faster. For some reason I assumed the configuration would be the same, so it's good to have clarification on how good_job is expected to be deployed in production. I think I have my solution now: run more EC2 tasks for workers!

[bensheldon]

Awesome! Happy to help!