[SOLVED] last_error backtrace showing jobs being run from different release / version of codebase

[brunzino]

As the title says, some of my jobs are throwing errors, and the backtrace last_error indicates that the job is being run from a different version of the codebase.

For example, while I was testing my failure hook, I had one of my jobs raise a "Test Exception" and set the max_attempts = 3. Since then I have removed that code, yet the error still gets thrown, and from the backtrace I see that the code that's getting run is from release 87 (see below) where the current version is 91 (code resides in /var/www/staging/releases/87/app/mailers...). Also, the max_attempts value of 3 may or may not be obeyed.

Test Exception
/var/www/staging/releases/87/app/mailers/applog_mailer.rb:9:in `send_open_applications_log'
....

Furthermore, I'll see other jobs (from same deployment version) have a last_error backtrace from different release , not just 87.

I'm using the mina gem to deploy.

[albus522]

You deploy script is failing, leaving workers running old code. I am afraid there is nothing we can do about that.

[brunzino]

[SOLUTION]
If it's a problem with the deploy, I should be able get it to work correctly by manually stopping and restarting the delayed_job daemon, right? I've done this and the problem persists. The jobs throw errors from different versions of the code for different attempts of the same job. I.e. one job will get run on release 87 codebase, fail, then try to run again on release 89 codebase.

It's as if there are still different daemons running that are not getting shut off from the deploy. RAILS_ENV=production bin/delayed_job status says there are no jobs running, but this yields different results in different release directories.

When running ps -eLf | grep \delayed_job I see there are several daemons running.
Using pidof your_delayed_job_instance_name and kill -9 12345 23456 34567 for the output helped kill all daemons.

Killing the other instances of the delayed_job daemon did indeed solve the problem. Each instance of the daemon preserves its configs and apparently codebase to execute.