reserve method does not respect job's max_run_time parameter

[kevinrobinson]

The documentation for Delayed::Job suggests that you can tune the max_run_time parameter globally or in each Job class.

To set a per-job max run time that overrides the Delayed::Worker.max_run_time you can define a max_run_time method on the job

This doesn't seem to work for me testing this out with the ActiveRecord backend, and it looks like it's because the reserve code in this repo uses the Worker.max_run_time value here when querying for jobs that could be run. I'm not sure how to resolve this with a small change, since it seems like you'd need to look at the handler field for each one and deserialize it to get the max_run_time value in order to have enough information to do this correctly.

One guess as to what's happening is that max_run_time is being used as two things: 1) the timeout value for how long a particular job can run before being interrupted (which isn't backend-specific) and 2) it's also being used as the value that determines how long to wait to free up an abandoned job (eg, where the worker process has died). The second one is backend-specific, and so my read is that this backend doesn't support tuning that per-job as the overall Delayed::Job docs suggest.

One path forward might be to query the database more permissively, convert those into Job objects and then consider max_run_time for each job. I suspect that this would be an expensive change and need some thought (since the code nearby has a bunch of optimizations). Alternately, the value for max_run_time could be written into the delayed_jobs table for faster scanning at the database level (with the tradeoff that the value might be stale if new code updates that value over time).