Queueing a job as part of an ActiveRecord transaction

[orangethunder]

Due to queue_classic using its own connection, jobs are queued immediately, even if you're inside a transaction. One of the benefits of Delayed Job is that you could queue something and only have it run once the transaction around it has been committed. I'm beginning to think that queue_classic needs tighter AR integration on the producer side - completely optional of course. It would then just use the current AR connection for queueing jobs.

Workers could remain as they are, using the QC_DATABASE_URL to construct a connection, or could also have an AR integration done. The delayed_job_active_record gem contains some tried and tested code for getting this to work, which I might use for ideas. If I coded this up (starting with the producer side), do you think you could incorporate it?

Deferring the job won't work if the NOTIFY is triggered before the containing transaction is actually committed. Can you comment on whether this will be the case? I don't have enough knowledge of how that works.

Regards
Dave

[orangethunder]

I've now tested this, and the NOTIFY is triggered only after the commit, as expected.

While coding this up, it dawned on me that it's going to depend on the use case whether you want your job to run immediately or only if / when the containing transaction commits. So the interface needs to allow you to express your wish. I'm thinking maybe an enqueue_transactional method. Or conversely, make that behaviour the default for enqueue and add an enqueue_oob method (out of bounds) if it should disregard any transactions, i.e. use the built-in QC connection.

I think enqueue_transactional would be best for backwards compatibility. I intend to make it raise an error if ActiveRecord is not loaded.

I'm not going to be doing anything on the worker side soon. I think this should be seen as completely separate.

Thoughts?

[ukd1]

@orangethunder we use it on a single connection to PG, which does it as you want I think. It's currently possible with no code changes to QC, just change your initializer. Ours looks like this:

require 'queue_classic'
QC::Conn.connection = ActiveRecord::Base.connection.raw_connection

class QC::Worker
  def setup_child
    log(:at => "setup_child", :message => "Reconnecting to DB")

    ActiveRecord::Base.establish_connection
    QC::Conn.connection = ActiveRecord::Base.connection.raw_connection
  end
end

[jipiboily]

Closing as there was an answer ~ 6 months ago and no news here.

[senny]

This is related to the connection sharing with Active Record. See #232

[jipiboily]

@senny are you suggesting we change something?

[senny]

@jipiboily yes, from my point of view it should be the default behavior when you use QC with Rails. AR and QC should share the same transaction scope.

[jipiboily]

I have a WIP branch which will make use of AR connection by default: #232

Should be enough as far as I understand.