Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow GoodJob global configuration accessors to also be set via Rails config hash #460

Merged
merged 1 commit into from
Nov 29, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 46 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
1. Configure the ActiveJob adapter:

```ruby
# config/application.rb
# config/application.rb or config/environments/{RAILS_ENV}.rb
config.active_job.queue_adapter = :good_job
```

Expand Down Expand Up @@ -212,41 +212,48 @@ to delete old records and preserve space in your database.
### Configuration options
To use GoodJob, you can set `config.active_job.queue_adapter` to a `:good_job`.
ActiveJob configuration depends on where the code is placed:
- `config.active_job.queue_adapter = :good_job` within `config/application.rb` or `config/environments/*.rb`.
- `ActiveJob::Base.queue_adapter = :good_job` within an initializer (e.g. `config/initializers/active_job.rb`).
Additional configuration can be provided via `config.good_job.OPTION = ...`.
GoodJob configuration can be placed within Rails `config` directory for all environments (`config/application.rb`), within a particular environment (e.g. `config/environments/development.rb`), or within an initializer (e.g. `config/initializers/good_job.rb`).
Configuration examples:
```ruby
# config/application.rb
config.active_job.queue_adapter = :good_job
# Configure options individually...
config.good_job.execution_mode = :async
config.good_job.max_threads = 5
config.good_job.poll_interval = 30 # seconds
config.good_job.shutdown_timeout = 25 # seconds
config.good_job.enable_cron = true
config.good_job.cron = { example: { cron: '0 * * * *', class: 'ExampleJob' } }
config.good_job.queues = '*'
# ...or all at once.
config.good_job = {
execution_mode: :async,
max_threads: 5,
poll_interval: 30,
shutdown_timeout: 25,
enable_cron: true,
cron: {
example: {
cron: '0 * * * *',
class: 'ExampleJob'
Rails.application.configure do
# Configure options individually...
config.good_job.preserve_job_records = true
config.good_job.retry_on_unhandled_error = false
config.good_job.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
config.good_job.execution_mode = :async
config.good_job.max_threads = 5
config.good_job.poll_interval = 30 # seconds
config.good_job.shutdown_timeout = 25 # seconds
config.good_job.enable_cron = true
config.good_job.cron = { example: { cron: '0 * * * *', class: 'ExampleJob' } }
config.good_job.queues = '*'
# ...or all at once.
config.good_job = {
preserve_job_records: true,
retry_on_unhandled_error: false,
on_thread_error: -> (exception) { Raven.capture_exception(exception) },
execution_mode: :async,
max_threads: 5,
poll_interval: 30,
shutdown_timeout: 25,
enable_cron: true,
cron: {
example: {
cron: '0 * * * *',
class: 'ExampleJob'
},
},
},
queues: '*',
}
queues: '*',
}
end
```
Available configuration options are:
Expand All @@ -263,6 +270,14 @@ Available configuration options are:
- `shutdown_timeout` (float) number of seconds to wait for jobs to finish when shutting down before stopping the thread. Defaults to forever: `-1`. You can also set this with the environment variable `GOOD_JOB_SHUTDOWN_TIMEOUT`.
- `enable_cron` (boolean) whether to run cron process. Defaults to `false`. You can also set this with the environment variable `GOOD_JOB_ENABLE_CRON`.
- `cron` (hash) cron configuration. Defaults to `{}`. You can also set this as a JSON string with the environment variable `GOOD_JOB_CRON`
- `logger` ([Rails Logger](https://api.rubyonrails.org/classes/ActiveSupport/Logger.html)) lets you set a custom logger for GoodJob. It should be an instance of a Rails `Logger` (Default: `Rails.logger`).
- `preserve_job_records` (boolean) keeps job records in your database even after jobs are completed. (Default: `false`)
- `retry_on_unhandled_error` (boolean) causes jobs to be re-queued and retried if they raise an instance of `StandardError`. Instances of `Exception`, like SIGINT, will *always* be retried, regardless of this attribute’s value. (Default: `true`)
- `on_thread_error` (proc, lambda, or callable) will be called when an Exception. It can be useful for logging errors to bug tracking services, like Sentry or Airbrake. Example:
```ruby
config.good_job.on_thread_error = -> (exception) { Raven.capture_exception(exception) }
```
By default, GoodJob configures the following execution modes per environment:
Expand All @@ -283,7 +298,7 @@ config.good_job.execution_mode = :external
### Global options
Good Job’s general behavior can also be configured via several attributes directly on the `GoodJob` module:
Good Job’s general behavior can also be configured via attributes directly on the `GoodJob` module:
- **`GoodJob.active_record_parent_class`** (string) The ActiveRecord parent class inherited by GoodJob's ActiveRecord model `GoodJob::Job` (defaults to `"ActiveRecord::Base"`). Configure this when using [multiple databases with ActiveRecord](https://guides.rubyonrails.org/active_record_multiple_databases.html) or when other custom configuration is necessary for the ActiveRecord model to connect to the Postgres database. _The value must be a String to avoid premature initialization of ActiveRecord._
- **`GoodJob.logger`** ([Rails Logger](https://api.rubyonrails.org/classes/ActiveSupport/Logger.html)) lets you set a custom logger for GoodJob. It should be an instance of a Rails `Logger`.
Expand Down
11 changes: 10 additions & 1 deletion lib/good_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
#
# +GoodJob+ is the top-level namespace and exposes configuration attributes.
module GoodJob
DEFAULT_LOGGER = ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new($stdout))

# @!attribute [rw] active_record_parent_class
# @!scope class
# The ActiveRecord parent class inherited by +GoodJob::Execution+ (default: +ActiveRecord::Base+).
Expand All @@ -34,7 +36,7 @@ module GoodJob
# @return [Logger, nil]
# @example Output GoodJob logs to a file:
# GoodJob.logger = ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new("log/my_logs.log"))
mattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new($stdout))
mattr_accessor :logger, default: DEFAULT_LOGGER

# @!attribute [rw] preserve_job_records
# @!scope class
Expand Down Expand Up @@ -66,6 +68,13 @@ module GoodJob
# @return [Proc, nil]
mattr_accessor :on_thread_error, default: nil

# Called with exception when a GoodJob thread raises an exception
# @param exception [Exception] Exception that was raised
# @return [void]
def self._on_thread_error(exception)
on_thread_error.call(exception) if on_thread_error.respond_to?(:call)
end

# Stop executing jobs.
# GoodJob does its work in pools of background threads.
# When forking processes you should shut down these background threads before forking, and restart them after forking.
Expand Down
2 changes: 1 addition & 1 deletion lib/good_job/cron_manager.rb
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ class CronManager
def self.task_observer(time, output, thread_error) # rubocop:disable Lint/UnusedMethodArgument
return if thread_error.is_a? Concurrent::CancelledOperationError

GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
GoodJob._on_thread_error(thread_error) if thread_error
end

# Execution configuration to be scheduled
Expand Down
2 changes: 1 addition & 1 deletion lib/good_job/notifier.rb
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ def listen_observer(_time, _result, thread_error)
return if thread_error.is_a? AdapterCannotListenError

if thread_error
GoodJob.on_thread_error.call(thread_error) if GoodJob.on_thread_error.respond_to?(:call)
GoodJob._on_thread_error(thread_error)
ActiveSupport::Notifications.instrument("notifier_notify_error.good_job", { error: thread_error })

connection_error = CONNECTION_ERRORS.any? do |error_string|
Expand Down
2 changes: 1 addition & 1 deletion lib/good_job/poller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ def restart(timeout: -1)
# @param thread_error [Exception, nil]
# @return [void]
def timer_observer(time, executed_task, thread_error)
GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
GoodJob._on_thread_error(thread_error) if thread_error
ActiveSupport::Notifications.instrument("finished_timer_task", { result: executed_task, error: thread_error, time: time })
end

Expand Down
2 changes: 1 addition & 1 deletion lib/good_job/probe_server.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ class ProbeServer
def self.task_observer(time, output, thread_error) # rubocop:disable Lint/UnusedMethodArgument
return if thread_error.is_a? Concurrent::CancelledOperationError

GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
GoodJob._on_thread_error(thread_error) if thread_error
end

def initialize(port:)
Expand Down
17 changes: 14 additions & 3 deletions lib/good_job/railtie.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ class Railtie < ::Rails::Railtie

initializer "good_job.logger" do |_app|
ActiveSupport.on_load(:good_job) do
self.logger = ::Rails.logger
self.logger = ::Rails.logger if GoodJob.logger == GoodJob::DEFAULT_LOGGER
end
GoodJob::LogSubscriber.attach_to :good_job
end
Expand All @@ -22,8 +22,19 @@ class Railtie < ::Rails::Railtie
end
end

config.after_initialize do
GoodJob::Adapter.instances.each(&:start_async)
initializer 'good_job.rails_config' do
config.after_initialize do
GoodJob.logger = Rails.application.config.good_job.logger unless Rails.application.config.good_job.logger.nil?
GoodJob.on_thread_error = Rails.application.config.good_job.on_thread_error unless Rails.application.config.good_job.on_thread_error.nil?
GoodJob.preserve_job_records = Rails.application.config.good_job.preserve_job_records unless Rails.application.config.good_job.preserve_job_records.nil?
GoodJob.retry_on_unhandled_error = Rails.application.config.good_job.retry_on_unhandled_error unless Rails.application.config.good_job.retry_on_unhandled_error.nil?
end
end

initializer "good_job.start_async" do
config.after_initialize do
GoodJob::Adapter.instances.each(&:start_async)
end
end
end
end
4 changes: 2 additions & 2 deletions lib/good_job/scheduler.rb
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def create_thread(state = nil)
# @return [void]
def task_observer(time, output, thread_error)
error = thread_error || (output.is_a?(GoodJob::ExecutionResult) ? output.unhandled_error : nil)
GoodJob.on_thread_error.call(error) if error && GoodJob.on_thread_error.respond_to?(:call)
GoodJob._on_thread_error(error) if error

instrument("finished_job_task", { result: output, error: thread_error, time: time })
create_task if output
Expand Down Expand Up @@ -206,7 +206,7 @@ def warm_cache
end

observer = lambda do |_time, _output, thread_error|
GoodJob.on_thread_error.call(thread_error) if thread_error && GoodJob.on_thread_error.respond_to?(:call)
GoodJob._on_thread_error(thread_error) if thread_error
create_task # If cache-warming exhausts the threads, ensure there isn't an executable task remaining
end
future.add_observer(observer, :call)
Expand Down
8 changes: 0 additions & 8 deletions spec/test_app/config/application.rb
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,6 @@ class Application < Rails::Application

# config.middleware.insert_before Rack::Sendfile, ActionDispatch::DebugLocks
config.log_level = :debug

config.good_job.cron = {
example: {
cron: '*/5 * * * * *', # every 5 seconds
class: 'ExampleJob',
description: "Enqueue ExampleJob every 5 seconds",
},
}
end
end

2 changes: 0 additions & 2 deletions spec/test_app/config/environments/production.rb
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,6 @@
# Do not dump schema after migrations.
config.active_record.dump_schema_after_migration = false

config.active_job.queue_adapter = :good_job

# Inserts middleware to perform automatic connection switching.
# The `database_selector` hash is used to pass options to the DatabaseSelector
# middleware. The `delay` is used to determine how long to wait after a write
Expand Down
18 changes: 15 additions & 3 deletions spec/test_app/config/initializers/good_job.rb
Original file line number Diff line number Diff line change
@@ -1,11 +1,22 @@
Rails.application.configure do
config.good_job.cron = {
example: {
cron: '*/5 * * * * *', # every 5 seconds
class: 'ExampleJob',
description: "Enqueue ExampleJob every 5 seconds",
},
}
end

case Rails.env
when 'development'
ActiveJob::Base.queue_adapter = :good_job

GoodJob.retry_on_unhandled_error = false
GoodJob.preserve_job_records = true
GoodJob.on_thread_error = -> (error) { Rails.logger.warn(error) }

Rails.application.configure do
config.active_job.queue_adapter = :good_job
config.good_job.enable_cron = ActiveModel::Type::Boolean.new.cast(ENV.fetch('GOOD_JOB_ENABLE_CRON', true))
config.good_job.cron = {
frequent_example: {
Expand All @@ -28,11 +39,12 @@
when 'test'
# test
when 'demo'
ActiveJob::Base.queue_adapter = :good_job

GoodJob.preserve_job_records = true
GoodJob.retry_on_unhandled_error = false

Rails.application.configure do
config.active_job.queue_adapter = :good_job
config.good_job.execution_mode = :async
config.good_job.poll_interval = 30

Expand Down Expand Up @@ -67,5 +79,5 @@
}
end
when 'production'
# production
ActiveJob::Base.queue_adapter = :good_job
end