Clean up the support for split queues #78274
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#76410 introduces a way to deliver
messages to a set of split queues rather than relying on a single queue for
post process.
This was used to address an incident as we were saturating Rabbit resources
for a single queue (which is single threaded). Splitting messages across multiple
queues solves that problem.
This PR introduce a configurable support for split queues for scenarios where we pass
the queue name to
apply_async
.There is another PR to deal with tasks that define the queue in the task definition:
#76494
It introduces a router class SplitQueueRouter that map a queue to a list of split
queues picked in a round robin way.
This router is used by the client code that schedules a task with apply_async.
The configuration is held in CELERY_SPLIT_QUEUE_ROUTES and all the
split queues have to be declared as any other queue.
Rollout procedure to remove the hack from #76410
Merge this PR, now all the infra is in place. What was added by feat(postprocess):
INC-855 allow to split post_process_* queues #76410 is still working because the two options added to manage the router have a default value https://github.com/getsentry/sentry/pull/76494/files#diff-c4bf5bc2eae9efe5e948b5deda98c34fafd95674ae376aaf790c21a039e59f91R2699-R2706. SO the router falls back to the config added by Michal.
Update the config in the production regions to add the three queues to CELERY_SPLIT_QUEUE_ROUTES.
Switch the celery_split_queue_legacy_mode option to stop using the legacy config
Remove the legacy config and use the router only.