Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up the support for split queues #78274

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

fpacifici
Copy link
Contributor

#76410 introduces a way to deliver
messages to a set of split queues rather than relying on a single queue for
post process.

This was used to address an incident as we were saturating Rabbit resources
for a single queue (which is single threaded). Splitting messages across multiple
queues solves that problem.

This PR introduce a configurable support for split queues for scenarios where we pass
the queue name to apply_async.
There is another PR to deal with tasks that define the queue in the task definition:
#76494

It introduces a router class SplitQueueRouter that map a queue to a list of split
queues picked in a round robin way.
This router is used by the client code that schedules a task with apply_async.
The configuration is held in CELERY_SPLIT_QUEUE_ROUTES and all the
split queues have to be declared as any other queue.

Rollout procedure to remove the hack from #76410

Merge this PR, now all the infra is in place. What was added by feat(postprocess):
INC-855 allow to split post_process_* queues #76410 is still working because the two options added to manage the router have a default value https://github.com/getsentry/sentry/pull/76494/files#diff-c4bf5bc2eae9efe5e948b5deda98c34fafd95674ae376aaf790c21a039e59f91R2699-R2706. SO the router falls back to the config added by Michal.
Update the config in the production regions to add the three queues to CELERY_SPLIT_QUEUE_ROUTES.
Switch the celery_split_queue_legacy_mode option to stop using the legacy config
Remove the legacy config and use the router only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant