Quorum Queues v4

[kjnilsson]

This is a prioritised list of required and optional features for the next version of quorum queues. Due to quorum queues' Raft internals it makes sense for us to bundle a set of features into a single change and this issue aims to capture and discuss the features for the next iteration.

Required:

• Fair share high/low priorities using https://github.com/kjnilsson/hiloq/)
• Consumer timeouts implemented in the queue (rather than in the channel as currently done).
• SAC consumers honour consumer priorities
• Credited consumer refactoring to meet AMQP requirements.
• Use checkpoints feature. 4.0: use Ra checkpoints in rabbit_fifo for sub-linear time recovery of QQs on boot #10487
• Consumer cancel option that immediately removes consumer and returns all pending messages.
• More compact commands of the most common commands such as enqueue, settle and credit

Optional:

• Consumer return ordering option (to ensure messages in the return queue are ordered in their arrival order rather than their return order as is currently done). Should be optional as could have a negative performance impact.
• Option to return messages without increasing delivery count. This is a feature that AMQP 1.0 require when using the released disposition. This may well require consumer return ordering.
• Pausing consumers
• Message lock renewal (for queue based consumer timeouts).
• rejection annotations: AMQP 1.0 (via the modified outcome) and MQTT 5.0 can supply annotations along with the message outcome / rejection. If we add support for these in the queue type they could potentially be used to route dead letter messages in the future.

Implementation details

Fair/share priorities

Consumer timeouts

Single Active Consumer Priorities

The waiting consumers list should be sorted by consumer priority.
When a consumer with a higher priority than the current active one attaches it would demote the current active one. This could provide a means for users to achieve better consumer app balances and avoid the all consumer on one app issue when using multiple queues with SAC.

Checkpoints

Consumer cancel option that immediately removes consumer and returns all pending messages.

[ansd]

One more feature:

• Respect max-length limit for publications without confirms when overflow behaviour is reject-publish.

[chenchen300]

Priorities (at least two) from my POV are mandatory. We can't migrate our classic queues to quorum without priorities

[kjnilsson]

Please can you elaborate on your use case? Especially why you need priority queues rather than use by multiple queues. Also what kind of prefetch do your priority consumers use?

[michaelklishin]

Priorities (at least two) from my POV are mandatory. We can't migrate our classic queues to quorum without priorities

Yes, you usually can. By using multiple queues instead of just one with priorities.

This belongs to a discussion.