have a limit on number of consumer groups/offset

[jason-da-redpanda]

Who is this for and what problem do they have today?

We have seen numerous issues where huge numbers of consumer groups can cause issues.

In a number of these cases the customer/admin has been unaware of the large amount of consumer groups..
Typically it will be an app .. using group-id=${random.uuid}.... and the groups are in UUID format.

We have the ability to clean-up consumer group offsets .. via
group_offset_retention_sec
(defaults to 7 days )

but this is to provided a parameter /throttle to prevent the rapid growth of consumer groups/offsets

What are the success criteria?

Have a server config for maximum number of consumer group/offsets

Why is solving this problem impactful?

Huge numbers of consumer groups/offsets...causes issues such as

• Console cannot handle huge numbers of consumer groups /offsets is it tries to fetch all the offsets it timesout/hangs
• knock on memory issues ..such as <unknown.. > and known ones such as
• Oversized allocation: 413696 bytes in seastar::prometheus::metric_aggregate_by_labels::add #16949
• unknowns.. (what numbers of CG/offsets do we test?)

Additional notes

JIRA Link: CORE-7050