High GC pause cases high publish latency

[shibd]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

We have observed that using Individual ack frequently results in young generation GC and relatively high GC pauses.

• Resource: k8s cluster
  • broker(1.5CPU)(3CPU 8Gi)
  • bookie(2.5CPU 8Gi) 250Gi gp3 ssd
  • zookeeper(1CPU 4Gi)
• Version: 3.2.2

Performance cmd:

bin/pulsar-perf consume test-c 1200 -n 10 -st Shared -t 10
bin/pulsar-perf produce test-p -s 100 -time 1200 -n 10 -t 10 -ml 256M

Solution

The issue should be caused by the large long[] allocated by bitmap. The allocation invoking chain.

Maybe this is a normal phenomenon and can be solved by increasing the CPU, I would like to share it here to see if there are any similar problems and suggestions in the community.

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

Btw. The metric might be wrong. It's not necessarily pause time, but GC time that happens in the background. See #18891

[mattisonchao]

I think so. But we might need to improve here to avoid allocating a very large long[] to give GC pressure. :)

[lhotari]

I think so. But we might need to improve here to avoid allocating a very large long[] to give GC pressure. :)

One alternative would be to switch to use https://github.com/lemire/RoaringBitmap/ .