Share buffer pool across all partitions

[merlimat]

Motivation

When a producer is publishing on many partitions, there can be significant memory overhead in maintaining a per-partition pool. Instead, there's not significant perf impact in using a single shared buffer pool.

[wolfstudy]

Cool, the change LGTM+1, just only some concerns whether the use of sync.pool here will cause GC pressure because sync.Pool cannot specify a size, which is only subject to the GC threshold.

[wolfstudy]

In pulsar-perf produce about --batch-max-messages is abbreviated and the default value is as follows, can we consider keeping the two consistent?

   -bm, --batch-max-messages
       Maximum number of messages per batch
       Default: 1000

[merlimat]

This are 2 different settings. One is the number of messages in 1 batch and the other is the max size of the batch

[merlimat]

just only some concerns whether the use of sync.pool here will cause GC pressure because sync.Pool cannot specify a size, which is only subject to the GC threshold.

Using the pool will not causing GC pressure itself: it's actually there to avoid the GC pressure.

The pool having no max size is not a big problem. The max amount of memory is still determined by the "pending" messages whose payloads are buffered until they get acknowledged by the broker.

This change is to use a single pool to avoid that each pool will have a few buffers that cannot be immediately reused. After this change, the memory usage is the same as without the pooling.

[wolfstudy]

LGTM +1