[improve][broker] Reduce the pressure from the transaction buffer in rolling restarts

[BewareMyPower]

Motivation

During the rolling restarts, the namespace bundle ownerships will change. Assuming there is a producer created on a single topic, and the ownership was transferred to the new broker. Assuming the namespace bundle has N topics and the namespace is tenant/ns,

1. All N topics in the same bundle of that topic will be loaded.
2. For each topic, the managed ledger will be initialized, when the transaction coordinator is enabled, a TopicTransactionBuffer will be created. 2.1 A Pulsar producer will be created on tenant/ns/__transaction_buffer_snapshot concurrently. 2.2 A Pulsar reader will be created on tenant/ns/__transaction_buffer_snapshot concurrently.
3. Once all N readers are created, the owner of the snapshot topic will start dispatching messages to N readers. Each dispatcher will read messages from BookKeeper concurrently and might fail with too many requests error because BK can only have maxPendingReadRequestsPerThread pending read requests (default: 10000).

We have a numTransactionReplayThreadPoolSize config to limit the concurrency of transaction snapshot readers. However, it only limits the read loop. For example, if it's configured with 1, only 1 reader could read messages at the same time. However, N readers will be created concurrently. Each when one of these reader explicitly calls readNext, all N dispatchers at brokers side will dispatch messages to N readers.

The behaviors above brings much CPU pressure on the owner broker, especially for a small cluster with only two brokers.

Modifications

• Synchronize the reader creation, read loop and the following process on its result.
• Delay the snapshot producer creaton to when a message is written via LazySnapshotWriter.