[fix][broker] Ensure that PulsarService is ready for serving incoming requests

[lhotari]

Fixes #22975

Motivation

Pulsar broker will start serving requests while the broker is starting. This can cause issues and bugs which are hard to reproduce. Serving requests should be delayed until the broker is ready.
The brokerId bug #22975 is just one example of problems that this PR will address.

Modifications

• start Pulsar binary protocol connections with autoread set to false so that request handling can be postponed
• enable autoread after PulsarService has been started
  • add solution to PulsarService for running a function when the service has been started
• add a filter to REST implementation which blocks until PulsarService is started
  • add solution to PulsarService to wait until the service has been started

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[dao-jun]

Is this possible for us to start WebService/WebSocketService/BrokerService/ProtocolHander after the other Pulsar Services start?

[lhotari]

Is this possible for us to start WebService/WebSocketService/BrokerService/ProtocolHander after the other Pulsar Services start?

Yes, it makes sense to cover all cases in a way so that requests aren't handled before PulsarService has been started.

[lhotari]

Is this possible for us to start WebService/WebSocketService/BrokerService/ProtocolHander after the other Pulsar Services start?

Yes, it makes sense to cover all cases in a way so that requests aren't handled before PulsarService has been started.

@dao-jun

I renamed the concept to mean that the broker is ready to serve request instead of being fully started. I believe that this covers what is really needed and the original intention of this PR.

I checked the current solution and this is sufficiently covered already for the Pulsar broker with various configurations. There's no need to separately handle websockets since websocket servlets get added to the same Jetty server which already contains the filter to wait until PulsarService is ready for incoming requests.

ProtocolHandlers don't need any special handling since they get started as the last step in PulsarService.

[lhotari]

@heesung-sn I pushed an attempt to make this solution work with ExtensibleLoadManagerImpl.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 77.38095% with 19 lines in your changes missing coverage. Please review.

Project coverage is 73.39%. Comparing base (bbc6224) to head (fc593e7).
Report is 424 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22977      +/-   ##
============================================
- Coverage     73.57%   73.39%   -0.18%     
- Complexity    32624    32915     +291     
============================================
  Files          1877     1903      +26     
  Lines        139502   142711    +3209     
  Branches      15299    15571     +272     
============================================
+ Hits         102638   104749    +2111     
- Misses        28908    29941    +1033     
- Partials       7956     8021      +65     

Flag	Coverage Δ
inttests	27.73% <72.61%> (+3.14%)	⬆️
systests	24.73% <27.38%> (+0.40%)	⬆️
unittests	72.43% <77.38%> (-0.42%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...n/java/org/apache/pulsar/broker/PulsarService.java	83.10% <100.00%> (+0.73%)	⬆️
...ache/pulsar/broker/namespace/NamespaceService.java	73.21% <100.00%> (+0.97%)	⬆️
...ulsar/broker/service/PulsarChannelInitializer.java	95.08% <100.00%> (+0.08%)	⬆️
...va/org/apache/pulsar/broker/service/ServerCnx.java	72.18% <100.00%> (+0.03%)	⬆️
.../java/org/apache/pulsar/broker/web/WebService.java	90.78% <62.50%> (-3.36%)	⬇️
...dbalance/extensions/ExtensibleLoadManagerImpl.java	80.52% <74.50%> (+0.44%)	⬆️

... and 464 files with indirect coverage changes