[fix] [broker] Internal reader of __change_events can not started after metadata store session rebuilt

[poorbarcode]

Motivation & Modifications

1. Fix the issue below

• settings: enable systemTopic & topicLevelPolicies.
• Own topics
  • broker-2 loads a topic-a up.
  • broker-2 loads system topic({ns}/__change_events) up.
• The ZK session expires.
• Bundle shedding
  • topic-a was transferred to broker-0
  • the system topic({ns}/__change_events) was transferred to broker-0
• (Highlight) Issue occurs on broker-2
  • The internal reader for {ns}/__change_events lookup the topic's owner
  • broker-2 responds broker-2 due to its OwnershipCache contains the old value.
  • The internal reader for {ns}/__change_events can not be created successfully, all topics under this namespace can not work.
  • The consumers/producers who call lookup to broker-2 will get a wrong value, and keep getting a "Please redo the lookup" error.

2. Add a test to confirm the topic will be unloaded after the metadata store session is rebuilt.

• Broker load a topic up.
• The ZK session expires.
• The ZK session rebuild.
• Verify: The topics that the broker owned will be unloaded

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.45%. Comparing base (bbc6224) to head (e439b97).
Report is 475 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23018      +/-   ##
============================================
- Coverage     73.57%   73.45%   -0.13%     
- Complexity    32624    33216     +592     
============================================
  Files          1877     1917      +40     
  Lines        139502   144072    +4570     
  Branches      15299    15743     +444     
============================================
+ Hits         102638   105824    +3186     
- Misses        28908    30131    +1223     
- Partials       7956     8117     +161     

Flag	Coverage Δ
inttests	27.73% <80.00%> (+3.14%)	⬆️
systests	24.72% <0.00%> (+0.40%)	⬆️
unittests	72.52% <100.00%> (-0.33%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...dbalance/extensions/ExtensibleLoadManagerImpl.java	79.45% <ø> (-0.64%)	⬇️
...roker/loadbalance/impl/ModularLoadManagerImpl.java	84.98% <ø> (+<0.01%)	⬆️
...he/pulsar/metadata/impl/AbstractMetadataStore.java	84.94% <100.00%> (+0.38%)	⬆️

... and 509 files with indirect coverage changes

[heesung-sn]

Can we cover the new load balancer behavior too?

[poorbarcode]

@heesung-sn

Can we cover the new load balancer behavior too?

Sure, I will push two following PRs to do this:

• Enable system topic and topic-level policies. So far, this test can not pass when enabled system topic, I am checking why it does not work, and I will push a fix if there is an issue. Added into the current PR.
• Enable ExtensibleLoadManager, since it depends on the system topic, this test will be added after the above one.