[fix][broker] Avoid infinite bundle unloading #20822

Demogorgon314 · 2023-07-17T10:03:34Z

Motivation

The bundle ownership assignment logic doesn't know the previous unloaded broker when unloading happens. It might assign the bundle to the same broker. In this case, it might cause an infinite bundle unloading loop.

To resolve this issue, we can check the destination broker when doing the doLoadShedding stage. When the destination broker is the same as the current owner broker, we can skip this unload, and if it is different, we set the new owner in this stage.

Modifications

Transfer the bundle to the new owner when needed to unload.

Verifying this change

See the ModularLoadManagerImplTest#testLoadShedding unit test.

Documentation

doc
doc-required
doc-not-needed
doc-complete

codelipenghui

The PR title should be a fix for the infinite bundle unloading.

If I understand correctly, a breaking change will be introduced to the behavior of loadBalancerDistributeBundlesEvenlyEnabled because the current solution just filters out the previous owner, then the bundle can be moved to other brokers. I think the correct solution should be to skip the bundle unloading if the next owner is the same one as the current owner.

IMO, the main point is we don't care about the next owner broker when performing the bundle unloading. But the bundle unloading and the owner selection are based on the same load data. So is it worth checking the next owner at the unloading stage? @Demogorgon314 @heesung-sn

codelipenghui

BTW, I noticed no one put data to LoadData.brokerData and LoadData.bundleData except for the test. Is it still useful? or maybe I missed something.

Demogorgon314 · 2023-07-19T01:48:00Z

BTW, I noticed no one put data to LoadData.brokerData and LoadData.bundleData except for the test. Is it still useful? or maybe I missed something.

@codelipenghui Yes, it is still in use. The ModularLoadManagerImpl#updateAllBrokerData method puts data to LoadData.brokerData, and the ModularLoadManagerImpl#updateBundleData method puts data to LoadData.bundleData

Demogorgon314 · 2023-07-19T07:39:32Z

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

+                        pulsar.getAdminClient().namespaces()
+                                .unloadNamespaceBundle(namespaceName, bundleRange, destBroker.get());


The destinationBroker parameter is added in this PR: #18663

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

codelipenghui · 2023-07-19T07:58:38Z

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

                    try {
-                        pulsar.getAdminClient().namespaces().unloadNamespaceBundle(namespaceName, bundleRange);
+                        pulsar.getAdminClient().namespaces()
+                                .unloadNamespaceBundle(namespaceName, bundleRange, destBroker.get());


Just a reminder, The destBroker is only available after 2.11.0. We are not able to cherry-pick to branch-2.10

Or maybe we don't need to ensure the bundle will actually go to the dest broker? We just want to avoid the infinite loop as possible

When we have many brokers, it still has a chance to select the same broker if we don't pass the destBroker parameter. I wonder if this is acceptable.

But it will only sometimes go to the same broker, not always. In 2.10, we don't have this API to support a strict owner assignment at the loading stage.

Ok, I see. After this PR is merged, I will push a PR for branch 2.10.

For 2.10 we don't have a good way for that. What will happen if the same broker is still selected after unloading? Will it retry?

@BewareMyPower It will be selected to unload again. And hope next time it will be skipped.

Ok, I see. After this PR is merged, I will push a PR for branch 2.10.

To be clear, how are you going to push this fix to 2.10? According to @codelipenghui, the destBroker option in the unload admin is only available since 2.11.

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

…ce/impl/ModularLoadManagerImpl.java Co-authored-by: Yunze Xu <[email protected]>

codecov-commenter · 2023-07-21T10:47:21Z

Codecov Report

Merging #20822 (531e964) into master (57fbee4) will decrease coverage by 0.34%.
The diff coverage is 77.33%.

@@             Coverage Diff              @@
##             master   #20822      +/-   ##
============================================
- Coverage     72.97%   72.64%   -0.34%     
- Complexity    32157    32227      +70     
============================================
  Files          1868     1856      -12     
  Lines        139164   139139      -25     
  Branches      15314    15329      +15     
============================================
- Hits         101555   101077     -478     
- Misses        29562    29999     +437     
- Partials       8047     8063      +16

Flag	Coverage Δ
inttests	`24.24% <48.00%> (?)`
systests	`?`
unittests	`72.40% <76.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...roker/loadbalance/impl/ModularLoadManagerImpl.java	`82.75% <77.33%> (+0.27%)`	⬆️

... and 168 files with indirect coverage changes

codelipenghui · 2023-07-25T14:34:23Z

@Demogorgon314 Could you please help cherry-pick the PR into release branches?

(cherry picked from commit 3f63768)

heesung-sn · 2023-07-27T20:29:53Z

Though this is another great improvement in ModularLoadManager, this is a behavior change too.
For this behavior change,

Do we need to update any doc?
Do we need to put this logic under any config, as this is a behavior change?

It is great that this logic can block unloading bundles to the same broker.
e.g) Unloading to the same broker
A->A

However, could infinite unloading still happen if unloading bounces to multiple brokers?
e.g) Unloading bounces between two brokers
A->B->A->B

Demogorgon314 · 2023-07-28T00:52:12Z

@heesung-sn

Do we need to update any doc?

Yes, It's good to add a doc to notify the users.

Do we need to put this logic under any config, as this is a behavior change?

I think we can leave config as is, since unloading to the same broker is not make sense.

However, could infinite unloading still happen if unloading bounces to multiple brokers?

Yes, this can happen, not sure if we have a good way to prevent it.

…20877)

shibd · 2023-10-21T09:22:59Z

cherry-picked branch-2.11 by #20878

hanmz · 2024-02-21T11:39:43Z

Why not call selectBrokerForAssignment directly, but add a new selectBroker method?The only difference I see is that it is not added to preallocated . What is the impact if added to preallocated ?

BewareMyPower · 2024-08-23T17:32:23Z

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

+        final ConcurrentOpenHashMap<String, ConcurrentOpenHashSet<String>> namespaceToBundleRange =
+                brokerToNamespaceToBundleRange
+                        .computeIfAbsent(broker,
+                                k -> ConcurrentOpenHashMap.<String,
+                                        ConcurrentOpenHashSet<String>>newBuilder()
+                                        .build());
+        synchronized (namespaceToBundleRange) {
+            namespaceToBundleRange.computeIfAbsent(namespaceName,
+                    k -> ConcurrentOpenHashSet.<String>newBuilder().build())
+                    .add(bundleRange);
+        }


The synchronized block here seems meaningless?

Yes... not sure why used synchronized block here before.

Demogorgon314 added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages area/broker labels Jul 17, 2023

Demogorgon314 requested a review from codelipenghui July 17, 2023 10:03

Demogorgon314 self-assigned this Jul 17, 2023

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 17, 2023

Demogorgon314 force-pushed the Demogorgon314/Avoid-assign-bundles-to-previous-unloaded-broker branch from f953f89 to bcbc1af Compare July 17, 2023 10:04

Demogorgon314 marked this pull request as ready for review July 18, 2023 02:04

codelipenghui requested a review from hangc0276 July 18, 2023 10:31

codelipenghui reviewed Jul 19, 2023

View reviewed changes

Demogorgon314 changed the title ~~[improve][broker] Avoid assigning bundle to the previous unloaded broker~~ [fix][broker] Avoid infinite bundle unloading Jul 19, 2023

Demogorgon314 marked this pull request as draft July 19, 2023 04:49

Transfer the bundle to selected broker in doLoadShedding stage

bb2ed0d

Demogorgon314 force-pushed the Demogorgon314/Avoid-assign-bundles-to-previous-unloaded-broker branch from ef68011 to bb2ed0d Compare July 19, 2023 07:17

streamnativebot mentioned this pull request Jul 19, 2023

PR-20822: [fix][broker] Avoid infinite bundle unloading streamnative/pulsar-archived#5838

Open

4 tasks

Demogorgon314 marked this pull request as ready for review July 19, 2023 07:26

streamnativebot mentioned this pull request Jul 19, 2023

PR-20822: [fix][broker] Avoid infinite bundle unloading streamnative/pulsar-archived#5839

Open

4 tasks

Demogorgon314 commented Jul 19, 2023

View reviewed changes

streamnativebot mentioned this pull request Jul 19, 2023

PR-20822: [fix][broker] Avoid infinite bundle unloading streamnative/pulsar-archived#5841

Closed

4 tasks

codelipenghui reviewed Jul 19, 2023

View reviewed changes

codelipenghui added type/bug The PR fixed a bug or issue reported a bug and removed type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages labels Jul 19, 2023

Addressed comments

aaa4498

codelipenghui added this to the 3.1.0 milestone Jul 19, 2023

codelipenghui approved these changes Jul 19, 2023

View reviewed changes

codelipenghui added release/3.0.2 release/2.11.3 labels Jul 19, 2023

mattisonchao self-requested a review July 19, 2023 08:45

codelipenghui added the release/2.10.6 label Jul 19, 2023

Demogorgon314 added the ready-to-test label Jul 19, 2023

BewareMyPower approved these changes Jul 19, 2023

View reviewed changes

BewareMyPower reviewed Jul 19, 2023

View reviewed changes

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java Outdated Show resolved Hide resolved

Update pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalan…

531e964

…ce/impl/ModularLoadManagerImpl.java Co-authored-by: Yunze Xu <[email protected]>

Demogorgon314 closed this Jul 21, 2023

Demogorgon314 reopened this Jul 21, 2023

codelipenghui merged commit 3f63768 into apache:master Jul 25, 2023

Demogorgon314 added a commit to Demogorgon314/pulsar that referenced this pull request Jul 26, 2023

[fix][broker] Avoid infinite bundle unloading (apache#20822)

a0499bd

(cherry picked from commit 3f63768)

Demogorgon314 added a commit to Demogorgon314/pulsar that referenced this pull request Jul 26, 2023

[fix][broker] Avoid infinite bundle unloading (apache#20822)

19ae109

(cherry picked from commit 3f63768)

Demogorgon314 added a commit that referenced this pull request Jul 26, 2023

[fix][broker] Avoid infinite bundle unloading (#20822)

f674ad1

(cherry picked from commit 3f63768)

Demogorgon314 added the cherry-picked/branch-3.0 label Jul 26, 2023

RobertIndie added release/3.0.1 and removed release/3.0.2 labels Jul 26, 2023

Technoboy- pushed a commit that referenced this pull request Aug 8, 2023

[branch-2.10][fix][broker] Avoid infinite bundle unloading (#20822) (#…

31d59cf

…20877)

shibd added the cherry-picked/branch-2.10 label Oct 21, 2023

shibd added the cherry-picked/branch-2.11 label Oct 21, 2023

BewareMyPower reviewed Aug 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix][broker] Avoid infinite bundle unloading #20822

[fix][broker] Avoid infinite bundle unloading #20822

Demogorgon314 commented Jul 17, 2023 •

edited

Loading

codelipenghui left a comment

codelipenghui left a comment

Demogorgon314 commented Jul 19, 2023

Demogorgon314 Jul 19, 2023

codelipenghui Jul 19, 2023

codelipenghui Jul 19, 2023

Demogorgon314 Jul 19, 2023

codelipenghui Jul 19, 2023

Demogorgon314 Jul 19, 2023

BewareMyPower Jul 19, 2023

Demogorgon314 Jul 19, 2023

heesung-sn Jul 24, 2023

codecov-commenter commented Jul 21, 2023 •

edited

Loading

codelipenghui commented Jul 25, 2023

heesung-sn commented Jul 27, 2023 •

edited

Loading

Demogorgon314 commented Jul 28, 2023

shibd commented Oct 21, 2023

hanmz commented Feb 21, 2024 •

edited

Loading

BewareMyPower Aug 23, 2024

Demogorgon314 Aug 24, 2024

		pulsar.getAdminClient().namespaces()
		.unloadNamespaceBundle(namespaceName, bundleRange, destBroker.get());

[fix][broker] Avoid infinite bundle unloading #20822

[fix][broker] Avoid infinite bundle unloading #20822

Conversation

Demogorgon314 commented Jul 17, 2023 • edited Loading

Motivation

Modifications

Verifying this change

Documentation

codelipenghui left a comment

Choose a reason for hiding this comment

codelipenghui left a comment

Choose a reason for hiding this comment

Demogorgon314 commented Jul 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 21, 2023 • edited Loading

Codecov Report

codelipenghui commented Jul 25, 2023

heesung-sn commented Jul 27, 2023 • edited Loading

Demogorgon314 commented Jul 28, 2023

shibd commented Oct 21, 2023

hanmz commented Feb 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Demogorgon314 commented Jul 17, 2023 •

edited

Loading

codecov-commenter commented Jul 21, 2023 •

edited

Loading

heesung-sn commented Jul 27, 2023 •

edited

Loading

hanmz commented Feb 21, 2024 •

edited

Loading