Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve] [broker] disable balancing based on DirectMemory. #21168

Merged
merged 2 commits into from
Sep 14, 2023
Merged

[improve] [broker] disable balancing based on DirectMemory. #21168

merged 2 commits into from
Sep 14, 2023

Conversation

thetumbled
Copy link
Member

@thetumbled thetumbled commented Sep 12, 2023 •
edited
Loading

Motivation

As shown in the following figure, the throughput in/out, message in/out, bandwidht in/out usage and CPU usage go up and down periodically as time goes by, while the memory usage and direct memory usage has nothing to do with the actual load.
image
image
image
image

Using memory usage and direct memory usage to score the broker load will result into wrong result.

Modifications

Removing memory usage and direct memory usage for scoring.
As PR#19559 has removed memory usage, i will remove direct memory usage for scoring.
Change the default value of loadBalancerDirectMemoryResourceWeight.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: thetumbled#30

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Sep 12, 2023
@thetumbled thetumbled mentioned this pull request Sep 12, 2023
Closed
@thetumbled
Copy link
Member Author

Could you help to review this PR? thanks. @heesung-sn @Demogorgon314

Demogorgon314
Demogorgon314 approved these changes Sep 12, 2023
View reviewed changes
Copy link
Member

@Demogorgon314 Demogorgon314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@heesung-sn
Copy link
Contributor

Can we keep this direct memory signal but the default loadBalancerDirectMemoryResourceWeight to zero? I wonder if direct memory can be useful for some other cases.

@thetumbled
Copy link
Member Author

thetumbled commented Sep 13, 2023 •
edited
Loading

Can we keep this direct memory signal but the default loadBalancerDirectMemoryResourceWeight to zero? I wonder if direct memory can be useful for some other cases.

It is weird that we remove momory signal but retain direct memory signal, right?
IMO, either we retain both or remove both.

@heesung-sn
Copy link
Contributor

Can we keep this direct memory signal but the default loadBalancerDirectMemoryResourceWeight to zero? I wonder if direct memory can be useful for some other cases.

It is weird that we remove the memory signal but retain direct memory signal, right? IMO, either we retain both or remove both.

IMO, these removals need to be agreed by the community. I don't know if any users rely on these memory and direct memory signals in their production. This seems to be a radical decision.

@thetumbled
Copy link
Member Author

Can we keep this direct memory signal but the default loadBalancerDirectMemoryResourceWeight to zero? I wonder if direct memory can be useful for some other cases.

It is weird that we remove the memory signal but retain direct memory signal, right? IMO, either we retain both or remove both.

IMO, these removals need to be agreed by the community. I don't know if any users rely on these memory and direct memory signals in their production. This seems to be a radical decision.

It does. My original idea is to set the default value of loadBalancerDirectMemoryResourceWeight、loadBalancerMemoryResourceWeight to be zero. But, i found that PR #19559 has removed memory signal, so i follow it.
WDYT, @hangc0276 @Demogorgon314

@hangc0276
Copy link
Contributor

I think we can change the default value to 0

@thetumbled thetumbled changed the title [improve] [broker] remove balancing based on DirectMemory. [improve] [broker] disable balancing based on DirectMemory. Sep 13, 2023
@thetumbled
Copy link
Member Author

/pulsarbot rerun-failure-checks

@BewareMyPower BewareMyPower merged commit b10eed6 into apache:master Sep 14, 2023
46 of 47 checks passed
@Technoboy- Technoboy- added this to the 3.2.0 milestone Sep 18, 2023
Technoboy- pushed a commit that referenced this pull request Sep 18, 2023
@thetumbled @Technoboy-
[improve] [broker] disable balancing based on DirectMemory. (#21168)
4139291
@lhotari lhotari mentioned this pull request Feb 29, 2024
Open
2 tasks
@lhotari
Copy link
Member

lhotari commented Feb 29, 2024

There's a related enhancement proposal in #21973 , please take a look.

@thetumbled thetumbled mentioned this pull request Jun 3, 2024
Merged
15 tasks
@thetumbled thetumbled mentioned this pull request Jun 11, 2024
Merged
15 tasks
Demogorgon314 pushed a commit that referenced this pull request Jun 17, 2024
@thetumbled
[improve] [pip] PIP-358: let resource weight work for OverloadShedder…
9f8adc5 
…, LeastLongTermMessageRate, ModularLoadManagerImpl. (#22889)

Implementation PR: #22888

### Motivation

Initially, we introduce `loadBalancerCPUResourceWeight`, `loadBalancerBandwidthInResourceWeight`, `loadBalancerBandwidthOutResourceWeight`, `loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` in `ThresholdShedder` to control the resource weight for different resources when calculating the load of the broker. 

Then we let it work for `LeastResourceUsageWithWeight` for better bundle placement policy.

But #19559 and #21168 have point out that the actual load of the broker is not related to the memory usage and direct memory usage, thus we have changed the default value of `loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight` to 0.0.


There are still some places where memory usage and direct memory usage are used to calculate the load of the broker, such as `OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`. We should let the resource weight work for these places so that we can set the resource weight to 0.0 to avoid the impact of memory usage and direct memory usage on the load of the broker.


### Modifications

- Let resource weight work for `OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`.
@lhotari lhotari mentioned this pull request Jun 24, 2024
Merged
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@heesung-sn heesung-sn heesung-sn approved these changes

@aloyszhang aloyszhang aloyszhang approved these changes

@Demogorgon314 Demogorgon314 Demogorgon314 approved these changes

@BewareMyPower BewareMyPower BewareMyPower approved these changes

@hangc0276 hangc0276 hangc0276 approved these changes

Assignees

@thetumbled thetumbled

Labels
cherry-picked/branch-3.1 doc-not-needed Your PR changes do not impact docs release/3.1.1
Projects
None yet
Milestone
3.2.0
Development

Successfully merging this pull request may close these issues.
8 participants
@thetumbled @heesung-sn @hangc0276 @lhotari @BewareMyPower @Demogorgon314 @aloyszhang @Technoboy-