Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make ClusterInfo use immutable maps in all cases #88447

Merged

Conversation

original-brownbear
Copy link
Member

This class's maps are used very hot in the disk threshold allocation
decider. Moving them from hppc maps to unmodifiable map wrapping
HashMap has led to a measurable slowdown in the many-shards benchmark
bootstrapping. Lets use immutable map copies here exclusively to make
performance outright better and more predictable via a single implementation.

relates #77466

This class's maps are used very hot in the disk threshold allocation
decider. Moving them from hppc maps to unmodifiable map wrapping
`HashMap` has led to a measurable slowdown in the many-shards benchmark
bootstrapping. Lets use immutable map copies here exclusively to make
performance outright better and more predictable via a single implementation.
@original-brownbear original-brownbear added >non-issue :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) v8.4.0 labels Jul 11, 2022
@original-brownbear original-brownbear marked this pull request as ready for review July 11, 2022 16:25
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Jul 11, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@original-brownbear
Copy link
Member Author

Jenkins run elasticsearch-ci/part-2 (unrelated)

Map.copyOf(shardSizeByIdentifierBuilder),
Map.copyOf(shardDataSetSizeBuilder),
Map.copyOf(dataPathByShardRoutingBuilder),
Map.copyOf(rsrvdSpace)
Copy link
Contributor

@idegtiarenko idegtiarenko Jul 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the type of the map is critical, would it be better to move this inside of the constructor so that it is impossible to create the object with wrong (slower) map implementation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I thought about this but then I figured there's only this one spot that we actually do a real instantiation of this thing and it's private so it's probably not necessary to add extra verbosity.

@original-brownbear
Copy link
Member Author

Thanks Ievgen!

@original-brownbear original-brownbear merged commit 9ebbe1c into elastic:master Jul 12, 2022
@original-brownbear original-brownbear deleted the faster-cluster-info branch July 12, 2022 07:03
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Jul 12, 2022
* upstream/master:
  Pass IndexMetadata to AllocationDecider.can_remain (elastic#88453)
  [TSDB] Cache rollup bucket timestamp to reduce rounding cost (elastic#88420)
  Correct some typos/mistakes in comments/docs (elastic#88446)
  Make ClusterInfo use immutable maps in all cases (elastic#88447)
  Reduce map lookups (elastic#88418)
  Don't index geo_shape field in AbstractBuilderTestCase (elastic#88437)
  Remove usages of TestGeoShapeFieldMapperPlugin from enrich module (elastic#88440)
  Fix test memory leak (elastic#88362)
  Improve error when sorting on incompatible types (elastic#88399)
  Remove usages of BucketCollector#getLeafCollector(LeafReaderContext) (elastic#88414)
  Mute ReactiveStorageIT::testScaleWhileShrinking (elastic#88431)
  Clarify snapshot docs on archive indices (elastic#88417)
  [Stack Monitoring] Switch cgroup memory fields to keyword (elastic#88260)
  Fix RealmIdentifier XContent parser (elastic#88410)
  Make LoggedExec gradle task configuration cache compatible (elastic#87621)
  Update CorruptedFileIT so that it passes with new allocation strategy (elastic#88314)
  Update RareClusterStateIT to work with the new shards allocator (elastic#87922)
  Ensure CreateApiKey always creates a new document (elastic#88413)

# Conflicts:
#	x-pack/plugin/rollup/src/main/java/org/elasticsearch/xpack/rollup/v2/RollupShardIndexer.java
@original-brownbear original-brownbear restored the faster-cluster-info branch April 18, 2023 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >non-issue Team:Distributed Meta label for distributed team v8.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants