[Remote Store] Add setting to limit primary shards per node for an index / all indices

[gbbafna]

Is your feature request related to a problem? Please describe

Today we have a setting index.routing.allocation.total_shards_per_node , which can limit maximum number of shards per index/across all indices on a single OpenSearch node. For remote store based clusters, we would also like to limit maximum number of primary shards per index/across all indices . This is because on remote store , only the primaries index the data , while both primaries and replicas serve the search request. Hence ,the primary shards are more CPU heavy , as only they do the indexing .

Describe the solution you'd like

For remote store based clusters, we would also like to limit maximum number of primary shards per index/across all indices .

Related component

Storage:Durability

Describe alternatives you've considered

NA

Additional context

NA

[andrross]

This is a general segment replication concern and not specific to remote store, right?

[gbbafna]

This is a general segment replication concern and not specific to remote store, right ?

Yes, that is correct Andrew.

[sachinpkale]

@gbbafna There is a separate issue created to #12250 Do we still need this setting change?

[gbbafna]

@sachinpkale : rebalancing is not deterministic . This proposed setting will make things quite deterministic. Want to know thoughts of @Arpit-Bandejiya as well on this.

[gbbafna]

[Storage Triage meeting] We don't see the need for this right now. Will revisit when we see the need in future.