[BUG] cluster.max_shards_per_node should have an upper limit

[kkewwei]

Describe the bug

In our product, at first, user set cluster.max_shards_per_node to 500_000_000 to avoid shard unassigned. after days later, the nodes of the cluster are extended from 10 to 15. and the new created shard can't be unassigned because of the error:

"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [30101]/[-1089934592] maximum shards open;"

The cluster have 15 node, when calculating maxShardsInCluster, an int overflow occurred.

If the user don't extend the cluster from 10 to 15, an int overflow will not occur.

If we should limit the upper of cluster.max_shards_per_node, such as 5000_000, it can't be reached in the current es struct. I'm pleasure to fix it.

Related component

Cluster Manager

Additional Details

Host/Environment (please complete the following information):

• OS2.9

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@kkewwei Thank for creating this issue, could you create a pull request to address?