Allow adjustment of index resource constraints in ILM phase transitions

[DaveCTurner]

The shards allocator mostly does a good job of spreading the shards of each index out across the available nodes, but there are some corner cases where it might temporarily concentrate too many hot shards on a single node, leading to poor performance and possible resource exhaustion (#17213). It is occasionally useful to add the index.routing.allocation.total_shards_per_node constraint when indices are under heavy load, to prevent too many hot shards from being allocated to the same node. We are contemplating generalising this to a more nuanced set of per-index resource constraints too (#17213 (comment)).

If an index is managed by ILM then it is likely its resource requirements will change along with its lifecycle. In particular when the index leaves the hot phase it will no longer see such heavy indexing, and so any total_shards_per_node constraint is likely no longer necessary. It's important to keep the use of this sort of setting to a minimum to avoid over-constraining the allocator since this can lead to unassigned shards (#12273). I therefore think it'd be a good idea to allow this kind of setting to be changed in the appropriate ILM phase transitions.

[elasticmachine]

Pinging @elastic/es-core-features

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

I raised this question with the distrib team today and although we're not super-keen on recommending the widespread use of the total_shards_per_node constraint, we think it's a good idea to allow it to be relaxed on ILM phase transitions, particularly the hot-to-warm one.

[vigyasharma]

Are per-index resource constraints being planned as prescriptive - as in allocator makes a best effort to comply by constraints but allocates the shards anyway if no node is able to fulfill its requirements, or will it be a hard limit, and leave shards unallocated if not fulfilled?

Resource requirements are often quite dynamic. Even in the hot stage, and index may have off peak hours when the reserved resources could've been shared by other indices.

Requirement prediction itself is hard with users almost always under/over provisioning in the first pass. It needs iterative fine tuning to get the limits right. I would vote for a prescriptive best case effort to allow for such flexibility.

Separately, +1 on revising constraints through ILM transitions.

[dakrone]

We discussed this and agreed on putting an option for setting index.routing.allocation.total_shards_per_node into the allocate action in ILM.

[avneesh91]

@DaveCTurner would it be ok if I started working on this?

[dakrone]

@avneesh91 sure if you'd like to contribute a PR for this that would be great.

[shoaib4330]

@avneesh91 I'd like to pick this issue, if you're not actively working on it?

[epicvinny]

It is impracticable to use ILM at scale :(

[Aloshi]

Another +1 to this issue. We've been having problems when we scale up our larger clusters by 2-3 nodes where ES decides to allocate all of the shards for the next index on the new nodes, which causes abysmal performance and slows down ingestion. We've been guarding against it by setting index.routing.allocation.total_shards_per_node to 1 or 2 on our big indices until the new nodes are mostly full, and we'd like to just leave this set all the time, but when these indices are moved to warm nodes there aren't enough warm nodes to satisfy the constraint (since having fewer warm nodes than hot nodes is kind of the point).

Is there any update on this? It sounds like the proposed solution would work great for us.

[ppf2]

+1 Shard imbalance necessitating the use of total_shards_per_node does not only apply to the heavy indexing use case, we have seen this affect other cluster operations such as force merge when a majority of shards of an index being force merged ended up concentrating on a single warm node.

[JohnLyman]

We discussed this and agreed on putting an option for setting index.routing.allocation.total_shards_per_node into the allocate action in ILM.

Resetting total_shards_per_node is also required when using the shrink action which necessitates that all primary shards be moved to the same node.

@dakrone - if this option is put in the allocate action, would that allow for the value to be reset prior to shrink? I'm not sure of the order of operations here (I don't think it's documented). In other words, does allocate come before shrink in the warm phase?

[dakrone]

if this option is put in the allocate action, would that allow for the value to be reset prior to shrink? I'm not sure of the order of operations here (I don't think it's documented). In other words, does allocate come before shrink in the warm phase?

allocate does come before shrink in the warm phase, though I think regardless we should endeavor to make shrink run regardless of the setting, unsetting the setting during the allocation, then resetting it if necessary.

[JohnLyman]

That seems reasonable @dakrone (and thanks for the quick reply). Should I open a new issue for your suggestion?

[dakrone]

@JohnLyman I think it can be subsumed into this issue for now, thanks for bringing it up!

[JohnLyman]

I was thinking for shrink in general, whether it's initiated by ILM or not.

[dakrone]

I think that is one of the main goals of #63519