Add a softlimit to the number of open scroll contexts per node

[s1monw]

Today if a user creates a large number of scroll contexts without clearing them using a scroll ID we go and clean things up once a timeout is reached. This has several problems:

• the user doesn't necessarily realize an API call to release resources is necessary
• the node piles up open readers behind the scenes that can have a big impact on disk space and monitoring.
• the node might go OOM due to too many open resources
• the node might suffer from too many open files

we do protect against some of these but it would be nicer to reject scroll requests / new sticky context creation if there are too many of them. This change is rather a low hanging fruit but I suspect a big impact if a users has the related problems. A rejected scroll search would immediately be visible to the user.

[s1monw]

relates to #11511

[nik9000]

Should the limit only apply to scrolls and not regular searches? Should the number of ongoing regular searches count or just scrolls?

[s1monw]

Should the limit only apply to scrolls and not regular searches? Should the number of ongoing regular searches count or just scrolls?

I first thought scrolls only but we can just keep it simple and also so all of them. I already have a branch that does all.

[humblefool1997]

@s1monw
I am up for it .Will Submit a patch within a week .This is my first pr.Can you suggest from where to look on.

[alexshadow007]

@s1monw
Is SearchOperationListener::onNewScrollContext(SearchContext context) good place to add verifying for limit or not?

[s1monw]

@alexshadow007 I wonder if we should rather put it directly into SearchService where we create the context?

[elasticmachine]

Pinging @elastic/es-search-aggs

[pushpavanthar]

@s1monw, @isaeef is this already fixed? Can I pick this up?

[s1monw]

@pushpavanthar it's not fixed afaik

[RuiGuedes]

Hi @pushpavanthar I would like to pick up this issue. Are you still working on it?

[pushpavanthar]

@RuiGuedes No, please go ahead.

[geekpete]

It might be nice to have different limits for different scroll activities, like two pools one for fast/short lived scrolls and another for longer running scrolls.
Then a way to do a scroll against one or other of the scroll types to differentiate between "regular" or short running scrolls and "special" or longer running scrolls that would take up a slot for longer.

The idea might be to have a much smaller longer running scroll max limit to keep those long open scroll contexts to a minimum number vs a larger max limit for short running scrolls.

This would also need some default if there was more than one way to scroll.
A configurable number of pools with configurable settings would also be quite nice to have.

For bonus points, it'd be awesome to be able to tie scroll limits into Security and limit resource usage based on security characteristics like role or user. I realise this might be a separate feature but might be good to keep in mind when designing the basic functionality as part of this feature.