[Caching] Perform removal notification operations outside lock for Cache.java

[sgup432]

Is your feature request related to a problem? Please describe

This is specifically related to Cache.java

This is an onHeap cache custom implementation inside OpenSearch which is used by RequestCache and FieldDataCache internally.

Overview of Cache

This cache implementation uses LRU based eviction policy. It uses standard linked list kind of a structure which maintains recently accessed items in the head of the linked list and least recently ones in the tail. We maintain a single global LRU lock(link) to ensure any updates happening to this list are done serially.

What is the problem?

Whenever we add a new item to the cache, we promote it to the head of the linked list under a LRU lock. And while doing it, we also check whether we need to remove any items due to capacity or expiration reason. Any item which is evicted due to capacity or expiration, we send removal notification(in a sync way) for it under the same lock.

How would it to help?

I don't think we need to send out synchronous removal notification under a lock. This can be outside the lock as well. As of now removal notification are used by the clients(like RequestCache etc) to calculate stats etc and eventually is going to be used to put item onto disk cache via tiered caching. So time to release the lock is more which makes other thread wait and impacts cache's read/write performance.

Describe the solution you'd like

Solution is to move the removal notification operations (link) outside the lru lock.

Related component

Search:Performance

Describe alternatives you've considered

We can also eventually provide a way to trigger this listeners in an asynchronous way.

Additional context

No response