[BUG] Concurrent index deletion during ongoing search request throws 5xx error

[jainankitk]

Describe the bug

If an index gets deleted while the shard search request has not been sent, the overall search request results in 5xx error. That should not be the case, since the customer has explicitly deleted index and not an internal server error. No index/shard, no search results/delete by query

Related component

Search

To Reproduce

1. Create 200 indices on a node
2. Start expensive search request for all the indices
3. Delete the indices on which request has not started execution yet
4. The search request eventually results in 5xx error

Expected behavior

During the search execution, only consider shard result as failure if the exception is anything other than index/shard not found. Eventually, if all the shards within search request do not return any results (probability of this is low), the 200 OK response can be changed to 404, but we should never be throwing 5xx.

Additional Details

No response

[jainankitk]

Related to #8418

[msfroh]

@jainankitk -- This only applies if the search request is happening as part of an AbstractAsyncBulkByScrollAction, right?

I was under the impression that the trigger is this line from ClientScrollableHitSource:

OpenSearch/server/src/main/java/org/opensearch/index/reindex/ClientScrollableHitSource.java

Line 89 in aca2e9d

firstSearchRequest.allowPartialSearchResults(false);

A regular search request will return with some failed shards mentioned in the response (unless allowPartialResults has explicitly been set to false).

The three forms of AbstractAsyncBulkByScrollAction right now are _reindex, _update_by_query, and _delete_by_query. I can see how both _update_by_query and _delete_by_query could gracefully handle an index disappearing mid-job. I feel like _reindex should fail if a source index suddenly disappears.

[peternied]

[Triage - attendees 1 2 3 4 5]
@jainankitk Thanks for filing, look forward to a pull request to address this issue.

[jainankitk]

The three forms of AbstractAsyncBulkByScrollAction right now are _reindex, _update_by_query, and _delete_by_query. I can see how both _update_by_query and _delete_by_query could gracefully handle an index disappearing mid-job. I feel like _reindex should fail if a source index suddenly disappears.

Not really right!? Since, _reindex is just copying over data from a source of truth. If the source of truth changes midway, it is expected that the copied data will be less than the original. But it is guaranteed to be more than the final source of truth since few deleted shards might have been copied over before they were deleted.

[jainankitk]

@msfroh - What do you think about using ignore_unavailable_shards parameter to let users take control of the behavior they want? We can expand #13209 further to use pass this parameter from _delete_by_query, _update_by_query and _reindex actions