[BUG] The split index's shards cannot be assigned because of the source index's index.blocks.read_only setting

[gaobinlong]

Describe the bug
When executing Split API , if we set the index.blocks.read_only setting of the source index to true, then the Split API responds slowly and finally return an action_request_validation_exception with 400 status code, but the new splitted index is created and only part of the new shards are STARTED, so the new split index's health is red. By checking the log, it shows that the unassigned shards cannot be recovered because the new split index has a read_only block:
Caused by: org.opensearch.cluster.block.ClusterBlockException: index [index111] blocked by: [FORBIDDEN/5/index read-only (api)]

I think it means that the new spit index's settings are copied from the source index, so the index.blocks.read_only setting of the new spit index is true. After changing the setting to false and executing POST _cluster/reroute?retry_failed=true to retry recovering the unassigned shards, the new spit index's health then became green.

So, should we add more validations for Split & Shrink & Clone API ? Maybe we can throw an action_request_validation_exception with 400 when the source index's index.blocks.read_only or index.blocks.metadata setting was set to true.

Besides, why only parts of new spit index's shards are assigned and in STARTED status? That's really confused and there might be some other underlying problems.

To Reproduce
Steps to reproduce the behavior:

1. Update the source index's index.blocks.read_only setting to true

PUT opensearch_dashboards_sample_data_ecommerce/_settings
{
  "index.blocks.read_only":true
}

2. Executing the Split API

POST opensearch_dashboards_sample_data_ecommerce/_split/index111
{
  "settings":{
    "index.number_of_shards":6,
    "index.number_of_replicas":0
  }
}

the response is:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "action_request_validation_exception",
        "reason" : "Validation Failed: 1: index.number_of_shards is required for split operations;"
      }
    ],
    "type" : "action_request_validation_exception",
    "reason" : "Validation Failed: 1: index.number_of_shards is required for split operations;"
  },
  "status" : 400
}

3. Check the new split index

GET _cat/shards/index111
GET index111/_recovery
GET _cluster/health/index111

we can see that only parts of the new shards are STARTED and the new index's health is red.

Expected behavior

Add more validations for Split & Shrink & Clone API to avoid the above situation.

Host/Environment (please complete the following information):

• OS: macOS
• Version: 12.5.1
  The opensearch version is 2.3.

[anasalkouz]

@nandi-github Could you look into this?

[gaobinlong]

From the code I found that there is a block check, but it's useless because the new split index has not been created yet:

OpenSearch/server/src/main/java/org/opensearch/action/admin/indices/shrink/TransportResizeAction.java

Line 128 in 31bcda6

return state.blocks().indexBlockedException(ClusterBlockLevel.METADATA_WRITE, request.getTargetIndexRequest().index());

.
So maybe we should check the MATADATA_WRITE block when creating the new split index, like this:

OpenSearch/server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

Line 1314 in 31bcda6

if (state.blocks().indexBlocked(ClusterBlockLevel.WRITE, sourceIndex) == false) {

.

I think we can throw an illegalArgumentException if the new split index has a METADATA_WRITE block because it will prevent the new shards from being allocated , there are two situations the new split index may have a METADATA_WRITE block:

1. User sets index setting index.blocks.read_only or index.blocks.metadata to true explicitly in the API's request body parameter settings.
2. The source index's setting index.blocks.read_only is true, and user does not set index setting index.blocks.read_only to false explicitly in the API's request body(if user does so, the setting in the request body will overwrite the setting copied from the source index).

For another situation, if the source index's setting index.blocks.metadata is true, a cluster_block_exception will be thrown early in getting source index's stats, so the new split index will not have a METADATA_WRITE block.

@nandi-github may I know your opinion about this?