Upgrade assistant tells the user to try again after reindex fails without clear remediation steps

[bhavyarm]

Kibana version: 7.17 BC2

Elasticsearch version: 7.17 BC2

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS x

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: If Kibana restarts during reindexing - Upgrade assistant displays reindex fails for some indexes and if user tries to reindex those indices again - it keeps failing. But "Try again" button is active which gives the impression that user can try to reindex again to resolve the failure.

This needs to be fixed to explain the remediation steps clearly.

Steps to reproduce:

1. Start Kibana/ES 7.17 BC2 with 6.8.20 data
2. Go to upgrade assistant - start reindexing the indices
3. Restart Kibana
4. Some of the indices will have reindex failed status

5. Click on index and you can see that reindexing failed because of the error: ``` {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [reindexed-v7-kibana_sample_data_logs/HITu6ptSSNyzWKwUxoW9Kg] already exists","index_uuid":"HITu6ptSSNyzWKwUxoW9Kg","index":"reindexed-v7-kibana_sample_data_logs"}],"type":"resource_already_exists_exception","reason":"index [reindexed-v7-kibana_sample_data_logs/HITu6ptSSNyzWKwUxoW9Kg] already exists","index_uuid":"HITu6ptSSNyzWKwUxoW9Kg","index":"reindexed-v7-kibana_sample_data_logs"},"status":400} ```

6. But "Try again" button is active which gives the impression to the user to that problem can be fixed.

Kibana logs:

  [10:12:45.659] [error][plugins][reindex_worker][upgradeAssistant] Reindexing step failed: ResponseError: resource_already_exists_exception: [resource_already_exists_exception] Reason: index [reindexed-v7-kibana_sample_data_logs/HITu6ptSSNyzWKwUxoW9Kg] already exists
    at onBody (/Users/bhavyarajumandya/Desktop/compatibility_7.17_8.0/kibana-7.17.0-darwin-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:367:23)
    at IncomingMessage.onEnd (/Users/bhavyarajumandya/Desktop/compatibility_7.17_8.0/kibana-7.17.0-darwin-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)
    at IncomingMessage.emit (node:events:402:35)
    at endReadableNT (node:internal/streams/readable:1343:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[cjcenizal]

@bhavyarm The steps to reproduce look very similar to the steps in #123616. Are they the same? If not, could you clarify the difference? If the steps are indeed the same then can you explain how the two issues are different? Thank you!

[bhavyarm]

@cjcenizal that ( #123616) is dealing with the bug of resume reindex without any errors whereas this one is for the case when reindex fails with error but upgrade assistant UI is saying "Try again". So two different states. Thanks!

[cjcenizal]

Thanks @bhavyarm. Am I correct that the steps to repro are identical, and yet we're still encountering two different possible outcomes? So we need to dig deeper to understand what's causing this variability?

[bhavyarm]

@cjcenizal yep. Looks like it. Thanks!

[sebelga]

@cjcenizal I am going to look at both issue at the same time 👍

[alisonelizabeth]

@bhavyarm are you able to consistently reproduce this? I'm having a hard time. Do you know at what step in the reindexing process you are restarting Kibana to trigger this error?

[alisonelizabeth]

I'm still not able to reproduce this yet. However, based on the error in your screenshot, it would appear that UA thinks it's on a different step in the reindexing process than it actually is, i.e., it thinks it has not created a new index yet so it attempts to create one again and the error occurs because it already exists.

This is the area of the code where the error is triggered: https://github.com/elastic/kibana/blob/7.17/x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts#L493.

[bhavyarm]

@alisonelizabeth I started reindexing the indices from the top - had multiple reindex operations running when I restarted Kibana. I have the ES/Kibana in which this error happened in my local. Let me know if it helps you to take a look at it. Thanks!

[alisonelizabeth]

Thanks @bhavyarm! @sabarasaba and @sebelga found a way to replicate the behavior.

[alisonelizabeth]

Fixed via #123817.