Kibana migration to 7.13.2 issue

[arcenik]

Kibana version:
7.13.2-amd64

Elasticsearch version:
7.13.2-amd64

Server OS version:
n/a using docker images

Original install method (e.g. download page, yum, from source, etc.):
docker images

Describe the bug:
Migrating Kibana from 7.10.1 to 7.13.2

Steps to reproduce:

1. Change the image name in docker-compose.yml
2. Deploy the news service with only one instance
3. See the container restart in loop

Expected behavior:
Kibana should make a backup copy and removes corrupt saved objects so it can start.

Provide logs and/or server output (if relevant):

{"type":"log","@timestamp":"2021-06-23T14:46:57+00:00","tags":["fatal","root"],"pid":952,"message":"Error: Unable to migrate the corrupt saved object document with _id: 'Maps-maps_telemetry'. To allow migrations to proceed, please delete this document from the [.kibana_task_manager_7.13.2_001] index.\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:162:15)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at async Promise.all (index 1)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:163:7)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:275:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:55:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:98:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:224:5)"}

{"type":"log","@timestamp":"2021-06-23T14:46:57+00:00","tags":["info","plugins-system"],"pid":952,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-06-23T14:46:57+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":952,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-06-23T14:46:57+00:00","tags":["info","savedobjects-service"],"pid":952,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 5538ms."}
{"type":"log","@timestamp":"2021-06-23T14:46:57+00:00","tags":["info","savedobjects-service"],"pid":952,"message":"[.kibana] Migration completed after 5605ms"}
{"type":"log","@timestamp":"2021-06-23T14:47:27+00:00","tags":["warning","plugins-system"],"pid":952,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}

 FATAL  Error: Unable to migrate the corrupt saved object document with _id: 'Maps-maps_telemetry'. To allow migrations to proceed, please delete this document from the [.kibana_task_manager_7.13.2_001] index.

[arcenik]

The manual solution for those who have the same issue:

$ curl -s -XDELETE 'your-es-server:9200/.kibana_task_manager_7.13.2_001/_doc/Maps-maps_telemetry'; echo
{"_index":".kibana_task_manager_7.13.2_001","_type":"_doc","_id":"Maps-maps_telemetry","_version":2,"result":"deleted","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":13,"_primary_term":2}

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[Bamieh]

@arcenik thanks for opening this issue!

maps no longer uses the task manager to store its usage. Does TM migrations handle removed registered tasks gracefully?

Expected behavior:
Kibana should make a backup copy and removes corrupt saved objects so it can start.

AFAIK this is not currently the expected behavior. the current expected behavior is to manually resolve failed migrations to continue (not every case is solved by deleting failed docs)

[gmmorris]

Does TM migrations handle removed registered tasks gracefully?

If a type is removed, its tasks are marked as unrecognized and left in the index.
It might be that we need a new mechanism for retiring task types, but there isn't anything in place right now.

This shouldn't have an impact on migrations though 🤔
Any thoughts @mikecote ?

[gmmorris]

AFAICT This doesn't appear to be a TM issue? Should this be labeled as Saved Objects instead?

[mikecote]

++ I don't believe the saved objects migration system supporting deletion at this time.

[pgayvallet]

I confirm that this is not a TM issue, but a migration one, and was resolved by #104507 in 7.14 (or #105213 which will land in the same version)

Closing.