[BUG] cancellation_time_millis is not getting added in .tasks index

[soosinha]

Describe the bug

When a task is cancelled, the .tasks index fails to update because this index has strict dynamic mapping enabled and cancellation_time_millis field is not present in the mappings.
The exception logged is below:

[2024-09-13T20:33:48,799][WARN ][o.o.t.TaskManager        ] [8a67d3f648f0be8167ed2246724ec0d2] couldn't store response org.opensearch.replication.task.CrossClusterReplicationTask$CrossClusterReplicationTaskResponse@74ef241c
 StrictDynamicMappingException[mapping set to strict, dynamic introduction of [cancellation_time_millis] within [task] is not allowed]
         at org.opensearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:835)
         at org.opensearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:684)
         at org.opensearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:457)
         at org.opensearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:415)
         at org.opensearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:520)
         at org.opensearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:541)
         at org.opensearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:443)
         at org.opensearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:415)
         at org.opensearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:137)
         at org.opensearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:92)
         at org.opensearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:256)
         at org.opensearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1188)
         at org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1145)
         at org.opensearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:1062)
         at org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:625)
         at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:471)
         at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
         at org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:535)
         at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:416)
         at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:125)
         at org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)
         at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:922)
         at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
         at java.lang.Thread.run(Thread.java:840)

Related component

Other

To Reproduce

1. Create a cluster with CCR plugin
2. Create an auto follow rule
3. delete the auto follow rule
4. Observe that the exception about .tasks index should appear in logs

Expected behavior

.tasks index should get updated correctly.
I see that the .tasks index is created using the mappings defined here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/resources/org/opensearch/tasks/task-index-mapping.json

These mappings should contain cancellation_time_millis fields as well.

Additional Details

Plugins
Please list all plugins currently enabled.
cross-cluster-replication

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.