[Remote Cluster State] Send remote cluster state only when all nodes in the cluster are remote state enabled

[soosinha]

Is your feature request related to a problem? Please describe

Before sending the remote cluster state during cluster state publication, we check if a node has remote cluster state enabled or not. We send the cluster state over transport layer if a node is not remote cluster state enabled.
This mixed behavior would be difficult to maintain.

Describe the solution you'd like

We should instead check if all nodes are remote cluster state enabled before starting the publication and then send remote state to all nodes.
We can have a new RemotePublicationContext which extends PublicationContext and will be used to remote cluster state.
Also, there should be a mapping of OpenSearch version to the codec version supported. The remote state should be published for the lowest codec version supported based on the node versions in the cluster.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response

[rwali-aws]

[Triage - attendees 1 2 3 4 5 6 ]

Thanks for filing this issue, please feel free to submit a pull request.