Send clear session as routable remote request

[Tim-Brooks]

This commit adds a RemoteClusterAwareRequest interface that allows a
request to specify which remote node it should be routed to. The remote
cluster aware client will attempt to route the request directly to this
node. Otherwise it will send it as a proxy action to eventually end up
on the requested node.

It implements the ccr clean_session action with this client.

[elasticmachine]

Pinging @elastic/es-distributed

[Tim-Brooks]

This includes some of the changes that we discussed related to routing requests to a specific node. If we like this work, it can also be used to implement the ccr file chunk requests.

[ywelsch]

@tbrooks8 Thanks for exploring this solution. What I don't like about this solution is that we're now introducing the concept of a target node in the Client interface (even though it's only in the RemoteClient sub-interface) whereas that interface has previously been completely agnostic of that.

An alternative would be to introduce a new RemoteClusterAwareRequest interface with a method DiscoveryNode getPreferredTargetNode() that returns a preferred target node. That interface can then be implemented by specific ActionRequests, and we then change RemoteClusterAwareClient to be aware of this interface and select the connection based on the preferred target node. @tbrooks8 WDYT?

[Tim-Brooks]

An alternative would be to introduce a new RemoteClusterAwareRequest interface with a method DiscoveryNode getPreferredTargetNode()

So do you:

1. Still want to have the remote client sub-interface that takes this request type
   or
2. Internally in the remote client do an instanceof check

[ywelsch]

I would be fine with option 2 (instanceof check). Option 1 would require adding two more execute methods to the RemoteClient interface.

[Tim-Brooks]

@ywelsch I have implemented your suggestion.

Unfortunately, I think that there is another issue we will need to work through. The remote cluster service will send an action that is not directly to the remote node as a proxy action internal:transport/proxy/. The system user is not authorized for this type of action so I think that the action will be rejected with security enabled.

The client (I think) will be using the system user because the client is passed in at repository creation time. And the actions will be executed in the restoreShard method in the ccr repository. This will happen on the generic thread pool so we will have lost the headers (and original user) from the put follow action request.

I'll mention @tvernum @jaymode here as they might know the best way I should confront this issue. I don't see a clear way to work the headers through into the repository since it is hooked up through the normal restore mechanism.

[ywelsch]

LGTM. Let's get @jaymode or @tvernum's input first before merging.

[jaymode]

I think we should introduce a CCR origin instead of relying on the system user for these actions (proxied or non proxied). The client can be wrapped with one that always uses the origin IF everything should be executed as a user specific to CCR. In #35667, the OriginSettingClient was introduced that would do the trick.