[Remote State] Add jitter to downloads from remote store

[soosinha]

Is your feature request related to a problem? Please describe

For remote cluster state publication, all the entity metadata in the cluster state are uploaded as separate files. If there are multiple entities in the cluster state that are changed, all these will be uploaded by the active cluster manager and downloaded at the same time by the follower nodes. This sudden burst in traffic to backing remote storage could cause throttling.

Describe the solution you'd like

Add jitter while downloading remote state files to mitigate the risk of throttling.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response

[rajiv-kv]

[Triage - attendees 1 2 3] - @soosinha is this tracked for 2.18 ?

[soosinha]

is this tracked for 2.18 ?

Yes @rajiv-kv

Assigning to @Arpit-Bandejiya