[BUG] [Remote store] S3 throttling during segment upload

[linuxpi]

Describe the bug
Saw s3 throttling during upload during segment upload flow.

[2023-05-02T08:00:42,844][WARN ][o.o.i.s.RemoteStoreRefreshListener] [node-1] Exception while uploading file _ah.cfs to the remote segment store
java.io.IOException: Unable to upload object [cluster-4/MYce-mgETOGu23bJrxMY7w/47/segments/data/_ah.cfs__QmF624cBFqzlrW2FRl23] using a single upload
        at org.opensearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:425) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:153) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.opensearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:61) ~[?:?]
        at org.opensearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:151) ~[?:?]
        at org.opensearch.index.store.RemoteIndexOutput.copyBytes(RemoteIndexOutput.java:41) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.apache.lucene.store.Directory.copyFrom(Directory.java:182) ~[lucene-core-9.6.0-snapshot-a3ae27f.jar:9.6.0-snapshot-a3ae27f a3ae27fd28135a80da36364592a72fb9d54c6cbe - 2023-03-30 19:18:02]
        at org.opensearch.index.store.RemoteSegmentStoreDirectory.copyFrom(RemoteSegmentStoreDirectory.java:327) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.store.RemoteSegmentStoreDirectory.copyFrom(RemoteSegmentStoreDirectory.java:339) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.shard.RemoteStoreRefreshListener.lambda$uploadNewSegments$6(RemoteStoreRefreshListener.java:214) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) [?:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) [?:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) [?:?]
        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715) [?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) [?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) [?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) [?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) [?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) [?:?]
        at org.opensearch.index.shard.RemoteStoreRefreshListener.uploadNewSegments(RemoteStoreRefreshListener.java:212) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.shard.RemoteStoreRefreshListener.afterRefresh(RemoteStoreRefreshListener.java:131) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.apache.lucene.search.ReferenceManager.notifyRefreshListenersRefreshed(ReferenceManager.java:275) [lucene-core-9.6.0-snapshot-a3ae27f.jar:9.6.0-snapshot-a3ae27f a3ae27fd28135a80da36364592a72fb9d54c6cbe - 2023-03-30 19:18:02]
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:182) [lucene-core-9.6.0-snapshot-a3ae27f.jar:9.6.0-snapshot-a3ae27f a3ae27fd28135a80da36364592a72fb9d54c6cbe - 2023-03-30 19:18:02]
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240) [lucene-core-9.6.0-snapshot-a3ae27f.jar:9.6.0-snapshot-a3ae27f a3ae27fd28135a80da36364592a72fb9d54c6cbe - 2023-03-30 19:18:02]
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:432) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:412) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167) [lucene-core-9.6.0-snapshot-a3ae27f.jar:9.6.0-snapshot-a3ae27f a3ae27fd28135a80da36364592a72fb9d54c6cbe - 2023-03-30 19:18:02]
        at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213) [lucene-core-9.6.0-snapshot-a3ae27f.jar:9.6.0-snapshot-a3ae27f a3ae27fd28135a80da36364592a72fb9d54c6cbe - 2023-03-30 19:18:02]
        at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1755) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1734) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4163) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:983) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1116) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: XKP9ENPQ99EVYSYA; S3 Extended Request ID: /XOh0RMCQ/jaLqbC90+igD4ykglJoC61qtWVVHPP8oiiVvZ6XzTNCOGPoiEjPLm9QY6uA7eJMYM=; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[?:?]

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.