Support S3 transferring use "ProcessPoolExecutor" with s3tranfer #8852
Labels
feature-request
A feature should be added or improved.
needs-review
This issue or pull request needs review from a core team member.
p2
This is a standard priority issue
s3transfer
s3
Describe the feature
As I know, aws-cli uses "s3tranfer" to use s3 and it uses
ThreadPoolExecutor
like thishttps://github.com/boto/s3transfer/blob/da68b50bb5a6b0c342ad0d87f9b1f80ab81dffce/s3transfer/futures.py#L402-L403
In some environment like enough available network bandwidth, enough CPU cores and lots for files to be downloaded, then using
ProcessPoolExecutor
would be better.And
s3transfer
has implemented an interface to useProcessPoolExecutor
https://github.com/boto/s3transfer/blob/develop/s3transfer/processpool.py
So I think addding feature flag for selecting
thread
orprocess
for using s3 that could be determined by user would be better.Use Case
If we have to download many files and the environment that using aws-cli has enough resources(CPU, Memory, Network bandwitdh) than we can choice that use more CPUs for boost the S3 throughput
Proposed Solution
No response
Other Information
No response
Acknowledgements
CLI version used
2.15.30
Environment details (OS name and version, etc.)
Amazon Linux 2023
The text was updated successfully, but these errors were encountered: