S3 interface

[parasj]

Fast S3 interface to read/write files from disk using the AWS S3 common runtime library.

Downloads 1GB file at >5Gbps and uploads 1GB file at 4Gbps.

(base) ubuntu@ip-172-31-14-137:~/skylark$ python skylark/replicate/s3_interface.py --file-size-mb 1000
2021-12-22 21:31:15.779 | DEBUG    | __main__:<module>:108 - Generated random file /tmp/tmpy2dhwo1b
2021-12-22 21:31:17.874 | DEBUG    | __main__:<module>:110 - File md5: aed9e292204cd78c0cb32873f3e0eb94
2021-12-22 21:31:19.861 | INFO     | __main__:<module>:117 - UL /tmp/tmpy2dhwo1b -> s3://us-east-1/sky-us-east-1/test.txt in 1.99s (~4.03Gbps)
2021-12-22 21:31:21.077 | INFO     | __main__:<module>:128 - DL s3://us-east-1/sky-us-east-1/test.txt -> /tmp/tmpspa4lxp6 in 1.11s (~7.24Gbps)
2021-12-22 21:31:23.147 | INFO     | __main__:<module>:135 - MD5 match: uploaded aed9e292204cd78c0cb32873f3e0eb94 == downloaded aed9e292204cd78c0cb32873f3e0eb94

For a large (uncached) file, downloads at 1.6Gbps and uploads at 5Gbps.

(base) ubuntu@ip-172-31-14-137:~/skylark$ python skylark/replicate/s3_interface.py --file-size-mb 10000
2021-12-22 21:33:44.194 | DEBUG    | __main__:<module>:108 - Generated random file /tmp/tmp74vccmi4
2021-12-22 21:34:04.986 | DEBUG    | __main__:<module>:110 - File md5: 8ccd6d271bf4f25a35be411403cca192
2021-12-22 21:34:20.403 | INFO     | __main__:<module>:117 - UL /tmp/tmp74vccmi4 -> s3://us-east-1/sky-us-east-1/test.txt in 15.42s (~5.19Gbps)
2021-12-22 21:35:11.091 | INFO     | __main__:<module>:128 - DL s3://us-east-1/sky-us-east-1/test.txt -> /tmp/tmpuy1dbc21 in 49.74s (~1.61Gbps)
2021-12-22 21:35:31.867 | INFO     | __main__:<module>:135 - MD5 match: uploaded 8ccd6d271bf4f25a35be411403cca192 == downloaded 8ccd6d271bf4f25a35be411403cca192