20240301-minutes.md

S3 Working Group Minutes

Friday, 2024/03/01

3:00 pm ET

Attendees:

Terrell Russell, Kory Draughn, Justin James, Alan King, Tony Edgin (CyVerse), Illyoung Choi (CyVerse)

Minutes

DISCUSSION

• iRODS S3 API
  • Removed coroutines - merged
  • Multipart - demonstrated/tested
    • Not yet merged
    • Copy object uses 'multipart' internally
      • Not currently supported, needs more investigation
      • Workaround is to specify no-multipart for 'cp' operation
  • 0.2.0 coming soon
    • Justin in Geneva at CS3
  • Tony Testing Update
    • Ran openstack cluster testing harness
• Next Meeting
  • April 5, 2024, maybe

TESTING

Testing Harness now public

• https://github.com/cyverse/irods-s3-api-perf

1G multipart - aws 26s, mc 21s

• expected, we're storing and forward

128MB - aws 5s, mc 3.26s

AWS flag...

• aws/aws-cli#6130
• AWS_EC2_METADATA_DISABLED=true

v0.2.0

• S3 API 0.2.0 - now with multipart and --cli-read-timeout=300

1 kiB Transfer

Tool         | Download Duration (s) | Upload Duration (s)
-------------|-----------------------|--------------------
iRODS S3 API | 0.74 [0.68, 0.80]     | 0.69 [0.60, 0.78]
GoCommands   | 0.40 [0.38, 0.42]     | 0.43 [0.42, 0.43]
iCommands    | 0.22 [0.22, 0.22]     | 0.34 [0.27, 0.41]

1 GiB Transfer

Tool         | Download Duration (s) | Upload Duration (s)
-------------|-----------------------|--------------------
iRODS S3 API | 142 [142, 143]        | 180 [171, 191]
GoCommands   | 13 [ 13,  13]         |  14 [ 14,  16]
iCommands    | 11 [ 11,  12]         |  13 [ 12,  14]