Handling transfer acceleration discrepencies

[softloft38p-michael]

I have config in ~/.aws/config which specifies use_accelerate_endpoint = true, but the bucket I'm trying to connect to does not have transfer acceleration enabled so when I do an S3 request I get the error:

botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the GetBucketAccelerateConfiguration operation: S3 Transfer Acceleration is not configured on this bucket

I would like to handle this gracefully such that if the config is wrong I can catch it and just disable the acceleration in the client before doing anything else.

I have the below code to detect when the bucket does not have acceleration enabled:

def bucket_supports_accelerated_transfer(Bucket, client: BaseClient = None, **kwargs):
    if not Bucket:
        return
    c = client or make_s3_client(**kwargs)
    supports_acceleration = False
    try:
        response = c.get_bucket_accelerate_configuration(Bucket=Bucket) or {}
        supports_acceleration = response.get('Status', '') == "Enabled"
    except Exception:
        logger.exception("Couldn't get bucket acceleration configuration.")
    return supports_acceleration


def setup_client_acceleraetion(client, **kwargs):
    bucket = kwargs.pop('Bucket', None)
    if not client or not bucket:
        return
    logger.info("Client associated with bucket: %s" % (bucket))
    supports_acceleration = bucket_supports_accelerated_transfer(bucket, client=client, **kwargs)
    try:
        del client._client_config.s3['use_accelerate_endpoint']
    except Exception:
        logger.exception("Could not set transfer acceleration to: %s." % (supports_acceleration))
    return

The issue is that the client must have been created to make the request to check for acceleration support. Is there a way to alter the config of an existing client or an easy way to check if the config for the session asked for it, but the Bucket doesn't handle it so I can make a new session with that option disabled? Is this something that could be handled by botocore itself so that requests just gracefully degrade instead of failing?

[aaoswal]

Hi @softloft38p-michael,
Unfortunately in order to check for bucket acceleration configuration, a call would need to be made to S3. You could pass the config separately as well according to the acceleration state of the bucket. Hence, botocore has limited functionality in this case and would not be able to automatically degrade the config instead of failing.

Example:

client = boto3.client("s3", config(s3={"use_accelerate_endpoint": True})

Hope this helps!