Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add s3 copy tools for s3 IO #103

Open
dazza-codes opened this issue Jun 28, 2023 · 1 comment
Open

Add s3 copy tools for s3 IO #103

dazza-codes opened this issue Jun 28, 2023 · 1 comment

Comments

@dazza-codes
Copy link
Owner

For example, a synchronous s3 copy can use

def copy_s3_object(src_s3_uri: str, dst_s3_uri: str, *args, s3_client: BaseClient = None, **kwargs) -> bool:
    """
    Copy s3 URI

    :param src_s3_uri: a fully qualified S3 URI for the s3 object to read
    :param dst_s3_uri: a fully qualified S3 URI for the s3 object to write
    :param s3_client: an optional botocore.client.BaseClient for s3
    :return: boolean (True on success, False on failure)
    """
    # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/copy_object.html
    # CopySource={‘Bucket’: ‘bucket’, ‘Key’: ‘key’, ‘VersionId’: ‘id’}
    # Note that the VersionId key is optional and may be omitted.
    if s3_client is None:
        s3_client = s3_io_client(*args, **kwargs)
    try:
        src_s3_uri = S3URI(src_s3_uri)
        dst_s3_uri = S3URI(dst_s3_uri)
        logger.info(f"Copy: {src_s3_uri} to {dst_s3_uri}")
        response = s3_client.copy_object(
            ACL='bucket-owner-full-control',
            Bucket=dst_s3_uri.bucket,
            Key=dst_s3_uri.key,
            CopySource={'Bucket': src_s3_uri.bucket, 'Key': src_s3_uri.key},
        )
        if response:
            return True
    except botocore.exceptions.ClientError as err:
        logger.error(f"Failed S3 Copy: {src_s3_uri} to {dst_s3_uri}")
        logger.error(err)
        return False

@dazza-codes
Copy link
Owner Author

For synchronous requests, this is fixed in

Leaving this issue open until the asyncio is implemented too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant