Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrieve MD5 from object metadata #414

Merged
merged 2 commits into from
Jun 26, 2024
Merged

Retrieve MD5 from object metadata #414

merged 2 commits into from
Jun 26, 2024

Conversation

joneubank
Copy link
Contributor

For S3 downloads, when handling downloads in cases where the .meta file is not found, we need to retrieve the MD5 sum from the Object metadata. Our current implementation of this is reading the value of the eTag, but this is not guaranteed to contain an MD5 hash (and for multipart uploads is known to never contain the MD5 hash).

This change introduces a function to get the MD5 hash from the metadata. The function has two cases to check for where the MD5 hash might be.

  1. Priority is to use the built in S3 getContentMD5() method. If the hash is found here then this will be returned.
  2. Allow reading of a custom (user defined) metadata property. This allows a system administrator to assign the Base64 encoded MD5 hash in a custom location. This is given the default property name md5chksum which corresponds to the property name where the md5 is stored in Azure systems. This will match for cases where data is moved from Azure Blob storage into an S3 Object Storage system.

@joneubank joneubank requested a review from Azher2Ali June 26, 2024 19:58
Copy link
Contributor

@Azher2Ali Azher2Ali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@joneubank joneubank merged commit e5505bc into develop Jun 26, 2024
2 checks passed
@joneubank joneubank deleted the fix/s3-download-md5 branch June 26, 2024 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants