Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

objtools: fix 5GiB+ server-side copies on S3 #8427

Closed
wants to merge 2 commits into from

Conversation

andyasp
Copy link
Contributor

@andyasp andyasp commented Jun 19, 2024

What this PR does

S3 only allows CopyObject to be used on objects <= 5GiB in size (docs). This limitation was encountered when trying to use undelete-blocks on a large block (I anonymized object information):

2024/06/18 12:14:08 ERROR failed to undelete block tenant=(tenant) block=(block) err="The specified copy source is larger than the maximum allowable size for a copy source: 5368709120\nfailed to restore object (tenant)/(block)/index with version (version)

This fix works around this by first checking the object to be copied with a HEAD request to find out its size. If it is small enough to be copied with CopyObject it is. Otherwise if it is too large then a multipart upload is used where each part is an increasing byte range of the object. Minio unfortunately adds an unnecessary HEAD request for each part, but there's no way to work around that while using their client.

Opening as a draft until this is manually tested.

Which issue(s) this PR fixes or relates to

Fixes https://github.com/grafana/mimir-squad/issues/2163

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@andyasp andyasp added the bug Something isn't working label Jun 19, 2024
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you

@dimitarvdimitrov
Copy link
Contributor

The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG document. Thanks!

@andyasp
Copy link
Contributor Author

andyasp commented Sep 20, 2024

I finally got around to testing this and opened #9357 as a replacement.

@andyasp andyasp closed this Sep 20, 2024
@andyasp andyasp deleted the aasp/objtools-s3-copy-fix branch September 20, 2024 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants