Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

only do a multipart s3 copy when above the regular copy limit #41914

Merged
merged 2 commits into from
Dec 29, 2023

Conversation

icewind1991
Copy link
Member

Multipart copy seems problematic with some backends so avoid it when possible.

@icewind1991 icewind1991 added the 3. to review Waiting for reviews label Nov 29, 2023
@icewind1991 icewind1991 added this to the Nextcloud 29 milestone Nov 29, 2023
@icewind1991 icewind1991 requested review from juliushaertl, a team, ArtificialOwl, sorbaugh and come-nc and removed request for a team November 29, 2023 16:33
@icewind1991
Copy link
Member Author

/backport to stable28

@icewind1991
Copy link
Member Author

/backport to stable27

Copy link
Member

@solracsf solracsf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't test if MultiPart is supported? Something like:

if ($this->getObjectStore() instanceof IObjectStoreMultiPartUpload) {
   // multipart
} else {
  // not multipart
}

Copy link
Member

@juliushaertl juliushaertl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems good, do you know what s3 providers/solutions enforce such a limit?

@juliushaertl
Copy link
Member

Why don't test if MultiPart is supported?

It might need a different interface, I'm not sure if all providers that support multipart upload also support multipart copy

@icewind1991
Copy link
Member Author

do you know what s3 providers/solutions enforce such a limit?

No, but there are at least 2 different cases where the multipart copy is failing

Why don't test if MultiPart is supported? Something like:

I'm not aware of any good way to tell if an S3 implementation supports it

@Ziyann
Copy link

Ziyann commented Dec 13, 2023

For reference, I'm also affected by this issue, using Google Cloud Storage (but haven't tested this patch yet). Some built-in solution would be greatly appreciated, either a configurable limit like copySizeLimit here (which we could set to a very high value, thus never triggering multipart copy), or even better, some enable/disable switches for the various operations, because right now I'm manually reverting #40577 on production after upgrades.
There are many S3 providers with different features supported, so I think it would be great to have as many configuration options as possible (avoiding manual file changes on production).

@Ziyann
Copy link

Ziyann commented Dec 19, 2023

You should propagate copySizeLimit to the S3 SDK (mup_threshold), otherwise, it will use the hardcoded 5 GB threshold at some places: https://github.com/nextcloud/3rdparty/blob/v28.0.0/aws/aws-sdk-php/src/S3/ObjectCopier.php#L96-L103
See: #42068 (comment)

@juliushaertl
Copy link
Member

Pushed an additional commit to be able to fully turn off multipart copy for s3 providers not supporting it at all.

@juliushaertl juliushaertl added the pending documentation This pull request needs an associated documentation update label Dec 28, 2023
@juliushaertl juliushaertl merged commit 77cec80 into master Dec 29, 2023
61 of 62 checks passed
@juliushaertl juliushaertl deleted the s3-copy-size-limit branch December 29, 2023 10:28
@blizzz blizzz mentioned this pull request Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review Waiting for reviews pending documentation This pull request needs an associated documentation update
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants