Refactor writeObject to only use MultipartUpload when required

[juliushaertl]

Ensure that multipart uploads are only used when needed. For cases where the file is smaller than 5242880. Since streams may not have a fixed size before reading them some contortions performed by @tsdicloud makes sure that we read that amount of bytes into a buffer to determine if a simple putObject can be used or if the multipart upload is triggered with an AppendStream of the buffer and the rest of the original one.

Furthermore there is additional cleanup added to properly remove leftovers from failed multipart upload attempts.

I pulled out the changes of the branch with the s3 encryption and added some further test cases.

[juliushaertl]

I'd vote for adding this to the baseline, since while the abstract declaration is considered protected the implementation is a public method.

[artonge]

Found it weird that they did not think of that, so I dug a little and:

can't we use that $e->getState()->getId(); ?

This is from https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s3-multipart-upload.html which actually recommend using ObjectUploader to get this kind of optimization. But maybe I am not looking at the right thing ^^

[juliushaertl]

Good catch, I'll check if that works as expected 👍

[blizzz]

🐘 tests 💪

[tsdicloud]

Thanks Julius, to take the merge work over for me. Proud that my code will make it into product.

[icewind1991]

What is the core problem being solved by not always using the multipart uploader?

I'm not a fan of having to buffer the first chunk for every upload

[tsdicloud]

Besides the fact that the AWS classes themselves always use ObjectUploader for Objects smaller 5 MB,
you have ti understand that the Multipart uploader mechanism is more complex behind the scenes than Objectuploader.
Because of the Multipart nature, it creates so called fragmments in the background. And having too many open fragments in parallel heavily degrades S3 upload performance. Knowing this, you should avoid Multipart uploads for as many objects as possible.
And, additionally, if a Multipart upload is not properly cleaned up on a fail, a funnel of stalled fragments does also slowly degrade the performance of your installation.

BTW: the buffer is only 5 MB and the code is an improved version of the original AWS code in the PHP library,

[icewind1991]

Fair enough, thanks for the explanation

[kesselb]

Nice work 👍

Super nice to use multipart less often. I've added some nitpicks mostly about phpdoc and possible type hints. But not a blocker for merge.

Could we use a higher value for single put operations? Like 25 or 50 MB. I understand the buffer uses the memory hence we should not pick a value to high. However most pictures have more than 5 MB nowadays.

[kesselb]

We append buffer and psrStream because the psrStream might be non-seekable?

[juliushaertl]

Yes

[juliushaertl]

Could we use a higher value for single put operations? Like 25 or 50 MB. I understand the buffer uses the memory hence we should not pick a value to high. However most pictures have more than 5 MB nowadays.

Will do a follow up PR for that to check about the memory usage in detail with different values.

Edit: draft in #28542

[juliushaertl]

All other addressed.