fix(core): Gcs's RangeWrite doesn't support concurrent write #4806
Conversation
Signed-off-by: Xuanwo <[email protected]>
| @@ -362,7 +362,7 @@ impl Access for GcsBackend { | |||
| // It's recommended that you use at least 8 MiB for the chunk size. | |||
| // | |||
| // Reference: [Perform resumable uploads](https://cloud.google.com/storage/docs/performing-resumable-uploads) | |||
| write_multi_align_size: Some(256 * 1024 * 1024), | |||
| write_multi_align_size: Some(8 * 1024 * 1024), | |||
There was a problem hiding this comment.
Should we keep it with no change?
There was a problem hiding this comment.
Should we keep it with no change?
No, the original value is incorrect (256MiB! instead of 256KiB) and has been hiding our CI failure for a long time. All our tests use write_once in wrong, which allows our CI to keep passing.
|
It seems we can use this API: https://cloud.google.com/storage/docs/multipart-uploads, which allows us to upload parts in parallel.
|
I tracked this at #4807, would you like to help implementing it? |
Yes, let me fix it🤩 |
Thanks a lot! |
We need to find a new way to implement concurrent write for gcs.