-
Notifications
You must be signed in to change notification settings - Fork 595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slow gcloud storage upload for small files? #1078
Comments
Can you run another upload immediately after the first one completes and measure the time difference? |
Sure. It seems subsequent calls are faster. Here's the same upload repeated 5x in a row (~10kB with resume and validation off): 4935ms, 3336ms, 2336ms, 2551ms, 2695ms. Still seems on the slow side to me - what do you think? Maybe my expectations are wrong though. I suppose there are overheads like auth etc, which disadvantage small files. |
This definitely seems wrong to me. Uploading 10k shouldn't be taking 1000 |
@jgeewax - Right, that was my feeling too. Does anyone know if there's some kind of debug mode, which might give some visibility into what's happening, time-wise? |
With my own tests, I complete uploads consistently under 1s. It went up to around 3 if I uploaded to a bucket overseas. Any chance location could be playing a role? |
That's a good point. If you're in the US and your bucket is in Asia (or
vice versa), that would explain the extra latency (mainly wire time)
|
Thanks, I tried both US and EU storage locations, but times seemed similar (I'm in EU). |
@mziccard (who is also in EU and may be able to help us with some debugging
:P)
Can you do a traceroute to be sure that your upload request is actually
being routed to endpoints in the EU?
|
If you have time gsutil cp 10kfile.txt gs://your-bucket |
Here's the test I've used: https://gist.github.com/stephenplusplus/15d0d7e96a40887c74aa |
@jgeewax , @stephenplusplus - Thanks, I'll test traceroute, gsutil and that test script and report back. Worth noting that I'm doing this from an external machine, not google-compute, and am specifying credentials like so (which may be a factor):
|
The credentials will cause a delay, but will only affect the first request. Have you had a chance to run any more tests? @mziccard any chance you can run the sample here and report your findings? https://gist.github.com/stephenplusplus/15d0d7e96a40887c74aa |
@bgSosh I'm going to close this out as this is likely a geographic issue, beyond what this library can help with. If you have a chance to run some more tests or find anything we might not have thought of that would help, please re-open. |
Hello guys my uploads are taking too long for small files of 500kbs upto 98 seconds how do I sort this issue please |
@Evrine19 please open a new issue and follow the issue template. That will help us get up to speed and see if we can reproduce the issue you're having. Thanks! |
Co-authored-by: meredithslota <[email protected]>
I'm seeing upload times of over 6000ms for small 10kB files. Please could someone confirm if this is expected for Google cloud uploads, or is it more likely there is some client problem happening?
I've tried disabling resumable and validation, and this improves things a bit, but not substantially. Our upload code looks like this:
The text was updated successfully, but these errors were encountered: