Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase chunk_size max to 256 Mb instead of 64 Mb #23489

Closed
wants to merge 2 commits into from
Closed

Increase chunk_size max to 256 Mb instead of 64 Mb #23489

wants to merge 2 commits into from

Conversation

nmjacob
Copy link

@nmjacob nmjacob commented Mar 6, 2017

Increasing the max size of the chunk from 64mb to 256mb.

Closes #23446.

Increase chunk_size max to 256 Mb instead of 64 Mb
Isue=23446 Increase Azure max chunk size.
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@nmjacob
Copy link
Author

nmjacob commented Mar 6, 2017

This is my first time doing a public repo pr, please let me know if I need to correct anything.

@dadoonet dadoonet changed the title Increase chunk_size max to 256 Mb instead of 64 Mb #23446 Increase chunk_size max to 256 Mb instead of 64 Mb Mar 17, 2017
@dadoonet dadoonet self-requested a review March 17, 2017 13:24
@dadoonet dadoonet self-assigned this Mar 17, 2017
@rjernst
Copy link
Member

rjernst commented Jun 9, 2017

@nmjacob Since the chunk size is already configurable, I'm not sure why a change to the default is needed. Can you explain why you think 256mb is a better default for most users? @dadoonet thoughts?

@dadoonet
Copy link
Member

Can you explain why you think 256mb is a better default for most users?

@rjernst If you want to send to Azure repo a 1Gb file, with the current default, it will require 16 HTTP requests. With that change we can do it with 4 HTTP requests.
I expect it to be a bit faster. Just a guess though. @abeyad WDYT?

@abeyad
Copy link

abeyad commented Jul 5, 2017

Reading the documentation, it appears that 256mb would be the optimal chunk size corresponding to an Azure blob block. We haven't tested it so its an unknown, but it makes sense to go with the documented recommendation from Azure. I do think we should run tests on this against an actual Azure storage account to see if it makes a difference.

@rjernst
Copy link
Member

rjernst commented Jul 5, 2017

If it is the recommended then +1 to change (with testing as @abeyad mentions). One thing to double check with that testing is how it performs with default settings (ie from a laptop as a new user would try it out).

@clintongormley clintongormley added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Repository Azure labels Feb 14, 2018
@ywelsch
Copy link
Contributor

ywelsch commented Jul 31, 2018

superseded by #32101

@ywelsch ywelsch closed this Jul 31, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants