UploadStreamToBlockBlob: only require body be io.Reader #15958
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution serprex! We will review the pull request and get back to you soon. |
ghost
added
the
customer-reported
mohsha-msft
requested review from
ArcturusZhang,
ausfeldt,
benbp,
chlowell,
ealsur,
lirenhe,
richardpark-msft,
RickWinter and
a team
as code owners
mohsha-msft
suggested changes
Nov 3, 2021
There was a problem hiding this comment.
Track2 (New SDK) was introduced
- To provide more stable SDKs to avoid (or minimize) breaking changes for the end-user.
- To introduce big design changes and decouple components such as logging, authentication, etc.
We've kept the similar architecture and use case across all new SDKs. So, I'll have to decline to this PR. Can you tell me why it should be changed?
| @@ -33,7 +33,7 @@ type blockWriter interface { | |||
| // well, 4 MiB or 8 MiB, and auto-scale to as many goroutines within the memory limit. This gives a single dial to tweak and we can | |||
| // choose a max value for the memory setting based on internal transfers within Azure (which will give us the maximum throughput model). | |||
| // We can even provide a utility to dial this number in for customer networks to optimize their copies. | |||
| func copyFromReader(ctx context.Context, from io.ReadSeekCloser, to blockWriter, o UploadStreamToBlockBlobOptions) (BlockBlobCommitBlockListResponse, error) { | |||
| func copyFromReader(ctx context.Context, from io.Reader, to blockWriter, o UploadStreamToBlockBlobOptions) (BlockBlobCommitBlockListResponse, error) { | |||
There was a problem hiding this comment.
We're moving to use io.ReadSeekCloser across all the SDKs. You can write simple struct like here
There was a problem hiding this comment.
Right. I'd implemented that in my PR to workaround for now, but then if this interface does change to use these unused methods that'll introduce a bug rather than a compile time error. Unless your api holds itself to not break callers who rely on the implicit contract that you won't call close/seek, in which case you don't have the flexibility to make use of ReadSeekCloser interface in the future anyways so you're stuck to io.Reader implicitly. It also makes the functions more confusing to use because then I can't tell from the function signature whether the reader will be kept open or not
But alright, if there's some design ideology at play I'll have to accept it
There was a problem hiding this comment.
For context, we require a seekable stream to facilitate our retry logic. If the PUT request fails, we can rewind the stream to try again.
@mohsha-msft I'm confused in this case. The API takes a ReadSeekCloser but doesn't actually require a Seeker or a Closer. Is it used elsewhere (maybe via a type-assertion)?
There was a problem hiding this comment.
@serprex sorry, just to clarify, this is a private method right? How are you leveraging it exactly?
There was a problem hiding this comment.
It's called from BlockBlobClient.UploadStreamToBlockBlob() which also takes a ReadSeekCloser
There was a problem hiding this comment.
I think this requirement is hard to meet in services.
Often I stream data right from a client through my service to blob storage. The HTTP connection I am feeding from can't be a Seeker. If azblob wants a retry mechanism, I think it should implement this internally with some kind of (configurable?) buffer. The burden should not be on the caller.
There was a problem hiding this comment.
@zezha-msft I thought we had an API that requires just an io.Reader for exactly this reason and does what's described above?
This function only takes an io.Reader in azure-storage-blob-go, & wal-g abstracts over multiple upload mechanisms which all operate on io.Reader
serprex
force-pushed
the
upload-stream-to-block-blob-takes-io-reader
branch
from
cbdf650
to
ba8db25
Compare
|
Running CI. |
mohsha-msft
approved these changes
Dec 16, 2021
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Dec 30, 2021
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary There are two other PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Dec 30, 2021
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary There are two other PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Dec 30, 2021
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary There are two other PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Dec 30, 2021
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary There are two other PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
jhendrixMSFT
added a commit
that referenced
this pull request
Jan 13, 2022
* Make azblob.BlobClient.downloadBlobToWriterAt public (#15899) * azblob.downloadBlobToWriterAt: initialDownloadResponse: remove as unused * azblob.DownloadBlobToWriterAt: make public * Replace the advancer (#16354) * Handle error response XML with whitespace (#16639) * [chunkwriting] Return original buffer to the pool (#16067) When the buffer isn't filled to capacity a new slice is created that's smaller than the original. In that case the smaller slice is returned to the pool which prevents the rest of the capacity from being used again. The solution is to pass the original slice through and attach the length. This allows the original slice to be returned to the pool once the operation is complete. This change also simplifies the `sendChunk` method, ensuring that the buffer is returned to the TransferManager even when no bytes were read from the reader. * UploadStreamToBlockBlob: only require body be io.Reader (#15958) This function only takes an io.Reader in azure-storage-blob-go, & wal-g abstracts over multiple upload mechanisms which all operate on io.Reader * uncomment o.Progress(progress) in getDownloadBlobOptions() (#16727) * Updated azblob README sample code (#16721) * Updated README sample code container name should be all lowercase * Fix typo GetAccountSASToken to GetSASToken I believe 'GetAccountSASToken' was written by mistake. It's inconsistent with what is described on line 97 and I don't see this method in the documentation. * [KeyVault] updating examples to use ClientSecretCredential, fix parsing keyID (#16689) * updating examples to use ClientSecretCredential, fix parsing keyID * fixing constructor test URL * adding test for no keyid * key version, not key ID, better error reporting * changes from working with maor and daniel * working for both hsm and non hsm, need to fix up for recorded tests * fixed implementation, i think... * working for hsm too * working challenge policy for hsm and non hsm * new recordings * adding all final recordings * reveerting back to DefaultAzCred * using streaming package from azcore * Add spell check warnings (#16656) * Add spell check warnings * Basic cspell.json * Ignore files in .vscode except cspell.json, new line before EOF in cspell.json * Spell check, ignore thyself * Adding Smoketests to nightly runs (#16226) * Adding Smoketests to nightly runs * updating location, fixes to script * starting go script * finishing script * updating yml file * formatting * adding snippet for finding go code * adding funcitonality for copying examples * trimming out unused funcs * fixed regexp, thanks benbp * fixing smoke test program to create go.mod file correctly, update powershell for nightly * removing need for argument in go program, updating yml and powershell to reflect * scripts not common * smoketests, plural not singular * finally got the right directory * fixed script locally, running into permissions issue on ci * updating script to exit properly, logging an error instead of panicing * manually set go111module to on * removing references to go111module * issue with duplicated function names... * updating to only pull examples from the service directory if one is provided * runs samples now too! * adding 'go run .' step to ps1, triggering for tables * adding step to analyze.yml file * adding debugging for ci * updating to work in ci * updating to specify go module name, removing print statements * updating scripts to fmt for prettier printing, find all environment variables * working on loading environment variables from file * removing env vars from example_test.go for testing * adding the environment variable portion to the generated main.go file * forgot to remove change to nightly script * adding import to the main file * cleaning up code, adding comments * don't import os if no env vars * small changes for checking all packages * removing _test suffix on copied files * converting to use cobra for better support * formatting * Sync eng/common directory with azure-sdk-tools for PR 2464 (#16730) * Support AAD graph and Microsoft Graph service principal APIs * Consolidate service principal wrapper creation Co-authored-by: Ben Broderick Phillips <[email protected]> * Make EndpointToScope resilient to subdomains (#16737) * Add user assigned managed identity example to docs (#16738) * [KeyVault] fixing broken live test (#16752) * fixing broken live test * coverage * forcing ci * Sync eng/common directory with azure-sdk-tools for PR 2484 (#16753) * Add weekly pipeline generation to prepare-pipelines template * Add succeeded condition to pipeline generation pipelines Co-authored-by: Ben Broderick Phillips <[email protected]> * Release v61.1.0 1641448664 (#16762) * Generated from specification/automation/resource-manager/readme.md tag package-2020-01-13-preview (commit hash: 3b9b0a930e29cbead33df69ae46c7080408e4c0f) * Generated from specification/compute/resource-manager/readme.md tag package-2021-08-01 (commit hash: 3b9b0a930e29cbead33df69ae46c7080408e4c0f) * v61.1.0 * Handle skipping docker build when PushImages is set and there is no dockerfile (#16555) Co-authored-by: Ben Broderick Phillips <[email protected]> * add new config (#16774) * Add offline test for Azure Arc managed identity (#16771) * Rename armmanagedapplications ci.yml files to avoid pipeline name definition collisions (#16770) * Refactor IMDS discovery to remove probing, stop caching failures (#16267) * Sync eng/common directory with azure-sdk-tools for PR 2500 (#16779) * Update pipeline generator tool feed to azure-sdk-for-net * Update pipeline generator tool version Co-authored-by: Ben Broderick Phillips <[email protected]> * feat: add generator cmd to generate or update readme.go.md file for track2 sdk param (#16780) * [azservicebus] Updating to handle breaking changes in azcore (#16776) Updating sb to handle breaking changes in azcore At the moment we're not using azcore's HTTP library, so we don't need to test for specific errors that come from azcore. * [Core] Bump version of internal (#16793) <!-- Thank you for contributing to the Azure SDK for Go. Please verify the following before submitting your PR, thank you! --> - [ ] The purpose of this PR is explained in this or a referenced issue. - [ ] The PR does not update generated files. - These files are managed by the codegen framework at [Azure/autorest.go][]. - [ ] Tests are included and/or updated for code changes. - [ ] Updates to [CHANGELOG.md][] are included. - [ ] MIT license headers are included in each file. [Azure/autorest.go]: https://github.com/Azure/autorest.go [CHANGELOG.md]: https://github.com/Azure/azure-sdk-for-go/blob/main/CHANGELOG.md * [Core] Update Changelog.md (#16794) Update changelog for release * Update CHANGELOG.md (#16795) * Increment version for azcore releases (#16798) Increment package version after release of azcore * [azidentity] Making ChainedTokenCredential re-use the first successful credential (#16392) This PR makes it so instances of `ChainedTokenCredential` will now re-use the first successful credential on `GetToken` calls. Fixed #16268 * Align authentication errors with azcore.ResponseError (#16777) * Increment version for azidentity releases (#16799) Increment package version after release of azidentity * [KeyVault] release ready for keyvault keys (#16731) * release ready for keyvault keys * updating the api surface to latest azcore * updating to ResponseError * update with latest codegen * fixing ci * formatting * bumping azcore to v0.21.0 * updating azidentity and autorest version * updating go.sum * final upgrade for changelog Co-authored-by: Joel Hendrix <[email protected]> * [Tables] preparing tables for release (#16733) * preparing tables for release * prepping for release with latest azcore * update with latest code generator * formatting * updating to released azcore version * upgrading azidentity * updating autorest.go version * final changes to readme Co-authored-by: Joel Hendrix <[email protected]> * [KeyVault] prepare secrets for release (#16732) * prepare secrets for release * updating to latest azcore * update with latest codegen * fixing secret version issues * formatting * manual checkout of azkeys * updating to azcore v0.21.0 * updating autorest.go and azidentity * updating Changelog * updating changelog * updating changelog * modifying moduleVersion * undoing changes to keys and tables Co-authored-by: Joel Hendrix <[email protected]> * chore: pump codegen version in scripts (#16802) * Update azblob with the latest azcore (#16784) * Update azblob with the latest azcore This tactially resolves the small number of breaking changes in azcore. * clean-up Co-authored-by: Benoit Perrot <[email protected]> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: adreed-msft <[email protected]> Co-authored-by: John Stairs <[email protected]> Co-authored-by: Philip Dubé <[email protected]> Co-authored-by: Brandon Kurtz <[email protected]> Co-authored-by: Kim Ying <[email protected]> Co-authored-by: Sean Kane <[email protected]> Co-authored-by: Daniel Jurek <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: Ben Broderick Phillips <[email protected]> Co-authored-by: Charles Lowell <[email protected]> Co-authored-by: Jiahui Peng <[email protected]> Co-authored-by: Dapeng Zhang <[email protected]> Co-authored-by: Chenjie Shi <[email protected]> Co-authored-by: Richard Park <[email protected]> Co-authored-by: Daniel Rodríguez <[email protected]>
serprex
pushed a commit
to citusdata/wal-g
that referenced
this pull request
Feb 5, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary There are two other PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to citusdata/wal-g
that referenced
this pull request
Feb 8, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary which turns out to be a broken fix, so a fork is used in the meantime which includes a proper fix: Azure/azure-sdk-for-go#16992 On some containers paging happens much sooner than MaxResults would imply, causing restoring LATEST to either fail because wal-g can't find backups, or wal-g selects an old backup There are two other notable PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Feb 8, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary which turns out to be a broken fix, so a fork is used in the meantime which includes a proper fix: Azure/azure-sdk-for-go#16992 On some containers paging happens much sooner than MaxResults would imply, causing restoring LATEST to either fail because wal-g can't find backups, or wal-g selects an old backup There are two other notable PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Feb 8, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary which turns out to be a broken fix, so a fork is used in the meantime which includes a proper fix: Azure/azure-sdk-for-go#16992 On some containers paging happens much sooner than MaxResults would imply, causing restoring LATEST to either fail because wal-g can't find backups, or wal-g selects an old backup There are two other notable PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Feb 14, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary which turns out to be a broken fix, so a fork is used in the meantime which includes a proper fix: Azure/azure-sdk-for-go#16992 On some containers paging happens much sooner than MaxResults would imply, causing restoring LATEST to either fail because wal-g can't find backups, or wal-g selects an old backup There are two other notable PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Feb 14, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary which turns out to be a broken fix, so a fork is used in the meantime which includes a proper fix: Azure/azure-sdk-for-go#16992 On some containers paging happens much sooner than MaxResults would imply, causing restoring LATEST to either fail because wal-g can't find backups, or wal-g selects an old backup There are two other notable PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Feb 14, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary which turns out to be a broken fix, so a fork is used in the meantime which includes a proper fix: Azure/azure-sdk-for-go#16992 On some containers paging happens much sooner than MaxResults would imply, causing restoring LATEST to either fail because wal-g can't find backups, or wal-g selects an old backup There are two other notable PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
serprex
pushed a commit
to wal-g/wal-g
that referenced
this pull request
Feb 15, 2022
Specific commit used because I'm not sure when a new version will be made, but there's a particularly important fix: Azure/azure-sdk-for-go#16354 fixes ListBlobsHierarchy when paging is necessary which turns out to be a broken fix, so a fork is used in the meantime which includes a proper fix: Azure/azure-sdk-for-go#16992 On some containers paging happens much sooner than MaxResults would imply, causing restoring LATEST to either fail because wal-g can't find backups, or wal-g selects an old backup There are two other notable PRs: Azure/azure-sdk-for-go#16067 fixes a chunk pooling bug Azure/azure-sdk-for-go#15958 removes the need for FakeIoReadCloser
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This function only takes an io.Reader in azure-storage-blob-go,
& wal-g abstracts over multiple upload mechanisms which all operate on io.Reader