[Cosmos] Bulk/Batch APIs with v1/v2 hashing in JS #10168
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
added
the
abhijitpai
reviewed
Jul 23, 2020
abhijitpai
reviewed
Jul 23, 2020
abhijitpai
reviewed
Jul 23, 2020
abhijitpai
reviewed
Jul 23, 2020
| @@ -54,6 +54,14 @@ export class ChangeFeedResponse<T> { | |||
| export class ClientContext { | |||
| constructor(cosmosClientOptions: CosmosClientOptions, globalEndpointManager: GlobalEndpointManager); | |||
| // (undocumented) | |||
| bulk<T>({ body, path, resourceId, partitionKeyRange, options }: { | |||
There was a problem hiding this comment.
partitionKeyRangeId for consistency
abhijitpai
reviewed
Jul 23, 2020
| @@ -759,6 +770,11 @@ export class ItemResponse<T extends ItemDefinition> extends ResourceResponse<T & | |||
| // @public | |||
| export class Items { | |||
| constructor(container: Container, clientContext: ClientContext); | |||
| // Warning: (ae-forgotten-export) The symbol "Operation" needs to be exported by the entry point index.d.ts | |||
abhijitpai
reviewed
Jul 24, 2020
| .map(async (batch: Batch) => { | ||
| try { | ||
| const response = await this.clientContext.bulk({ | ||
| body: batch.operations, |
There was a problem hiding this comment.
Each service request can take 100 operations and be up to 2 MB (recommend ~200 KB unless a single doc is bigger which which case fit only one operation into the request).
southpolesteve
suggested changes
Jul 24, 2020
| }; | ||
|
|
||
| request.headers = await this.buildHeaders(request); | ||
| request.headers[Constants.HttpHeaders.IsBatchRequest] = "True"; |
There was a problem hiding this comment.
Suggested change
| request.headers[Constants.HttpHeaders.IsBatchRequest] = "True"; | |
| request.headers[Constants.HttpHeaders.IsBatchRequest] = true; |
| import { doubleToByteArrayJSBI, writeNumberForBinaryEncodingJSBI } from "./encoding/number"; | ||
| import { writeStringForBinaryEncoding } from "./encoding/string"; | ||
| import { BytePrefix } from "./encoding/prefix"; | ||
| // tslint:disable-next-line |
There was a problem hiding this comment.
Why is this needed?
| @@ -0,0 +1,51 @@ | |||
| import { doubleToByteArrayJSBI } from "./encoding/number"; | |||
| import { BytePrefix } from "./encoding/prefix"; | |||
| // tslint:disable-next-line | |||
There was a problem hiding this comment.
Same. Why is this needed?
There was a problem hiding this comment.
i had requires in there originally. just leftover, i'll remove
|
@zfoster Can you also add a change log entry? |
sdk/cosmosdb/cosmos/CHANGELOG.md
Outdated
| }, | ||
| { | ||
| operationType: "Upsert", | ||
| resourceBody: { id: addEntropy("doc2"), name: "other", key: "A" } |
There was a problem hiding this comment.
Suggested change
| resourceBody: { id: addEntropy("doc2"), name: "other", key: "A" } | |
| resourceBody: { id: "doc2", name: "other", key: "A" } |
Co-authored-by: Steve Faulkner <[email protected]>
sdk/cosmosdb/cosmos/CHANGELOG.md
Outdated
| { | ||
| operationType: "Read", | ||
| id: "readItemId", | ||
| partitionKey: `["key"]` |
There was a problem hiding this comment.
Can we "headerify" the partition key for people? IE, let them supply "key"
southpolesteve
approved these changes
Jul 27, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds v1 and v2 partitionKey hashing on the client. See: #9821 for more details as the PR has more thorough notes.
This PR more generally adds support for the bulk/batch API in Cosmos. From a user perspective, this involves passing a list of operations to the
container.items.bulk()method. Here's an example of what those operations looks like and the following bulk call: https://github.com/Azure/azure-sdk-for-js/compare/master...zfoster:zf/bulk?expand=1#diff-536cb613fdfa2c9c4361ce96f07b223bR330Currently, this PR only supports bulk operations on items with partitionKey value types
string,number,{},boolean, ornull. To represent undefined, we can use{}. It supports v1 and v2 containers.