Skip to content

Commit

Permalink
[Internal] Msdata/Direct: Refactors msdata branch with latest v3 and …
Browse files Browse the repository at this point in the history
…direct release (#3726)

* [Internal] Client Telemetry: Fixes tests leaking environment variables (#3517)

* Adding log lines

* More logs

* Debugging further

* Removing other builds

* And more debugging

* Wrong build parameters

* Wrong category

* Removing noise

* Fixing test

* Adding utils

* Adding test with client telemetry enabled

* Fixing leak in Client Telemetry Tests

* Reenabling test

* cpu monitor initialization

* Adding name details

* Undo another file

* Updated change log and bumped up the version. (#3526)

* Query: Fixes performance regression on target partition on some ORDER BY queries with continuation (#3525)

* Revert performance regression caused by #1289

* Remove irrelevant comment

* Add a test for validating formatted filters for the target partition

* [Internal] AI Integration: Adds SubStatusCode Information in attributes (#3533)

* wip

* regenerated baseline

* add test

* fix test

Co-authored-by: Sourabh Jain <[email protected]>

* Diagnostics: Removes unused properties and reduces size (#3519)

* Remove starttime and id from diagnostics string

* Remove starttime and id from diagnostics string

* Update xml files for trace baseline tests

* Update xml files for trace baseline tests

* Update xml files for trace baseline tests

* Whitespaces update

* Changed starttime position based on code review

Co-authored-by: Matias Quaranta <[email protected]>

* [Internal] docs: Add address caches conceptual flow (#3534)

* Rough draft

* Some more changes

* Adding a new section

* Some more refinement

* Some changes 

Lets get the feedback

* Some MISC changes

* Release: Adds SDK version and changelog for 3.31.2 (#3546)

* Bumped SDK version and changelog

* Added contracts file for 3.31.2

* [Internal] Documentation: Removes invalid comment from ReadThroughputAsync (#3516)

* [Internal] nugetconfig: Removes specific overrides (#3551)

* Query: Fixes partition range evaluation for spatial queries (#3495)

* Initial commit

* Update.

* Pull/Rebase

* Addressed comments.

* Build fix

* [Internal] Documentation: Refactors cache content on its own document (#3554)

* Add cache file

* Moving content away

* [Internal] Emulator unit tests: Adds IdEncoding unit tests for ComputeGateway (#3556)

* Adding IdEncoding unit tests for ComputeGateway

* Reacting to code review feedback

* Updating md files

* Change Feed Processor: Adds support for Resource Tokens (#3566)

* Adding rid parsing

* Test

* Removing dead code

* Contract update

* [Internal] AI integration: Refactor code how container and database name is flowing to opentelemetry module (#3532)

* wip

* WIP

* Revert "WIP"

This reverts commit 71275de54b9e67fa54a37e79d450b9597e173934.

* Revert "wip"

This reverts commit 586fa9865cc3f40dabd7ef90fb3e0cf499a045bc.

* wip add containe and database info

* redesign how container and database name information flows into opne telemetry data

* test fix

* fix test

* fix tests

* fix typos

* baseline test fix

Co-authored-by: Sourabh Jain <[email protected]>

* [Internal] AI Integration: Adds a new flag in DistributedTracingOptions (#3562)

* add flag in diagnostic options

* test fixes

* rename variable

* test fix

* add validation

* fix baseline

* test fix

Co-authored-by: Sourabh Jain <[email protected]>

* Query: Fixes default to BadRequestException in case of internal errors in ServiceInterop (#3399)

* Don't default to BadRequestException in case of errors in ServiceInterop

* Incorporate code review feedback

* Fix build error

* fix up failing test

* [Internal] Query: Adds unit tests for Merge/Split implementation with OptimisticDirectExecution pipeline (#3510)

* Added tests to test different aspects of merge/split support with OptimisticDirectExecution pipeline. Tests check for gone exception handling, pipeline switching etc.

* Added gone exception simulation tests.

* Added new tests and improved test infra

* Removed ParalleContEvocation test. Fixed comments

* Removed CreateParallelCrossPartitionPipelineStateAsync() as it is not being used anymore

* Removed while loop in CreateDocumentContainerAsync()

* Fixed comments.

* Updated ExecuteGoneExceptionOnODEPipeline()

* Added type Assert for ExecuteGoneExceptionOnODEPipeline()

* Updated OptimisticDirectExecution pipeline abbreviation

* Updated TestBaseline folder with new xml

* Client Retry Policy: Adds HTTP timeouts with request-level cross-region retry (#3555)

* Fixes to ReadThroughputAsync for databases with no provisioned throughput and null as request options

* fixed failure to ReadReplaceThroughputResponseTests

* Added Stream Method

* Ran UpdateContract.ps1

* Encryption implemtation

* Fixed spelling error

* Update Microsoft.Azure.Cosmos.Encryption/src/EncryptionDatabase.cs

Co-authored-by: Matias Quaranta <[email protected]>

* Variable name change

* Update Microsoft.Azure.Cosmos/src/Resource/Database/Database.cs

Co-authored-by: Matias Quaranta <[email protected]>

* Suggested Changes and fixes

* Removed manufactured ResponseMessage + nits

* Simplified PR

* nits

* nits

* initial changes TODO: Update tests

* updated tests

* nits'

* Ran UpdateContracts.ps1

* nits + requested changes

* Delete NuGet.Config

* Update Microsoft.Azure.Cosmos/src/HttpClient/CosmosHttpClientCore.cs

Co-authored-by: Matias Quaranta <[email protected]>

* Update Microsoft.Azure.Cosmos/src/HttpClient/HttpTimeoutPolicy.cs

Co-authored-by: Matias Quaranta <[email protected]>

* Update Microsoft.Azure.Cosmos/src/HttpClient/HttpTimeoutPolicy.cs

Co-authored-by: Matias Quaranta <[email protected]>

* Update Microsoft.Azure.Cosmos/src/HttpClient/HttpTimeoutPolicy.cs

Co-authored-by: Matias Quaranta <[email protected]>

* Use Cosmos Exception Factory, Simplified Tests

* removed unused code

* nits: removed unused code

* removed unused code

Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: Nalu Tripician <[email protected]>

* Documentation: Fixes EUAP in Comments (#3579)

* Query: Fixes incorrect FeedResponse.Count when result contains undefined elements (#3574)

* Do not maintain an independent count on QueryResponse that can go out of sync

* Add more test coverage for QueryResponse<T>.Count

* Output the correct count from CosmosElementSerializer when the input contains CosmosUndefined

* Add untyped tests for CosmosUndefined

* Remove commented code

* removed allr eference (#3581)

Co-authored-by: Sourabh Jain <[email protected]>

* Trace: Fixes Tracing/diagnostics hour-times to 24Hours (#3577)

* Trace: Fixes Tracing/diagnostics hour-times to 24Hours

* fixing baseline tests

Co-authored-by: Matias Quaranta <[email protected]>

* AI Integration: Adds cosmetic fixes (#3576)

* wip

* status code int and internal and client kind activity

* remove unused imports

* update baselines

* fix test

* fixed baseline tests

* fix tests

* update base tetss

Co-authored-by: Sourabh Jain <[email protected]>

* Query: Adds ALL Scalar Expression (#3509)

* Add SqlAllScalarExpression to v3 DOM

* updated generated parser files

* Parsing for ALL

* Added tests for ALL and baselines

* Added more tests

* added new test, cleanup

* cleaning & fix typos

* fixed typo

* Added new baseline test file names to csproj file

* renamed AggregateAll to AggregateSubquery to accomodate FIRST and LAST later

* Added keywords for 'left' and 'right' and respective function calls

* fixed bug from last commit

* cleaning

* replace tabs with spaces

* cleaning

* Release: Adds API contracts for 3.31.2-preview (#3586)

* [Internal] sccignore: Adds a .sccignore file to apply an exception for artifacts configuration issues (#3589)

* [Internal] PermissionTests: Adds CosmosPermissionTests Coverage (#3593)

* Ensures that both Direct and Gateway connection modes are tested
* Validates that container read works with PermissionMode.Read (test was previously only validating that Delete was blocked - i.e. the negative case).

* [Internal] AI Integration: Refactors to Operation prefix and add tests (#3583)

* add tests for otel and custome listener

* clean up

* null pointer fix

* fix tets

* handle event generation also at operation level

* added documentation

* wip

* change event sourcename

* rename event Name

Co-authored-by: Sourabh Jain <[email protected]>

* [Internal] HttpTimeoutPolicy: Removes Data Plane Writes from being able to be retried (#3607)

* data plane writes no longer failover on timeout

* removed duplication of test\

* [Internal] Performance Testing: Adds Distributed Tracing option in benchmarks (#3611)

Co-authored-by: Sourabh Jain <[email protected]>

* [Internal] Benchmark: Refactors code to make Memory Stream capacity configurable (#3624)

Co-authored-by: Sourabh Jain <[email protected]>

* add new LatestVersion changefeed mode that has same behavior as Incremental; renamed FullFidelity to AllVersionsAndDeletes (#3596)

* Query: Fixes handling of CosmosUndefined, CosmosGuid and CosmosBinary in unordered DISTINCT (#3632)

* Handle CosmosUndefined, CosmosGuid and CosmosBinary in DistinctMap.UnorderedDistinctMap

* Address code review feedback and remove unnecessary allocations from DistinctQueryPipelineStageTests

* [Internal] Subpartitioning: Adds updates to test coverage for subpartitioning (#3618)

* updates to test coverage for subpartitioning

* bug fixes

* now useses Assert.ThrowsException

* Seperated into multiple tests for clarity

* Put MultiHash test into seperate test file

* nit

* [Internal] ContainerProperties: Fixes version reset when setting PartitionKeyPath (#3637)

* Remember previous value

* test

* [Internal] AI Integration: Adds CorrelationId and Activity Id Attributes for query operation (#3630)

* add activityid in Otel attributes

* added correlation id

* operation type fix

* remove test changes

* test fix

* fix baseline test

* rename correlationId

* fix tests again

* include only not null attributes in test

* fixed tests

* changefeedxml

* test fix

* ordering activity in operationname oerder

* fix test

* review comments

* refator header getter setter

* clean up

Co-authored-by: Sourabh Jain <[email protected]>

* Documentation: Fixes CosmosClientBuilder.WithConnectionModeGateway parameter description (#3643)

* Fixed CosmosClientBuilder.WithConnectionModeGateway documentation

* Update Microsoft.Azure.Cosmos/src/Fluent/CosmosClientBuilder.cs

Co-authored-by: Ruben Bartelink <[email protected]>

Co-authored-by: Augsten <[email protected]>
Co-authored-by: Ruben Bartelink <[email protected]>

* Upgrade Resiliency: Refactors Implementation for Opening Rntbd Connections to Backend Replicas in Direct Mode. (#3640)

* Code changes to refactor implementations for opening connections to all replicas. Fixed test failures due to Direct package upgrade.

* Code changes to add poland central region as a part of Regions.cs

* Code changes to update contract to reflect new regions.

* Revert "Code changes to update contract to reflect new regions."

This reverts commit f171b3c1c9889043556ddf96bcd33ccd79565ad9.

* Revert "Code changes to add poland central region as a part of Regions.cs"

This reverts commit 1aafbf18f6d80e9a92baa301b6b23cf065e4b155.

* Code changes to implement replica validation in dotnet v3 sdk.

* Cosmetic changes to add inline code comments.

* [Preview] AI integration: Adds IsDistributedTracingEnabled flag as public API to enable/disable this feature (#3598)

* make api public for preview

* add null check

* fix tests

* singleton listener initialization

* assign null to listeners

* fix test

* concurrent bag in listener

* renamed to LatencyThresholdForDiagnosticEvent

* renamed to IsDistributedTracingEnabled

* updated xml

* update contract

* made latency threshold flag internal

* fix test

* regeneratebaselines

* update documentation

* rename builder api

* add docs

* updated contracts and all

* doc update

* import cleanup

Co-authored-by: Sourabh Jain <[email protected]>

* Code chages to address review comments.

* Change Feed Processor: Fixes behavior with StartTime on Local (#3645)

* To UTC

* Test

* Code changes to cover a scenario for async cache.

* [Internal] Client Telemetry: Refactors code to use base useragent string (#3653)

* [Internal] AI Integration: Refactors code to rename event name (#3648)

* first draft

* rename event name

* updated xmls

* update files

* Code changes to refactor async non-blocking cache code.

* Region Availability: Adds Poland Central Region For Public Usage (#3656)

* Client Encryption: Adds validation code to check if the Key Vault URI provided in wrap metadata is a valid key identifier. (#3642)

* Check if the key vault uri provided is a valid Kid

* test fix.

* update changelog and build props

* Update Directory.Build.props

* Update Microsoft.Azure.Cosmos.Encryption.csproj

* Fixed preview version

* Refactor

* Update EncryptionDatabaseExtensions.cs

* [Internal] Query: Adds Split Support for Ode (#3572)

* Added tests to test different aspects of merge/split support with OptimisticDirectExecution pipeline. Tests check for gone exception handling, pipeline switching etc.

* Added gone exception simulation tests.

* Added new tests and improved test infra

* Removed ParalleContEvocation test. Fixed comments

* Removed CreateParallelCrossPartitionPipelineStateAsync() as it is not being used anymore

* Removed while loop in CreateDocumentContainerAsync()

* Fixed comments.

* Updated ExecuteGoneExceptionOnODEPipeline()

* Added type Assert for ExecuteGoneExceptionOnODEPipeline()

* Replaced try-catch with if statement in MoveNextAsync()

* Added delegate to access TryCreateCoreContextAsync()

* Added check to confirm Ode pipeline is not called in fallback plan

* Updated method name from OptimisticDirectExecutionContext() to TryCreateOptimisticDirectExecutionContext()

* Using delegate instead of Func<>.

* Ode fallback plan always calls Specialized pipeline

* Using ServiceInterop/Gateway to get QueryPlan for Specialized Pipeline

* Added new test to check handling of failing fallback pipeline

* Code cleanup

* Added logic for handling non ODE continuation tokens

* Moved delegate away from member variables

* Added tests for Merge case

* Updated method names

* Added checks for tryCatch

* Updated SetCancellationToken() to use Try

* Updated TryUnwrapContinuationToken()

* Removed changes in FlakyDocumentContainer.cs

* Removed unused imports

* Updated comments

* Fixed comments and cleaned up test code

* Added CosmosElement null check in TryUnwrapContinuationToken()

* Removed FlakyDocumentContainer.cs from pull request

* Removed unused imports

* Updated TryUnwrapContinuationToken()

* Update MoveNextAsync() call in OptimisticDirectExecutionQueryBaselineTests.cs

* Made MergeTestUtil.IsFailedFallbackPipelineTest a readonly property

* Added IsPartitionSplitException() overload to take CosmosElement

* Fixed bug regarding syntax error queries

* [Internal] AI Integration : Fixes operation type  for batch (#3660)

* fix op type

* fix conflict

* CosmosClientOptions: Adds ServerCertificateCustomValidationCallback for Http and TCP (#3636)

* Adding ServerCertificateCustomValidationCallback in clientoptions

* Adding Server callback for Http and fixing tests

* Fixing failing E2Etests

* Resolving merge conflicts

* Running update contracts script

* Running Update contracts script

* Running Update contracts script

* Reverting the v3 version change

* Update based on review comments

* Added unit tests

* Added remarks for callback delegate

* Ran update contracts script

* Update based on review comments

* Ran update contracts script

* Updated unit tests

* Making ssl validation function private

* Updating test files

* Update remarks for sslvalidation public contract

* Added emulator tests for server validation

Co-authored-by: Debdatta Kunda <[email protected]>

* Query: Adds EnableOptimisticDirectExecution flag to QueryRequestOptions (#3664)

* Added new flag to QueryRequestOptions to allow customers to use Ode pipeline

* Updated comments in QueryRequestOptions.cs

* Renamed enabledOde to enableOde

* Removed default setting for EnableOptimisticDirectExecution

* [Internal] Tests: Removes Direct/HTTPS emulator tests (#3679)

* Removing direct/https tests

* mppreference

* [Internal] Benchmark : Fixes issue with dependency on Cosmos Project (#3673)

* users/sourabhjain/benchmarkfix

* update pipeline

* Revert "users/sourabhjain/benchmarkfix"

This reverts commit 81b48f0.

* fix compilation error

* add parama for preview pkg also

---------

Co-authored-by: Matias Quaranta <[email protected]>

* LocalQuorum: Adds Quorum reads on Consistent Prefix Accounts (#3680)

Co-authored-by: DESKTOP-ED57J7H\Prashanth Venkataram <[email protected]>

* 3.32.0: Adds new SDK version and contract files (#3687)

* 3.32.0: Adds new SDK version and contract files

* 3.32.0: Adds new SDK version and contract files

* Updating changelog version

* Updating changelog version

* Added more commits to changelog and updated release contract

* Added documentation tags PR in changelog

* Updated changelog based on reviews

* Updated PR decsription in changelog

* Update changelog.md

Updated full fidelity change description

Co-authored-by: Matias Quaranta <[email protected]>

---------

Co-authored-by: Matias Quaranta <[email protected]>

* [Internal] Samples: Adds change feed pull model samples (#3646)

* add change feed pull samples

* refactor appsettings validation

* addressing pr comments

* move task delay

* update sample to use latest change feed mode names

---------

Co-authored-by: Matias Quaranta <[email protected]>

* [Internal] Tests: Refactors emulator CI (#3688)

* [Internal] GitHub Template: Adds needs-investigation label (#3708)

By default, all "Bug report" issues will have "needs-investigation"

* Adding fabric bot action (#3709)

* CosmosNullReferenceException: Refactors CosmosNullReferenceException to pass along InnerException property on parent NullReferenceException (#3713)

* Passed inner exception details to NullReferenceException ctor when instantiating CosmosNullReferenceException.

* Added unit tests.

* Addressed PR feedback.

* [Internal] PriorityRequests: Fixes header value (#3714)

Co-authored-by: Matias Quaranta <[email protected]>

* [Internal] Query: Adds single physical partition check for OptimisticDirectExecution queries (#3699)

* Added single physical partition check for Ode queries. Updated test infrastructure for Ode emulator tests too.

* Refactored emulator tests to have all the test cases at the top of the file

* Updated TryGetTargetRangeOptimisticDirectExecutionAsync()

* Uodated logic on how many times CreateIngestQueryDeleteAsync() gets called

* Added debug asserts for partitionKeyDefinition

* Added pageSizeOptions parameter in CreateInput() in EmulatorTests

* Fixed comments

* [Internal] CTL: Fixes Reservoir Sampling Logic (#3712)

* Code changes to fix the reservoir sampling logic in CTL

* Code changes to modify help text on reservoir type.

* Code changes to address minor code refactor.

* Code changes to port latest direct and v3 master changes.

---------

Co-authored-by: Matias Quaranta <[email protected]>
Co-authored-by: neildsh <[email protected]>
Co-authored-by: Sourabh Jain <[email protected]>
Co-authored-by: Sourabh Jain <[email protected]>
Co-authored-by: aavasthy <[email protected]>
Co-authored-by: Kiran Kumar Kolli <[email protected]>
Co-authored-by: Aditya <[email protected]>
Co-authored-by: Fabian Meiswinkel <[email protected]>
Co-authored-by: akotalwar <[email protected]>
Co-authored-by: Nalu Tripician <[email protected]>
Co-authored-by: Nalu Tripician <[email protected]>
Co-authored-by: Pramod Valavala <[email protected]>
Co-authored-by: Ezra Haleva <[email protected]>
Co-authored-by: Vivek Ravindran <[email protected]>
Co-authored-by: Prasad Ullal <[email protected]>
Co-authored-by: Philip Thomas <[email protected]>
Co-authored-by: Arthur Augsten <[email protected]>
Co-authored-by: Augsten <[email protected]>
Co-authored-by: Ruben Bartelink <[email protected]>
Co-authored-by: Santosh Kulkarni <[email protected]>
Co-authored-by: pravengithub <[email protected]>
Co-authored-by: DESKTOP-ED57J7H\Prashanth Venkataram <[email protected]>
Co-authored-by: Justine Cocchi <[email protected]>
Co-authored-by: Abhijeet Mohanty <[email protected]>
Co-authored-by: Achint-Agrawal <[email protected]>
  • Loading branch information
1 parent dcfbf16 commit 5addb1c
Show file tree
Hide file tree
Showing 361 changed files with 27,659 additions and 14,632 deletions.
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
labels: needs-investigation
assignees: ''

---
Expand Down
56 changes: 54 additions & 2 deletions .github/fabricbot.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
"capabilityId": "ScheduledSearch",
"subCapability": "ScheduledSearch",
"version": "1.1",
"id": "a-PNOJWK3Jbr7QY7vkc2O",
"config": {
"frequency": [
{
Expand Down Expand Up @@ -50,7 +49,60 @@
]
},
"disabled": true
},
{
"taskType": "trigger",
"capabilityId": "IssueResponder",
"subCapability": "IssuesOnlyResponder",
"version": "1.0",
"config": {
"conditions": {
"operator": "and",
"operands": [
{
"operator": "not",
"operands": [
{
"name": "isAssignedToSomeone",
"parameters": {}
}
]
},
{
"name": "isAction",
"parameters": {
"action": "opened"
}
},
{
"operator": "not",
"operands": [
{
"name": "hasLabel",
"parameters": {
"label": "needs-investigation"
}
}
]
}
]
},
"eventType": "issue",
"eventNames": [
"issues",
"project_card"
],
"taskName": "Add needs-investigation",
"actions": [
{
"name": "addLabel",
"parameters": {
"label": "needs-investigation"
}
}
]
}
}
],
"userGroups": []
}
}
3 changes: 3 additions & 0 deletions .sscignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"cfs" : ["CFS0011"]
}
6 changes: 6 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,12 @@ Or all through `Re-run failed checks` on the top right corner:

![Re-run all failures](docs/images/contributing-rerunall.png)

## Usage of Cosmos DB Emulator for running unit tests

- - The same version of the emulator used in the CI pipelines can be downloaded [here](https://aka.ms/cosmosdb-emulator)
- More information about how ro use the emulator for development is documented [here](https://github.com/Azure/azure-documentdb-dotnet/blob/master/docs/documentdb-nosql-local-emulator.md)
- You can start the emulator with same parameters as the emulator unit tests in the CI pipeline via `.\CosmosDB.Emulator.exe /DisableRateLimiting /PartitionCount=100 /Consistency=Strong /enableRio /EnablePreview /EnableAadAuthentication /EnableSqlComputeEndpoint`

## Troubleshooting

- [General .NET SDK Troubleshooting](https://docs.microsoft.com/azure/cosmos-db/sql/troubleshoot-dot-net-sdk)
Expand Down
11 changes: 5 additions & 6 deletions Directory.Build.props
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<ClientOfficialVersion>3.31.0</ClientOfficialVersion>
<ClientPreviewVersion>3.31.0</ClientPreviewVersion>
<ClientOfficialVersion>3.32.0</ClientOfficialVersion>
<ClientPreviewVersion>3.32.0</ClientPreviewVersion>
<ClientPreviewSuffixVersion>preview</ClientPreviewSuffixVersion>
<DirectVersion>3.29.2</DirectVersion>
<EncryptionOfficialVersion>2.0.0</EncryptionOfficialVersion>
<EncryptionPreviewVersion>2.0.0</EncryptionPreviewVersion>
<DirectVersion>3.30.2</DirectVersion>
<EncryptionOfficialVersion>2.0.1</EncryptionOfficialVersion>
<EncryptionPreviewVersion>2.0.1</EncryptionPreviewVersion>
<EncryptionPreviewSuffixVersion>preview</EncryptionPreviewSuffixVersion>
<CustomEncryptionVersion>1.0.0-preview04</CustomEncryptionVersion>
<HybridRowVersion>1.1.0-preview3</HybridRowVersion>
Expand Down
10 changes: 10 additions & 0 deletions Microsoft.Azure.Cosmos.Encryption/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@ Preview features are treated as a separate branch and will not be included in th
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### <a name="2.0.1"/> [2.0.1](https://www.nuget.org/packages/Microsoft.Azure.Cosmos.Encryption/2.0.1) - 2023-03-11

#### Added
- [#3642](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/3642) Adds validation code to check if the Key Vault URI provided in wrap metadata is a valid key identifier.

### <a name="2.0.1-preview"/> [2.0.1-preview](https://www.nuget.org/packages/Microsoft.Azure.Cosmos.Encryption/2.0.1-preview) - 2023-01-11

#### Added
- [#3642](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/3642) Adds validation code to check if the Key Vault URI provided in wrap metadata is a valid key identifier.

### <a name="2.0.0"/> [2.0.0](https://www.nuget.org/packages/Microsoft.Azure.Cosmos.Encryption/2.0.0) - 2022-06-28

#### Added
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,17 @@ public static async Task<ClientEncryptionKeyResponse> CreateClientEncryptionKeyA
+ " Please refer to https://aka.ms/CosmosClientEncryption for more details.");
}

if (string.Equals(encryptionCosmosClient.KeyEncryptionKeyResolverName, KeyEncryptionKeyResolverName.AzureKeyVault))
{
// https://KEYVAULTNAME.vault.azure.net/keys/KEYNAME/KEYVERSION
string[] keyVaultUriSegments = new Uri(encryptionKeyWrapMetadata.Value).Segments;

if (keyVaultUriSegments.Length != 4 || !string.Equals(keyVaultUriSegments[1], "keys/", StringComparison.InvariantCultureIgnoreCase))
{
throw new ArgumentException($"Invalid Key Vault URI'{encryptionKeyWrapMetadata.Value}' passed. Pass the complete Azure keyvault key identifier. Please refer to https://aka.ms/CosmosClientEncryption for more details.");
}
}

KeyEncryptionKey keyEncryptionKey = KeyEncryptionKey.GetOrCreate(
encryptionKeyWrapMetadata.Name,
encryptionKeyWrapMetadata.Value,
Expand Down Expand Up @@ -192,6 +203,17 @@ public static async Task<ClientEncryptionKeyResponse> RewrapClientEncryptionKeyA
+ " Please refer to https://aka.ms/CosmosClientEncryption for more details.");
}

if (string.Equals(encryptionCosmosClient.KeyEncryptionKeyResolverName, KeyEncryptionKeyResolverName.AzureKeyVault))
{
// https://KEYVAULTNAME.vault.azure.net/keys/KEYNAME/KEYVERSION
string[] keyVaultUriSegments = new Uri(newEncryptionKeyWrapMetadata.Value).Segments;

if (keyVaultUriSegments.Length != 4 || !string.Equals(keyVaultUriSegments[1], "keys/", StringComparison.InvariantCultureIgnoreCase))
{
throw new ArgumentException($"Invalid Key Vault URI'{newEncryptionKeyWrapMetadata.Value}' passed. Pass the complete Azure keyvault key identifier. Please refer to https://aka.ms/CosmosClientEncryption for more details.");
}
}

ClientEncryptionKeyProperties clientEncryptionKeyProperties = await clientEncryptionKey.ReadAsync(cancellationToken: cancellationToken);

RequestOptions requestOptions = new RequestOptions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@
</ItemGroup>

<ItemGroup Condition=" '$(SdkProjectRef)' != 'True' AND '$(IsPreview)' != 'True' ">
<PackageReference Include="Microsoft.Azure.Cosmos" Version="[3.28.0,3.29)" />
<PackageReference Include="Microsoft.Azure.Cosmos" Version="[3.31.0,3.32)" />
</ItemGroup>

<ItemGroup Condition=" '$(SdkProjectRef)' != 'True' AND '$(IsPreview)' == 'True' ">
<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.28.0-preview" />
<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.31.2-preview" />
</ItemGroup>

<ItemGroup Condition=" '$(SdkProjectRef)' == 'True' ">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,69 @@ await MdeEncryptionTests.CreateClientEncryptionKeyAsync(
if (ex is CosmosException cosmosException)
Assert.AreEqual(HttpStatusCode.Conflict, cosmosException.StatusCode);
}

cekId = "testAkvKid";
CosmosClient client = TestCommon.CreateCosmosClient();
TestKeyEncryptionKeyResolver testKeyEncryptionKeyResolver = new TestKeyEncryptionKeyResolver();

EncryptionKeyWrapMetadata metadata = MdeEncryptionTests.CreateEncryptionKeyWrapMetadata(KeyEncryptionKeyResolverName.AzureKeyVault, "key1", "https://testkeyvault.vault.azure.net/keys/testkey/12345678");

CosmosClient encryptionCosmosClient = client.WithEncryption(
testKeyEncryptionKeyResolver,
KeyEncryptionKeyResolverName.AzureKeyVault,
TimeSpan.Zero);

Database database = await encryptionCosmosClient.CreateDatabaseAsync(Guid.NewGuid().ToString());

ClientEncryptionKeyResponse clientEncrytionKeyResponse = await database.CreateClientEncryptionKeyAsync(
cekId,
DataEncryptionAlgorithm.AeadAes256CbcHmacSha256,
metadata);

Assert.AreEqual(HttpStatusCode.Created, clientEncrytionKeyResponse.StatusCode);

metadata = MdeEncryptionTests.CreateEncryptionKeyWrapMetadata(KeyEncryptionKeyResolverName.AzureKeyVault, "key1", "https://testkeyvault.vault.azure.net/keys/testkey/9101112");

clientEncrytionKeyResponse = await database.RewrapClientEncryptionKeyAsync(
cekId,
metadata);

Assert.AreEqual(HttpStatusCode.OK, clientEncrytionKeyResponse.StatusCode);

// complete key identifier not passed
metadata = MdeEncryptionTests.CreateEncryptionKeyWrapMetadata(KeyEncryptionKeyResolverName.AzureKeyVault, "key1", "https://testkeyvault.vault.azure.net/keys/testkey");

try
{
clientEncrytionKeyResponse = await database.CreateClientEncryptionKeyAsync(
cekId,
DataEncryptionAlgorithm.AeadAes256CbcHmacSha256,
metadata);

Assert.Fail("Key creation should have failed.");

}
catch(Exception ex)
{
Assert.AreEqual(true, ex.Message.Contains("Invalid Key Vault URI"));
}

// rewrap old key with new key vault uri without complete key identifier
try
{
clientEncrytionKeyResponse = await database.RewrapClientEncryptionKeyAsync(
cekId,
metadata);

Assert.Fail("Key rewrap should have failed.");

}
catch (Exception ex)
{
Assert.AreEqual(true, ex.Message.Contains("Invalid Key Vault URI"));
}

encryptionCosmosClient.Dispose();
}

[TestMethod]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,9 @@ public class BenchmarkConfig
[Option(Required = false, HelpText = "Enable Client Telemetry")]
public bool EnableTelemetry { get; set; }

[Option(Required = false, HelpText = "Enable Distributed Tracing")]
public bool EnableDistributedTracing { get; set; }

[Option(Required = false, HelpText = "Client Telemetry Schedule in Seconds")]
public int TelemetryScheduleInSec { get; set; }

Expand Down Expand Up @@ -140,7 +143,9 @@ internal void Print()
Utility.TeeTraceInformation($"{nameof(BenchmarkConfig)} arguments");
Utility.TeeTraceInformation($"IsServerGC: {GCSettings.IsServerGC}");
Utility.TeeTraceInformation("--------------------------------------------------------------------- ");
Utility.TeeTraceInformation(JsonHelper.ToString(this));
Utility.TeeTraceInformation(JsonHelper.ToString(
input: this,
capacity: 2048));
Utility.TeeTraceInformation("--------------------------------------------------------------------- ");
Utility.TeeTraceInformation(string.Empty);
}
Expand Down Expand Up @@ -219,6 +224,8 @@ internal Microsoft.Azure.Cosmos.CosmosClient CreateCosmosClient(string accountKe
clientOptions.ConsistencyLevel = (Microsoft.Azure.Cosmos.ConsistencyLevel)Enum.Parse(typeof(Microsoft.Azure.Cosmos.ConsistencyLevel), this.ConsistencyLevel, ignoreCase: true);
}

clientOptions.IsDistributedTracingEnabled = this.EnableDistributedTracing;

return new Microsoft.Azure.Cosmos.CosmosClient(
this.EndPoint,
accountKey,
Expand Down
12 changes: 6 additions & 6 deletions Microsoft.Azure.Cosmos.Samples/Tools/Benchmark/JsonHelper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ internal static class JsonHelper
});
private const int DefaultCapacity = 1024;

public static string ToString<T>(T input)
public static string ToString<T>(T input, int capacity = JsonHelper.DefaultCapacity)
{
using (MemoryStream stream = JsonHelper.ToStream(input))
using (MemoryStream stream = JsonHelper.ToStream(input, capacity))
using (StreamReader sr = new StreamReader(stream))
{
return sr.ReadToEnd();
Expand All @@ -31,15 +31,15 @@ public static T Deserialize<T>(string payload)
return JsonConvert.DeserializeObject<T>(payload);
}

public static MemoryStream ToStream<T>(T input)
public static MemoryStream ToStream<T>(T input, int capacity = JsonHelper.DefaultCapacity)
{
byte[] blob = System.Buffers.ArrayPool<byte>.Shared.Rent(JsonHelper.DefaultCapacity);
MemoryStream memStreamPayload = new MemoryStream(blob, 0, JsonHelper.DefaultCapacity, writable: true, publiclyVisible: true);
byte[] blob = System.Buffers.ArrayPool<byte>.Shared.Rent(capacity);
MemoryStream memStreamPayload = new MemoryStream(blob, 0, capacity, writable: true, publiclyVisible: true);
memStreamPayload.SetLength(0);
memStreamPayload.Position = 0;
using (StreamWriter streamWriter = new StreamWriter(memStreamPayload,
encoding: JsonHelper.DefaultEncoding,
bufferSize: JsonHelper.DefaultCapacity,
bufferSize: capacity,
leaveOpen: true))
{
using (JsonWriter writer = new JsonTextWriter(streamWriter))
Expand Down
4 changes: 2 additions & 2 deletions Microsoft.Azure.Cosmos.Samples/Tools/Benchmark/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@ sleep 10 #Wait
dotnet run -c Release -- -n 2000000 -w ReadStreamExistsV3 --WorkloadName ReadStreamExistsV3WithTelemetry --enableTelemetry --telemetryScheduleInSec 60 --telemetryEndpoint $TELEMETRY_ENDPOINT --tcp 10 --pl $PL -e $ACCOUNT_ENDPOINT -k $ACCOUNT_KEY --enablelatencypercentiles --disablecoresdklogging --publishresults --resultspartitionkeyvalue $RESULTS_PK --commitid $COMMIT_ID --commitdate $COMMIT_DATE --committime $COMMIT_TIME --branchname $BRANCH_NAME --database testdb --container testcol --partitionkeypath /pk
sleep 10 #Wait

# Open telemetry enabled ReadStreamExistsV3. This is needed to see the impact of open telemetry.
dotnet run -c Release -- -n 2000000 -w ReadStreamExistsV3 --WorkloadName ReadStreamExistsV3WithOpenTelemetry --enableOpenTelemetry --tcp 10 --pl $PL -e $ACCOUNT_ENDPOINT -k $ACCOUNT_KEY --enablelatencypercentiles --disablecoresdklogging --publishresults --resultspartitionkeyvalue $RESULTS_PK --commitid $COMMIT_ID --commitdate $COMMIT_DATE --committime $COMMIT_TIME --branchname $BRANCH_NAME --database testdb --container testcol --partitionkeypath /pk
# Open telemetry enabled ReadStreamExistsV3. This is needed to see the impact of distributed tracing (without listener)
dotnet run -c Release -- -n 2000000 -w ReadStreamExistsV3 --WorkloadName ReadStreamExistsV3WithDistributedTracingWOListener --enableDistributedTracing --tcp 10 --pl $PL -e $ACCOUNT_ENDPOINT -k $ACCOUNT_KEY --enablelatencypercentiles --disablecoresdklogging --publishresults --resultspartitionkeyvalue $RESULTS_PK --commitid $COMMIT_ID --commitdate $COMMIT_DATE --committime $COMMIT_TIME --branchname $BRANCH_NAME --database testdb --container testcol --partitionkeypath /pk
sleep 10 #Wait

#Point read operations
Expand Down
9 changes: 8 additions & 1 deletion Microsoft.Azure.Cosmos.Samples/Tools/CTL/CTLConfig.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ namespace CosmosCTL
using CommandLine.Text;
using Microsoft.Azure.Cosmos;
using Newtonsoft.Json;
using static CosmosCTL.ReservoirProvider;

public class CTLConfig
{
Expand Down Expand Up @@ -76,7 +77,7 @@ public string DiagnosticsThresholdDuration
}

[Option("ctl_content_response_on_write", Required = false, HelpText = "Should return content response on writes")]
public bool IsContentResponseOnWriteEnabled { get; set; } = true;
public bool? IsContentResponseOnWriteEnabled { get; set; } = true;

[Option("ctl_output_event_traces", Required = false, HelpText = "Outputs TraceSource to console")]
public bool OutputEventTraces { get; set; } = false;
Expand All @@ -102,6 +103,12 @@ public string DiagnosticsThresholdDuration
[Option("ctl_telemetry_schedule_in_sec", Required = false, HelpText = "telemetry task schedule time in sec")]
public string TelemetryScheduleInSeconds { get; set; }

[Option("ctl_reservoir_type", Required = false, HelpText = "Defines the reservoir type. Valid values are: Uniform, SlidingWindow and ExponentialDecay. The default value is SlidingWindow.")]
public ReservoirTypes ReservoirType { get; set; } = ReservoirTypes.SlidingWindow;

[Option("ctl_reservoir_sample_size", Required = false, HelpText = "The reservoir sample size.")]
public int ReservoirSampleSize { get; set; } = 1028;

internal TimeSpan RunningTimeDurationAsTimespan { get; private set; } = TimeSpan.FromHours(10);
internal TimeSpan DiagnosticsThresholdDurationAsTimespan { get; private set; } = TimeSpan.FromSeconds(60);

Expand Down
Loading

0 comments on commit 5addb1c

Please sign in to comment.