…andler for .NET 6 and above (#3762)

* initial commit

* removed unneeded usings

* added validation callback, still needs tests

* nits + fixes

* added additional test

* test change

* removed unneeded Dispose calls

* removed unnneed dispose calls

* requested changes

* added pooledConnectionLifetime as client option

* nit

Co-authored-by: Kevin Pilch <[email protected]>

* Update Microsoft.Azure.Cosmos/src/HttpClient/CosmosHttpClientCore.cs

Co-authored-by: Matias Quaranta <[email protected]>

* Update Microsoft.Azure.Cosmos/src/CosmosClientOptions.cs

Co-authored-by: Matias Quaranta <[email protected]>

* suggested changes

* remove test, reorder usings

* updated contracts

* removed all non XXXAPI.json changes from UpdateContracts run

* removed public surface, added random timespan

* removed change from unrelated file

* Update Microsoft.Azure.Cosmos/src/HttpClient/CosmosHttpClientCore.cs

Co-authored-by: Matias Quaranta <[email protected]>

* added thread safe random method

* nit

* fixed merge mistake

* added reflection failsafe/error tracing

* nits

* added back removed if

* fixed formatting

* changed random method, fixed serverCertificateCustomValidation

---------

Co-authored-by: Kevin Pilch <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>

…3805)

* Fixing test

* New baseline

* Undo some changes

…ata processing task in background. (#3783)

* first draft

* remove failure count test

* refactporing

* code refactor

* create task with timeout

* fix test

* code refactoring

* fix timeout code

* space fix

* not failing if processor is taking time

* fix procrsser test

* code refactor

* refactor and test fix

* Basic changes to introduce move operator

* Added "from" object in patch spec operation.
Added testcase block.

* Fixed testcase.

* Changes made to address comments'

* Added comment regarding enum mutations

* Formatted comment

Co-authored-by: Matias Quaranta <[email protected]>

* Moved summary location.

* Ran UpdateContracts.ps1

---------

Co-authored-by: Amaan Haque <[email protected]>
Co-authored-by: Amaan Haque <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>

* Support cleaning

* wire previous content delete

* as text

* with variable

* another test

* param with types

* as string

* no delete

* no quotes

* undoing

* re-adding quotes

* testing empty

* trying another test

* readding version

* fixing publishing artifacts

* fixing parameter

* Fixing official pipeline

* version 5

* fixing main pipeline

* test with true

* using start time

* nightly preview

* passing parameters to pack

* Fixing nuget version

* arguments on the nuget pack

* folder structure

* testing v5

* Using only content

* Removing currentDate

…tic files (#3797)

* Direct Package update and replacing dagnostic files

* Resolve merge conflicts

* Running updateCOnstracts script

* Removed LinqTranslationWithCustomSerializerBaseline file

* Adding isDistributedTracingEnabled flag

* Running update contracts

* Running update contracts

* Running update contracts

* fix test

* Code cleanup for test fix

* Code cleanup for test fix

* Making regex expression readable

* Adding comment for regex expression

---------

Co-authored-by: Sourabh Jain <[email protected]>

* Add config changes

* Polishing automerge config

…cas along with `Unhealthy`. (#3820)

* Code changes to add aggressive validation logic.

* Code changes to enable aggressive validation for all regions.

* Code changes to pull in msdata cosmos.direct changes related to aggresive validation logic.

* Code changes to make minor cosmetic changes.

* Code changes to address review comments.

…ode when EnableContentResponseOnWrite is false (#3814)

* Do not call serializer if ResponseMessage.Content is empty.

* Add unit test

* Update unit tests

* Remove unused usings

---------

Co-authored-by: Matias Quaranta <[email protected]>

…#3822)

* Add doc

* Move benchmark

* Fixing texts

…Microsoft.Azure.Cosmos.Encryption.Custom (#3799)

* cosmos version change

* changing preview to release

* resolving code review comments

---------

Co-authored-by: Santosh Kulkarni <[email protected]>

* release 3.30.0

* added changelog

* updated changelog

* updated changelog

* suggested change to changelog

* updated changelog

---------

Co-authored-by: Matias Quaranta <[email protected]>

* Code changes to add msdata/direct sync-up documentation.

* Code changes to address review comments.

* Code changes to address further review comments.

* Code changes to address minor review comments.

* Removed internal links.

* add trim support

* Added some test coverage

* address reviews

---------

Co-authored-by: Minh Le <[email protected]>

…3832)

* add fix

* Add cultural info to test to verify correct behavior

* address pr review to restore to restore culture

* fix comment

---------

Co-authored-by: Minh Le <[email protected]>

…and FetchDataEncryptionKeyAsync to get DEK without and with raw key respectively. (#3809)

* added raw key to MdeEncryption

* adding ray key to Mde Algo

* test case changes

* resolving code review comments

* code optimization to reduce keyvault calls

* removed Microsoft.Data.Encryption.Cryptography nuget package

* added api for dek with raw key

* resolved code review comments

* adding change log

* code review changes

* Initial commit

* Restored settings.json changes.

* Update

* Addressed comments; still need to be tested using Emulator.

* Fixes after test run.

* Ignored the computed property tests based on the sync this morning (to allow for preview release).

* Suite0 fixes.

* Test update.

* Suite0 fixes

…es (#3818)

* add opentelemetry and application insights samples

* address pr comments

…nToSQL.cs (#3722)

* Ensure enum as string is preserved for custom serializer

* Failing test

* Added failing tests

* Updated requested names

* Ignore result of test for now

* Added additional comment on why the test is ignored

* Replaced with sample code

* Remove ignore attribute from tests, documented misbehavior for future use

* Updated comment

---------

Co-authored-by: leminh98 <[email protected]>

…tensions (#3724)

* Added the remaining Cosmos Type checking functions to the CosmosLinqExtensions

* Added comments requested

* Updated comment

* Updated baseline

* Improve readability of dictionary initialization

* Aligned with code style guide

* Revert change to baseline

* Executed update baseline script

---------

Co-authored-by: neildsh <[email protected]>
Co-authored-by: leminh98 <[email protected]>

… since PackageLicenseUrl is deprecated (#3847)

* proposal to add PackageLicenseFile since PackageLicenseUrl is deprecated. NuGet/Home#4628

* adding attribute Visible=false

…ests. (#3845)

* first draft

* second draft

* 3rd draft

* remove untouched file

* test fix

* fix order

* change order

* refactor

* skip network activities in test

* remove network attributes

* SDK 3.34.0: Adds version bump and changelog

* adding changelog changes

* added a missing PREVIEW PR

* Update changelog.md

Co-authored-by: Justine Cocchi <[email protected]>

* Update changelog.md

Co-authored-by: Kiran Kumar Kolli <[email protected]>

* removed 3840 as it was not committed

* change text for 3832

* fix merge issue

* add 3724

* Update changelog.md

Co-authored-by: Matias Quaranta <[email protected]>

* Update changelog.md

Co-authored-by: Matias Quaranta <[email protected]>

* including 3845

---------

Co-authored-by: Justine Cocchi <[email protected]>
Co-authored-by: Kiran Kumar Kolli <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>

…3864)

…ch (#3860)

* Linking limit documentation to Container.CreateTransactionalBatch(PartitionKey) method

* Resolved PR comments

* Links update

* Using learn.microsoft instead of docs.microsoft in the links

---------

Co-authored-by: Matias Quaranta <[email protected]>

* initial commit

* cleanup

* update test output

* cleanup

* typo

* Pr comments

* first draft

* redesign

* ädd link

* updated observability url

* initial commit DONT REVIEW

* fixes and documentation

* Apply suggestions from code review

Co-authored-by: Matias Quaranta <[email protected]>

* requested changes

* Apply suggestions from code review

Co-authored-by: Matias Quaranta <[email protected]>

* name changes

* readme changes

* nits + changing case of parameters file

---------

Co-authored-by: Matias Quaranta <[email protected]>

URL typo.

… latest direct (#3877)

* upgrade to 3.31.0

* add more regions

* enable dt for operations

* updated contract file

…tewayRequestOptions (#3836)

* Integrated cache: Add BypassIntegratedCache to DedicatedGatewayRequestOptions

Currently, integrated cache is used by default for Dedicated Gateway.
Customers cannot skip cache for particular requests or data unless they
shift to multi-tenant Gateway,which will lose the benefits of Dedicated
Gateway.

For customers to have more control over integrated cache, we're
introducing a new "RequestOption" called "BypassIntegratedCache". This
option will allow the customer to decide whether to use integrated cache
for each request or not. If this value is set to true, the item/query
will be served from backend and won't be cached in Dedicated Gateway.

* Move this feature to public preview

* Address comments

1. Add more tests
2. Add more detail and example code for BypassIntegratedCache

* Revert changes in EncryptionCustomAPI

---------

Co-authored-by: Jiajun Peng <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>

…on 3.34.0 (#3874)

* chaging Microsoft.Azure.Cosmos support version

* resolved merge conflicts

…#3866)

* Added tracing when converting HTTP Timeout to 503

* Fixed tracing when converting HTTP Timeout to 503

* Resolved PR comments

* Using ITrace as part of ClientSideRequestStatisticsTraceDatum

* Refactoring

* Test update

* Unit tests fix

* first draft

* add filter

* revert csproj

* fix sample

* changed log message

* remove unused library

…on headers (#3882)

* Adding ODE and RequiresDistribution Headers

* Fixed comments

* Updated parameter in SwitchToFallbackPipelineAsync

* Renamed TryUnwrapContinuationToken to UnwrapContinuationToken

---------

Co-authored-by: neildsh <[email protected]>

#3883)

* Made EnableOptimisticDirectExecution a public flag

* Updated contract

…nality (#3801)

* Direct Package update and replacing dagnostic files

* Resolve merge conflicts

* Running updateCOnstracts script

* Add code changes for distributed tracing open telemetry changes

* Add distributed tracing tests

* Updated tests for distributed tracing

* Addin traceID for diagnostics

* Running update contract script

* Removed LinqTranslationWithCustomSerializerBaseline file

* Adding isDistributedTracingEnabled flag

* Running update contracts

* Running update contracts

* Updates based on differnt code review comments

* Running update contracts

* Running update contracts

* Running update contracts

* fix test

* Code cleanup for test fix

* Running Update contracts

* resolving merge conflicts

* resolving merge conflicts

* Set EnableDistributedTracing to true for performance tests

* Benchmark project change for distributed tracing

* Updating tests

* Updated unit tests

* Updated unit tests

* Updated tests and constructors based on review comments

* Updated scope name in recorder

* Updated distributedOtel tests to cover more scenarios

* Updated distributedOtel tests

* Reverting benchmark performance test changes

* Update DistributedOpentelemetry tests

* Update test cleanup

* Update distributed tests with custom builder

* Update distributed open telemetry tests

* Update contracts

* Cleanup files

* Update distributed Otel tests

* Update distributed Otel tests

* code refactoring

* fix custom listener

* Update direct package to 3.31.1

* Code clean up

* Update tests with display name

---------

Co-authored-by: Sourabh Jain <[email protected]>

* CosmosClient documentation improvements

* Cref fix

* Link fix

* Documentation fix

* Typo fix

---------

Co-authored-by: Matias Quaranta <[email protected]>

…sts trace (#3887)

* enable request level in end to end

* made some changes

* fix tests

* fix display name

* hardcoded containername and databasenam

* fix tests

* temp

* fix tests

* update contracts

* fix tests

* fixed display name

)

* sdk design for client telemetry

* Otel design

* update optel design

* added more nformation

* updated ct design

* remove otel design

* Client Telemetry Design

* update typos

* fix typos

* fix typos

* added limitation

* updated docs

* updated doc

* updated text

* Update docs/observability.md

Co-authored-by: Kiran Kumar Kolli <[email protected]>

* Update docs/observability.md

Co-authored-by: Kiran Kumar Kolli <[email protected]>

* Update docs/observability.md

Co-authored-by: Kiran Kumar Kolli <[email protected]>

* Update docs/observability.md

Co-authored-by: Kiran Kumar Kolli <[email protected]>

* Update docs/observability.md

Co-authored-by: Kiran Kumar Kolli <[email protected]>

* move stuff here and there.

---------

Co-authored-by: Kiran Kumar Kolli <[email protected]>

…t 2 (#3903)

* updated doc

* Update docs/observability.md

Co-authored-by: Justine Cocchi <[email protected]>

* updated text

---------

Co-authored-by: Justine Cocchi <[email protected]>

… 6 Seconds. (#3902)

* Code changes to reduce default request timeout to 6 seconds.

* Code changes to update API doc default request timeout to 6 seconds.

…#3835)

* Code changes to add replica health status in diagnostics.

* Code changes to fix performance test build failure.

* Code changes to add health state capture logic in address cache.

* Code changes to fix benchmark test execution.

* Code changes to add tests to validate health state cache.

* Code changes to reduce default request timeout to 5 seconds.

* Revert "Code changes to reduce default request timeout to 5 seconds."

This reverts commit 139f37e588fc9dfed608431f4186c567a080e622.

* Initial Commit DO NOT REVIEW

* bug fix, needs Direct Package Changes

* fix for change feed and query plus tests

* clean up

* query + clean up

---------

Co-authored-by: Kiran Kumar Kolli <[email protected]>

* inital commit

* cleanup

* test cleanup

* PR comments

* PR comment

… default in Preview mode (#3909)

* Setting EnableODE to true by default in Preview Mode.

* Added seperate if block for default value

* Updated property

* Removed unused Usings

* Updated contracts

* Updated test

* Updated directive indentation

* Added links to PatchItems docs

* Undo changes from internal file

…Direct` Package to `3.31.2` (#3918)

* Code changes to bump up the direct version.

* Code changes to mark the Israel Central region as public.

* Code changes to update contracts.

* Code changes to fix test failure. Some clean ups.

* Code changes to add detailed message for open channels count.

* release PR

* updated changelog.md

* more changelog updates

* release PR

* updated changelog.md

* more changelog updates

* changelog fix

* Update changelog.md

* Update changelog.md

* Binary Serilaization Response test

* Added new header SupportedSerializationFormats

* Modified existing use of CosmosSerializationFormatOptions

* Modified tests and removed unused code

* Addressed comments

* Added more negative cases

* Revert changes

* Added spaces

* Addressed comments

* Addressed comments

* Removed SupportedSerializationFormats from Headers file

* Removed unused JsonSerilazationFormats option

* Addressed comments

* Addressed comments

* Addressed comments

* Addressed comments

* Added new enum TransportSerializationFormat

* Added new enum TransportSerializationFormat

* Addressed comments

* Removed unused parameter

* Addressed comments

* updating API

* remove tests

* Text fixes

* fix typo

* remove TransportSerializationFormat header

* text reverts

* revert

* test update

* PR comments

* remove test owner headers HeadersValidationTests.cs

* PR comments - remove unsupported tests and scope client

---------

Co-authored-by: Heet <[email protected]>
Co-authored-by: neildsh <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>

…ections in parallel. (#3939)

…3834)

* Support newtonsoft member access via ExtensionData

* Return null instead of empty string

* Added tests for select & where

* Updated baseline with note

---------

Co-authored-by: leminh98 <[email protected]>

… retries (#3944)

* Fix HttpTimeoutPolicies to not accidentally suppress retries

* Removing HttpTimeoutPolicy.MaxRetryTimeLimit altogether

* version bump

* changelog

* contract

* SDK 3.35.1: Adds version bump and changelog

* Update changelog.md

* Update changelog.md

* Update changelog.md

* Update changelog.md

* Update changelog.md

…ic.Core (#3957)

* Removing 1

* Removing 2

* Removing 3

… Feature for Preview (#3951)

* Code changes to add replica validation feature in cosmos client options.

* Code changes to upgrade the cosmos direct version to 3.31.3.

* Adding emulator test to cover replica validation.

* Code changes to address cosmetic clean ups.

* Code changes to address review comments. Fixed preview build failures.

* Code changes to enable replica validation for preview package by default.

* Code changes to address review comments.

* Code changes to fix preview unit tests.

* Code changes to disable environment variable at the end of the test.

…ion 3.35.1-preview (#3956)

* changing cosmos preview version

* updating build file

---------

Co-authored-by: Matias Quaranta <[email protected]>

…cBot decommissioning (#3966)

* Add prIssueManagement.yml to onboard repo to GitOps.ResourceManagement as FabricBot replacement

Owners of the FabricBot configuration should have received email notification. The same information contained in the email is published internally at: https://aka.ms/gim/fabricbot. Details on the replacement service and the syntax of the new yaml configuration file is available publicly at: https://microsoft.github.io/GitOps/policies/resource-management.html

Please review and merge this PR to complete the process of onboarding to the new service.

* Deleting fabricbot.json

---------

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>

…misticDirectExecution is set to true in 3.35.0-preview package (#3955)

* Updated emulator and baseline tests to not fail when ODE is set to default true in PREVIEW mode

* Fixed QueryAsync() test

* Fixed QueryAsync() in EndToEndTraceBaselineTests

* Undid changes to IndexMetrics baseline file

* Updated EndToEndTraceWriterBaselineTests.QueryAsync xml

* Updated xml

* Updated xml to have request options tag

…#3969)

…3917)

* Fixed malformed continuation token issue where Exception was not CosmosExceptionan and did not have the correct Status and Sub Status codes.

* Fixed incorrect indentation

* Added type check for incoming exception

* Replaced if/else with extra catch block

* Moved fix to a higher point in the call stack

* Removed unused Usings

* Updated test code

---------

Co-authored-by: Matias Quaranta <[email protected]>

…ation Feature Through `CosmosClientOptions` And Environment Variable (#3974)

* Code changes to use client options to enable or disable replica validation.

* Code changes to fix preview build failures.

#3668)

* string.Compare supported with LINQ to SQL

* Update tests

* Update test name

* Update tests

* Add test

* Create helper ReverseExpressionTypeForStrings

* PR feedback

* Update tests

* Update base line

---------

Co-authored-by: Aditya <[email protected]>
Co-authored-by: Matias Quaranta <[email protected]>

* first draft

* fix code

* included feedback

* flip condition

* updated docs

* Update docs/observability.md

Co-authored-by: Matias Quaranta <[email protected]>

* Update observability.md

* updated contract

---------

Co-authored-by: Matias Quaranta <[email protected]>

* Refactoring: base classes for operations.

* Updating comments.

* Adding new line at the end of the file.

* Fixing code review points.

* Restore PrepareAsync to be virtual.

* Updated change log and bumped up the version.

* Changing the version to 3.35.2

* Code changes to address review comments.

* Code changes to make minor fixes.

* Code changes to move some fixes into preview.

* Draft of local-quorum documentation

* Adding experimental to header

* Adding cross-region read guarantees

* Reads Bounded clarification

* Adding account consistency step also

* Non-Prod usage note at top

* Addressing review comments

* Some more refinement

…archical partitioning) (#3979)

* Hirarchical pk bug fix

* Hirarchical pk bug fix

* Hirarchical pk bug fix

* Hirarchical pk bug fix

* Hirarchical pk bug fix

* testing new version

* adding more tests

* adding more tests

* adding more tests

* code review changes

* test fix

* test fix

* test fix

* test fix

---------

Co-authored-by: Nalu Tripician <[email protected]>

…endation (#4004)

* Update changelog.md

This is a recommendation for customers if they would like to use the ODE features.

* Updated release notes for ODE

…ion pipeline (#3839)

* Infrastructure for performance testing with ODE pipeline.

* Resolve comments

* Removed randomization from data creation process

* Fixed comments

* Removed Query and EnableODE from QueryStatisticsMetrics, as they do not relate to query statistics.

* Removed try catch to make CreateItemAsync call always succeed

* Removed one liner functions

* Removed code from MetricsSerializer and QueryStatisticsDatumVisitor files

* Fixed comments

* Removed request Charge check

* Bug in Debug Assert

* Test

* Bug in debug assert fix

* Fixed second bug in Metrics Accumalator class

* Added ignore flag to ode perf tests so that they do not run on every loop build

* Added comment explaining the Ignore flag.

* Added code to throw exception if ODE continuation token goes into non ODE pipeline

* Removed count variable

* Updated test name

* Removed ODE continuation token logic from caller class

* Simplified code

* Fixed comments

* Updated continuation token cast

* Removed const string for continuation token

* Added Ignore flag for test

* Added baseline test

* Updated baseline test

* Updated change log and bumped up the patch version.

* Code changes to update the change log message.

…nd events using custom listener (#4021)

* custom listener example

* removed unwanted code

* add comments

* fix appsettings

* revert changes

…on) + Azure Cosmos DB typo (#4031)

* Documentation link fix

* Fixed Typo "Azure CosmosDB"→"Azure Cosmos DB"

* Adding metrics for Benchmark tool.

* Adding OpenTelemetry.

* Revert "Adding OpenTelemetry."

This reverts commit c7da088.

* Telemetry for windowed percentiles.

* OpenTelemetry, AppInsights and Dashboard.

* Removing DiagnosticDataListener.

* Code styling, comments and clean-up.

* Fixing issues with dashboard.

* Fixing positions of charts on the dashboard.

* Fixing the dashboard.

* Updating titles and subtitles.

* Removing ILogger and other not required references.

* Fixing code review points.

* Fixing issues after rebase.

* Removing unnecessary changes.

* Fixing code review points.

* Adding metrics for Benchmark tool.

* Adding OpenTelemetry.

* Revert "Adding OpenTelemetry."

This reverts commit c7da088.

* Telemetry for windowed percentiles.

* OpenTelemetry, AppInsights and Dashboard.

* Removing DiagnosticDataListener.

* Code styling, comments and clean-up.

* Fixing issues with dashboard.

* Fixing positions of charts on the dashboard.

* Fixing the dashboard.

* Updating titles and subtitles.

* Removing ILogger and other not required references.

* Fixing code review points.

* Fixing issues after rebase.

* Removing unnecessary changes.

* Fixing code review points.

* Fixing code review points.

* make MetrcisCollectorProvider non static and remove locks

* fix

* fixes

* use static class name TelemetrySpan.IncludePercentile

* use app insights connection string

* modified:   Microsoft.Azure.Cosmos.Samples/Tools/Benchmark/Program.cs

* modified:   Microsoft.Azure.Cosmos.Samples/Tools/Benchmark/Program.cs

* rename AppInsightsConnectionString

* fix

* fix comments

* fix if AppInsights c string is not set

* summary

* fix

* remove unnecessary collector types

* remove unnecesary metere provicer

* add event source

* remove folder

* fix

* split success and failed latencies

* fix

* fix

---------

Co-authored-by: David Chaava <[email protected]>
Co-authored-by: David Chaava <[email protected]>

…ress Refresh (#4039)

* Code changes to fix unobserved exception during background address refresh.

* Code changes to add exception handler in task.

* Code changes to fix null ref exception.

* Revert "Code changes to fix null ref exception."

This reverts commit 83f90d5.

* Revert "Code changes to add exception handler in task."

This reverts commit c49ed81.

* Code changes to address review comments.

* Revert "Code changes to address review comments."

This reverts commit d2b9f6b.

…4042)

* Added small remark for GetContactedRegions method documentation

* Moved to remarks

…ata to service (#3900)

* first draft

* comment other pipelines

* pint variables

* commnet other pipelines

* added env variable

* minor changes

* update env variable

* print env variable

* add space in end

* fix test

* fix tests

* fix test

* fix tests

* remove response interceptor

* logs

* debuug mode

* 3failing test to print llgs

* minor refactoring

* 2nd windows-2019

* fix ct tests

* 2remove debugging

* fix tests

* revert

* ncomment pipelines

* fix test

* minor changes

* release and emulator pipeline

* update pipelines

* ignore abstract class test

* fixing pipeline

* refactor code

* change it to class name to run tests

* added emulator setup

* 1 temp commit

* env variable

* renames env variable

* fix tests

* add condition

* fix tests

* reorder env variable

* revert pipeline

* did some clean up

* change to revert

* Revert "change to revert"

This reverts commit 03db3c104505dc7b8f3cea267835c92ca530f8f4.

* fix typos

* throw if exception intercepter is null

* remove modelling changes

* removed virtaul

* Update Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.EmulatorTests/Utils/HttpHandlerHelper.cs

Co-authored-by: Matias Quaranta <[email protected]>

* added condition for pipelines

* Revert "added condition for pipelines"

This reverts commit f9a208cd28e01badee97a2eb770a486cea67c1f0.

* changed cond

* fix codn

* more enhancement

* testing for release pipeline

* refactore code and using test category

* added comments on test

* refactor pipeline code

* fix variables

* fix pipeline

---------

Co-authored-by: Matias Quaranta <[email protected]>