-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cosmos Metrics: allow enabling only metrics of certain categories #33436
Merged
FabianMeiswinkel
merged 50 commits into
Azure:main
from
FabianMeiswinkel:users/fabianm/MetricsFixes
Feb 17, 2023
Merged
Cosmos Metrics: allow enabling only metrics of certain categories #33436
FabianMeiswinkel
merged 50 commits into
Azure:main
from
FabianMeiswinkel:users/fabianm/MetricsFixes
Feb 17, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FabianMeiswinkel
requested review from
kushagraThapar,
kirankumarkolli,
xinlian12,
milismsft,
aayush3011,
simorenoh,
jeet1995 and
Pilchie
as code owners
February 10, 2023 03:57
ghost
added
the
Cosmos
label
Feb 10, 2023
API change check APIView has identified API level changes in this PR and created following API reviews. |
xinlian12
reviewed
Feb 10, 2023
...ure-cosmos/src/main/java/com/azure/cosmos/implementation/clienttelemetry/MetricCategory.java
Show resolved
Hide resolved
xinlian12
reviewed
Feb 10, 2023
...in/java/com/azure/cosmos/implementation/directconnectivity/rntbd/RntbdClientChannelPool.java
Outdated
Show resolved
Hide resolved
xinlian12
reviewed
Feb 10, 2023
...src/main/java/com/azure/cosmos/implementation/directconnectivity/rntbd/RntbdRequestArgs.java
Show resolved
Hide resolved
xinlian12
reviewed
Feb 10, 2023
...main/java/com/azure/cosmos/implementation/directconnectivity/rntbd/RntbdServiceEndpoint.java
Outdated
Show resolved
Hide resolved
xinlian12
reviewed
Feb 10, 2023
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/models/CosmosClientTelemetryConfig.java
Outdated
Show resolved
Hide resolved
xinlian12
reviewed
Feb 10, 2023
...-cosmos/src/main/java/com/azure/cosmos/implementation/clienttelemetry/CosmosMetricNames.java
Outdated
Show resolved
Hide resolved
…s fields in AsyncDoclmplementations
…to users/fabianm/MetricsFixes
Azure Pipelines successfully started running 1 pipeline(s). |
1 similar comment
Azure Pipelines successfully started running 1 pipeline(s). |
…to users/fabianm/MetricsFixes
/azp run java - cosmos - spark |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
…to users/fabianm/MetricsFixes
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - spark |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - tests |
/azp run java - cosmos - spark |
Azure Pipelines successfully started running 1 pipeline(s). |
1 similar comment
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - spark |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - spark |
Azure Pipelines successfully started running 1 pipeline(s). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is modifying which MicroMeter metrics are collected by default when registering a MeterRegistry and allows enabling/disabling metrics based on categories - even at runtime based on configuration changes. This change is motivated by feedback from a few customers who have enabled metrics.
New public API
Filtering by metric category
The
CosmosMicrometerMetricsOptions
gets three new methods to modify the metric categories of to-be-emitted metrics.setMetricCategories
- allows adjusting the metric categories to a specific set of metricsaddMetricCategories
- allows adding metric categories to the list of already emitted metricsremoveMetricCategories
- allows removing a specific set of metric categories (except for MINIMUM categories (OperationSummary and System) which are required metrics)All three methods can be called on the
CosmosMicrometerMetricsOptions
instance even after building a Cosmos(Async)Client - and changes to the categories will be reflected at runtime. This allows changing the to-be-collected metrics via configuration at runtime without the need to restart CosmosClients or the app.Sample
Adjusting percentiles/histogram capturing
For some of the metrics collected in the Cosmos SDK by default percentiles (0.95 and 0.99) and histograms (to allow calculation of percentiles across multiple client machines) are published. Collecting percentiles and especially histograms has some overhead (memory and CPU) - so, there is a new API that allows more fine granular configuration of which percentiles to colelct and whether histograms should be published.
Default settings
The default settings can be overriden on the
CosmosMicrometerMetricsOptions
the defaults are applicable whenever a meter gets registered in MicroMeter for the first time unless there is a meter-specific override. Some of the settings (like whether a meter is enabled or not) can be changed afterwards at runtime.Sample
Overriding meter specific settings.
The sample below adds a meter-specific override of the settings (in this case for percentiles and whether to publish histograms for the
cosmos.client.op.RUs
meter.Sample
Configuring tags
For each of the meters emitted by the Cosmos DB SDK we associate a certain set of tags/dimensions. Tags have a certain overhead especially when they have a high-cardinality (for example the ServiceEndpoint, ServiceAddress or PartitionKeyRangeId have a cardinality varying by the number of physical partitions). While these tags/dimensions are super useful when trying to triage why latency is higher than expected or whether certain errors are only coming from a certain client-machine or backend endpoint, whether or not to collect them is a trade-off. The following APIs allow overriding the default behavior for which tags to use.
Defaults
Meter specific override
Disabling individual meters
As described above usually filtering of which meters to collect would be done by meter category. If there is a need to still control whether a meter should be emitted on a per meter base the
CosmosMicrometerMeterOptions.setEnabled(boolean)
API can be used.Behavioral breaking change
.setMetricCategories(CosmosMetricCategory.ALL)
- see below for an exampleCosmosClientTelemetryConfig.metricTagNames
in this PR - as described above the APIsCosmosMicrometerMetricsOptions.defaultTagNames
orCosmosMicrometerMeterOptions.suppressTagNames
can be used instead.Considered alternatives
Internal implementation considerations
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines