-
Notifications
You must be signed in to change notification settings - Fork 834
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exclude URLs from Tracing #1060
Exclude URLs from Tracing #1060
Comments
That seems like a reasonable approach :) |
Been trying auto instrumentation in a container lately, and was slightly annoyed myself with tracing of health check. Then a customer trying it out independently had the same feedback - it's exactly the kind of input we're hoping for in trials :) So I may actually mark this required for GA, the UX is impacted a lot with having no control of tracing by URL pattern. |
I have started looking into this. Is there a proposed design for this? In OpenTracing this was an instrumentation feature. The instrumentation check if the URL matches exclude pattern if yes then the span wasn't created. However if the excluded URL uses another instrumentation (or makes downstream call) that would create a span. The question is whether we want to exclude just specific URLs or the whole trace starting at that URL. |
Yes, we have a proposal right in the task's description:
This should lead to ignoring the whole subtree starting from that |
+1 I think that is the right way to go open-telemetry/opentelemetry-specification#173 (comment). It would also make sense to have a consolidated config property for this. @iNikem what is the API to create non-recording span? In OT there was |
I think the right way is to use one of the factory methods on |
Yeah, I wanted to avoid having two paths of span creation. |
Sorry if double-spam, I thought I had already posted this. How about we have a special |
Note that |
Yes, we want to do that eventually. |
Is there a way now to exclude health check traces? I checked processors but could not find any solution too. |
No, this functionality is not yet implemented. |
bump. Is there any workaround here in the meantime? Sampling of health checks is not ideal. |
The only known workaround is to write custom sampler. But I have plans to address this issue during the next month or so. |
The corresponding Sampler has been implemented in the contrib repo. It can be added to your deployment using extension mechanism. There is no immediate plans to add that sampler into this distribution, as this requires changes in Otel Specification and that requires some effort. |
for anyone who would like to work on adding this functionality to the base otel javaagent, here's a very high-level outline: use a yaml configuration file to dynamically configure the RuleBasedRoutingSampler during startup of the javaagent use a system property, e.g. check out the jmx-metrics and metric view yaml configuration for some inspiration |
Based on @jack-berg examples, and the possibility to embed extension directly into the agent, I've created this simple project that provides both the java agent and the docker auto instrumentation that will ignore |
We have similar requirements and wrote an extension that allows you to dynamically configure what sampler to use and add filtering for span creation. Configuration can be done by simply changing the configuration file while the application is running, or also by using the optional REST service for configuring multiple agents. The service also exposes Prometheus compatible metrics. You might find it useful. See https://github.com/domstolene/da-otel-agent for details. |
I did a work-around for this having a look at how the OpenTelemetryAutoConfiguration class builds the SdkTracerProvider where the sampler is being set for the bean in Spring. I duplicated the code in my bean configuration after I have seen the original bean is conditional and created a custom a sampler, which I used to drop any health check related traces. The caveat is that the actual health check traces are coming with no names to the sampler and denying all spans with no names may drop some unintended traces... For now we are just only doing a PoC for a product, so take this as it is :) Here is the sample for this:
|
We have some services using Micronaut and it has a simple way of excluding HTTP routes https://micronaut-projects.github.io/micronaut-tracing/latest/guide/#http but we have other tech stacks too to deal with. Also noticed those links in the contr project were broken. Think this is the new place. |
If you're using a central collector to send traces to your backends, you can filter out those spans - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md |
linking related: open-telemetry/oteps#240 |
my architecture - |
Is there any update on this? I understand the concerns raised by @iNikem about the specification being hard to change but this feature has been requested by numerous people for quite some time now. I think this is a very important and much-needed feature for a lot of users of OpenTelemetry. @trask Has anyone tried to contribute to the project with this feature? I would like to take a go at implementing it for the Otel Java agent. |
hi @nedcerneckis!
I don't believe we're blocked by specification work, since we can add this initially as an opt-in experimental feature (bypassing the need for a specification)
not yet, that would be great, check out #1060 (comment) for very high-level sketch that could fit well with existing feature set |
For Lumigo's distribution I implemented a My approach was to allow urls for client and server to be filter, or separate env vars to filter urls for only client or server spans. The urls are defined as an array of regex, such as |
Thank you very much @trask! Much appreciated for the info. I'm assuming this needs to be a little more advanced than just excluding certain HTTP endpoints and include other types of rules inside this YAML config file? |
life savier, for me this is the easiest solution, but I only needed the jar file: stages:
- get_jar
- distribute
variables:
OTEL_LIB_VERSION: 1.32.0
Get Jar:
stage: get_jar
image: ghcr.io/vmaleze/opentelemetry-java-ignore-spans:$OTEL_LIB_VERSION
script:
- cp /javaagent.jar ./
artifacts:
paths:
- './javaagent.jar'
expire_in: 1 week
Deploy JAR:
image: maven:3.6-openjdk-17-slim
stage: distribute
rules:
script:
- mvn deploy -DskipTests |
I'm one who also waiting for that feature, but I solved this problem from another side (this approach is probably not for everyone). Here is the documentation about processors: https://opentelemetry.io/docs/collector/configuration/#processors In total now I have such configuration on my project (which I'll certainly would like to change on config property when it'll be implemented): config:
exporters:
...
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, filter/drophttp] # do not forget to add processor otherwise it will not work
exporters: [otlp]
processors:
filter/drophttp:
error_mode: ignore
traces:
span:
- attributes["http.url"] == "/actuator/health"
- attributes["url.path"] == "/actuator/health" I would be appreciated to myself if I would find this comment hours before. 😄 |
I did the same but I ended up with orphaned spans. How are you dealing with those? |
@Kortex, does tail sampling not work? |
@shameekagarwal I did not try out tail sampling. Would it be possible to share your configuration? |
i linked my comment, where the code is in a collapsible....expand solution 2 |
So I decided to abandon the processors:
tail_sampling:
policies: [
{
name: filter_http_url,
type: string_attribute,
string_attribute: {
key: http.url,
values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
enabled_regex_matching: true,
invert_match: true
}
},
{
name: filter_url_path,
type: string_attribute,
string_attribute: {
key: url.path,
values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
enabled_regex_matching: true,
invert_match: true
}
}
] NOTE: there are two of them because in your project can be enabled both otel:
instrumentation:
spring-web:
enabled: true
spring-webmvc:
enabled: true Not sure why @shameekagarwal has key |
one comment here, tail sampling works absolutely fantastic for almost all issues here (related to actuator) but I have a SSE stream that can be open for more than 1 hour. Tail sampling can't keep up with such long trace. |
Re-opening until lingering issues are resolved which prevent declarative config from being used with the otel java agent: open-telemetry/opentelemetry-java-contrib#1440 (comment) |
Is your feature request related to a problem? Please describe.
As already mentioned here open-telemetry/opentelemetry-specification#173 I'd like to be able to exclude or sample a list of URLs / URL-Patterns from instrumentation. In my case particularly to avoid generating many events from health- and liveness-checks.
Describe the solution you'd like
I opened the issue open-telemetry/opentelemetry-java#1552 to discuss if / how tracing might be disabled from instrumentation.
To my knowledge there currently is no API / SDK method to disable tracing centrally on the context.
If I understood correctly a (maybe temporary) solution might be to create a non-recording / invalid span in the HTTP instrumentation, which due to the
ParentOrElse
-Sampler, would lead to ignoring child spans as well, if the request matches a URL pattern.Any hint to where this would be architecturally appropriately implemented is highly appreciated.
Maybe in the HttpServerTracer?
Describe alternatives you've considered
We already attempted to use the
otel.trace.classes.exclude
but only succeeded in completely disabling WebMvc instrumentation./CC @gabrielthunig @spaletta
The text was updated successfully, but these errors were encountered: