Should tracers/loggers/meters be enabled by default

[lmolkova]

Related to #3877 and #3867

Currently OTel does not control how traces/meters are enabled. It's usually controlled by users who install an instrumentation library (standalone or via distro) and sometimes explicitly enable/disable instrumentations or parts of them.
Different languages and distros use different approaches. Logs rely on existing logging library configuration.

As OTel adoption grows, we see more native instrumentations, more instrumentation libs in the distros, and more details recorded by them and we need a mechanism to enable and disable instrumentation scopes (see #3867 for the context and #3877 for the solution).

One of the problems is picking reasonable defaults for OTel SDKs (while allowing users and distros to tune/override them)

• All registered meters/tracers are on by default:
  • this is an overwhelming experience, especially if we get into lower level network stack, web framework details, etc
• All meters/tracers are off by default and users need to enable (e.g. in .NET you do tracerProviderBuilder.AddSource("Azure.*") to enable all tracers for Azure SDKs)
  • this is an underwhelming default experience
  • hard to discover what's instrumented, need to know what to enable
  • still hard to disable something specific or know which scope are more interesting than others
• Something in between (does not exist), e.g.
  • Client spans are on, internal are off
  • Scopes marked with specific attribute with value higher than "important" are on (similar to log level)

Distros are able to mitigate most of the issues (e.g. provide good initial set of enabled instrumentations) with either of those approaches, but they still need to manually curate list of 'good' scopes for each library and have a default for the rest.

Full context in available in the #3877 (comment)

[trask]

maybe also related to #3205

[tedsuo]

@lmolkova I suggest that we look into the ability to use file config syntax to disable instrumentation.

In general, my opinion is that instrumentation should be turned on by default. Otherwise, users have a very hard time discovering what instrumentation would be available.

[trask]

Distros are able to mitigate most of the issues (e.g. provide good initial set of enabled instrumentations) with either of those approaches, but they still need to manually curate list of 'good' scopes for each library and have a default for the rest.

👍

In the Java agent distro we've more or less landed on: "instrumentations which produce INTERNAL spans should be off by default, except for scheduled job instrumentations (which produce top-level INTERNAL spans)"

In general, my opinion is that instrumentation should be turned on by default. Otherwise, users have a very hard time discovering what instrumentation would be available.

this could also be a bad experience if say Hibernate natively instrumented itself producing INTERNAL spans. maybe that's not something we need to solve though, as there would likely be pushback on Hibernate to make this kind of instrumentation opt-in (via their own mechanism?)

[lmolkova]

I think we can only continue the journey of everything is on by default since we're already on it and changing it would be breaking anyway. But we can make it better with new features like verbosity.

We still have the consistency problem - at least some of the languages (.NET, not sure if there are any others) do "everything off by default" #3877 (comment)

[pyohannes]

We still have the consistency problem - at least some of the languages (.NET, not sure if there are any others) do "everything off by default" #3877 (comment)

All languages I'm aware of do "everything off by default" if you're just using the plain OTel SDK (no distro or auto-instrumentation).

Obviously the situation is different when using a distro or auto-instrumentation. The .NET auto-instrumentation turns on a bunch of instrumentation libraries by default.

I'd agree there's a conceived inconsistency between languages where auto-instrumentation is the default use case (Java) and others where it isn't (.NET).

[cijothomas]

All languages I'm aware of do "everything off by default" if you're just using the plain OTel SDK (no distro or auto-instrumentation).

I have a different understanding! All languages are on-by-default, and .NET is the exception with off-by-default, strictly from plain OTel SDK!

By "on-by-default", what I mean is - traces/metrics from every tracer/meter is automatically collected, without any particular user-action. In .NET, the Meter/TracerProvider must be explicitly configured with the list of Tracers/Meters to listen to.

For the below pseudo code, .NET requires the TracerProvider be explicitly configured with AddSource("foo") for the spans from this tracer to work. In other languages I have checked (C++, Rust, Python), there is no additional config required.

fooTracer = get-tracer("foo");
fooTracer.StartSpan("myspan");

[pyohannes]

I have a different understanding! All languages are on-by-default, and .NET is the exception with off-by-default, strictly from plain OTel SDK!

@cijothomas You're right. For languages besides .NET it is enough to install and initialize instrumentation libraries, without additional SDK configuration. For .NET, additional SDK configuration is needed besides installing and initializing instrumentation libraries (for example AddSource or AddMeter on the SDK).

I was looking at it more from a user than from an SDK point of view: they have to install instrumentation libraries to "turn on" certain instrumentations. However, also from the user point of view my statement doesn't hold true when one takes into account native instrumentation (instrumentation in the instrumented library itself) which is enabled by default.

[austinlparker]

It seems like there's some general agreement that 'on by default' is the expectation, would a PR clarifying that help?

[svrnm]

@lmolkova please take a look