disable opentelemetry in rutime

[Alirezaeshaqi]

thats a follow up issue of open-telemetry/opentelemetry-java#1552
we need some feature to disable and also enable opentelemetry tracing exporters on runtime without redeploy on CD servers. the application is running with javaagent, is there any way to switch off the opentelemetry with properties files or something else in runtime ?

[jkwatson]

I'm going to move this issue over to the instrumentation repository, since it relates to agent configuration/runtime.

[trask]

hey @Alirezaeshaqi! check out https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/suppressing-instrumentation.md#disabling-the-agent-entirely

[Alirezaeshaqi]

thanks @trask , now I know how to disable javaagent before starting the application .
but we want to suppress the opentelemetry exporting while javaagent is running and during runtime,restart the agent brings us some overhead throughout our microservices and its inappropriate for our ecosystem .

[mateuszrzeszutek]

Hey @Alirezaeshaqi,

We currently don't support changing the javaagent configuration in the runtime.

Do you have any particular problems with the agent's performance? If possible, can you share some details with us?

[mityayak]

I think the use case would be large amount of traces which are expensive to store so that we want to disable all traces ootb and enable it runtime on some alert, i.e. when system is in a bad shape. I think in ideal scenario that will be controlled by jvm system property which we can change on/off runtime.

[trask]

this could potentially be done by writing a custom sampler that has a runtime-configurable sampling rate. e.g. jaeger has something like this: open-telemetry/opentelemetry-java#3368

[mityayak]

Thanks, if I create a custom sampler - how would I register it at agent side? Do I need to re-register it once we update system property at runtime?

[trask]

hey @mityayak, see https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/extension#introduction

[atennapel]

We have implemented a custom sampler which, based on a System property, drops all traces or not. We are very worried about the performance impact of this, since the traces are made before our custom sampler looks at them. If OpenTelemetry agent is disabled (through our System property) then traces will still be made, just not exported.

Are there any plans to support enabling/disabling the java agent at runtime, for example by reacting to configuration changes at runtime? For us this is an important feature.

[trask]

since the traces are made before our custom sampler looks at them

hey @atennapel! OpenTelemetry samplers are "head-based" as opposed to "tail-based" samplers, and run before spans are created.

that said, the javaagent still has to perform some work to capture the attributes that are fed to the Sampler, even when the sampler if "off". there's some discussions in OpenTelemetry specification repo to provide an earlier (or more optimized) hook for Sampling decisions without incurring this cost, so I expect to see something like that in the future

[awsiv]

With AWS x-ray, you could maintain a centralized sampling configuration that allows you to dynamically adjust what percent of traces you want to keep. Not sure if other backends support this though?

[trask]

Closing, as I think this question has been answered, but if not, let us know and we will re-open.