Support exposing internal metrics over OTLP rather than a Prometheus endpoint

[jrcamp]

Is your feature request related to a problem? Please describe.
Internal OT metrics are currently exposed through Prometheus endpoint which must then be scraped. This is more inefficient than if the metrics were sent directly through the pipeline. In addition, context is lost since OT has additional concepts like resource labels vs metric labels. The metric names are also transformed by being sent through Prometheus instead of directly through the pipeline.

This also results in additional complexity in the configuration since a Prometheus scraper must be configured.

Describe the solution you'd like
Be able to send internal metrics through the pipeline in OTLP format.

[jrcamp]

@bogdandrutu @tigrannajaryan Would like to put this in a GA milestone since changing this in the future will break metric names being exported.

[bogdandrutu]

@jrcamp I think we should send metrics directly to the backend not through the pipeline, because if there is a problem in the pipeline these metrics do not get to the backend.

[bogdandrutu]

I think the current approach is a "big" hack added long time ago. OpenCensus (which is currently used) can export directly to different backends and don't need this hack to export prometheus then scrape.

[jrcamp]

@ccaraman I think this falls under self-observability, do you want to take it?

[bogdandrutu]

To summarize, we have two problems here:

1. We expose internal metrics to a prometheus endpoint that we self-scrape and add data to the pipeline.
2. The data are sent to the backend using the internal pipeline, which smells bad because we send monitoring data about the pipeline using the same pipeline, so in case of a problem with a pipeline we cannot see monitoring data.

I think this issue tries to fix the first problem initially, then we can fix the second problem. @jrcamp am I correct?

[nilebox]

we should send metrics directly to the backend not through the pipeline

Note that the backend here may also differ from the "normal" one, e.g. could be using completely different exporter, or same exporter with a different config.

Also, the current Prometheus endpoint actually allows running a separate Collector instance for scraping and forwarding "self-observability" metrics, i.e. making isolation possible. The problem of course is that now you also need to monitor that "separate Collector".

Relevant issue for adding latency metrics: #542, which is blocked by the current approach of reusing the internal pipeline for self-observability.

[jrcamp]

Also, the current Prometheus endpoint actually allows running a separate Collector instance for scraping and forwarding "self-observability" metrics, i.e. making isolation possible. The problem of course is that now you also need to monitor that "separate Collector".

Is there any benefit in sending these metrics as Prometheus to a separate collector? Why not use the native OTLP format?

[jrcamp]

To summarize, we have two problems here:

1. We expose internal metrics to a prometheus endpoint that we self-scrape and add data to the pipeline.
2. The data are sent to the backend using the internal pipeline, which smells bad because we send monitoring data about the pipeline using the same pipeline, so in case of a problem with a pipeline we cannot see monitoring data.

I think this issue tries to fix the first problem initially, then we can fix the second problem. @jrcamp am I correct?

@bogdandrutu though can we address the second point by just using a dedicated pipeline? If the main pipeline is having issues (getting backed up, dropping data, etc.) it shouldn't affect the other pipeline (assuming the user hasn't configured it to).

[nilebox]

Is there any benefit in sending these metrics as Prometheus to a separate collector? Why not use the native OTLP format?

One minor potential benefit is that since Prometheus uses a "pull" model, the main collector won't have a dependency on a "separate" collector monitoring it. With a "push" model in OTLP, the main collector has to be configured to forward its self-observability metrics somewhere.

But that's probably not critical.

[tigrannajaryan]

Due to lack of time I am removing this for 1.0. If there are any objections please speak.

[mx-psi]

@codeboten should we close this in favor of your issue about making the Collector observable?

[mx-psi]

Closing in favor of #7532