Telegraf/InfluxDB exporter

[jacobmarble]

Description: Add influxdbexporter (WIP)

This change adds an exporter for InfluxData line protocol, the native wire transfer protocol for Telegraf and InfluxDB.

The conversion logic lives in package otel2influx, and is also imported by the proposed Telegraf OpenTelemetry input plugin. The objective of this strategy is to create a canonical mapping, and an open-source implementation, of OpenTelemetry <-> Telegraf/InfluxDB translation.

I am not proposing that otel2influx, in its current state, be the canonical mapping. Let's discuss that topic separate from this exporter.

One shortcoming of this approach is that both otel2influx and opentelemetry-collector package their own generated protocol buffer objects. This means that every OTLP protobuf object is serialized and deserialized in this exporter. I propose that opentelemetry-collector instead depend on opentelemetry-proto-go, for the benefit of this exporter and other third-party packages.

// r is a protobuf from otel2influx
// td is a protobuf wrapper from otelcol
var r otlpcollectortrace.ExportTraceServiceRequest
if protoBytes, err := td.ToOtlpProtoBytes(); err != nil {
	return err
} else if err = proto.Unmarshal(protoBytes, &r); err != nil {
	return err
}

Link to tracking Issue: #2951

Testing: Little testing so far. The translation logic should be tested in package otel2influx (but is not currently).

Documentation: README.md included in new exporter.

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ Jeff Cheng (7ac7a9756eba399accd045b5a45980c34a7c81ac)

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ Jacob Marble (7edc114d679800cf126b4226ec3778d547a39a71)

[codecov]

Codecov Report

Merging #2952 (86adb77) into main (17399fa) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 86adb77 differs from pull request most recent head db9726e. Consider uploading reports for the commit db9726e to get more accurate results

@@           Coverage Diff           @@
##             main    #2952   +/-   ##
=======================================
  Coverage   91.91%   91.91%           
=======================================
  Files         494      493    -1     
  Lines       23939    23947    +8     
=======================================
+ Hits        22003    22011    +8     
  Misses       1429     1429           
  Partials      507      507           

Flag	Coverage Δ
integration	63.32% <0.00%> (-0.37%)	⬇️
unit	90.93% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cmd/otelcontribcol/components.go	88.50% <100.00%> (+0.13%)	⬆️
exporter/signalfxexporter/config.go	83.67% <0.00%> (-2.38%)	⬇️
exporter/humioexporter/humio_client.go	94.36% <0.00%> (-0.08%)	⬇️
exporter/awsemfexporter/util.go	100.00% <0.00%> (ø)
receiver/redisreceiver/pdata.go	100.00% <0.00%> (ø)
exporter/humioexporter/config.go	100.00% <0.00%> (ø)
exporter/humioexporter/factory.go	100.00% <0.00%> (ø)
exporter/splunkhecexporter/client.go	93.75% <0.00%> (ø)
pkg/batchpersignal/batchpersignal.go	100.00% <0.00%> (ø)
receiver/redisreceiver/redis_metric.go	100.00% <0.00%> (ø)
... and 23 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84952e2...db9726e. Read the comment docs.

[tigrannajaryan]

@jacobmarble thanks for the contribution, this is very welcome!

I propose that opentelemetry-collector instead depend on opentelemetry-proto-go, for the benefit of this exporter and other third-party packages.

Unfortunately there is no easy way to do that. Collector uses Gogo Protobuf library (for performance reasons). opentelemetry-proto-go uses canonical Go Protobuf library. The in-memory representations of these 2 are different and incompatible.

[jacobmarble]

@jacobmarble thanks for the contribution, this is very welcome!

I propose that opentelemetry-collector instead depend on opentelemetry-proto-go, for the benefit of this exporter and other third-party packages.

Unfortunately there is no easy way to do that. Collector uses Gogo Protobuf library (for performance reasons). opentelemetry-proto-go uses canonical Go Protobuf library. The in-memory representations of these 2 are different and incompatible.

@tigrannajaryan I wanted to bring this up at the Golang SIG earlier today. Are you aware of a plan to consolidate?

[tigrannajaryan]

Great PR, thanks.
I am only blocking until we clarify the double-registration of Protobuf messages.

[tigrannajaryan]

This is a nice writeup. I wonder if it should be extracted and moved elsewhere (can be somewhere in this repo) since it is not Influxdb-specific.

[jacobmarble]

I agree that it doesn't belong here. My original audience for this section was other Influx employees, not the OpenTelemetry community.

The original is copied out to a text file here. Can you suggest a specific destination for it?

[tigrannajaryan]

@jacobmarble thanks for the contribution, this is very welcome!

I propose that opentelemetry-collector instead depend on opentelemetry-proto-go, for the benefit of this exporter and other third-party packages.

Unfortunately there is no easy way to do that. Collector uses Gogo Protobuf library (for performance reasons). opentelemetry-proto-go uses canonical Go Protobuf library. The in-memory representations of these 2 are different and incompatible.

@tigrannajaryan I wanted to bring this up at the Golang SIG earlier today. Are you aware of a plan to consolidate?

No existing plans AFAIK, but we likely want to do something about it anyway, since it was recently reported that Gogo Protobufs are no longer maintained. Good topic for the next Collector or Go SIG.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[jacobmarble]

(not stale)

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[jacobmarble]

FYI the corresponding Telegraf OpenTelemetry input plugin has been merged.
influxdata/telegraf#9077

[jacobmarble]

@tigrannajaryan FYI here is the corresponding receiver diff. Since there may be feedback here that improves that change, I'll wait to create that PR.

[tigrannajaryan]

@jacobmarble Mostly LGTM, thank you. I left a few minor comments.

[tigrannajaryan]

@tigrannajaryan FYI here is the corresponding receiver diff. Since there may be feedback here that improves that change, I'll wait to create that PR.

I glanced at the receiver and it appears to be doing the mirror of this exporter, which looks reasonable.

[jacobmarble]

@jacobmarble Mostly LGTM, thank you. I left a few minor comments.

@tigrannajaryan thanks for the thorough review. Clearly this PR has collected some dust over the last few weeks.