Support Sending raw logs via Elasticsearch exporter when format option is set

[raghu999]

Component(s)

No response

Is your feature request related to a problem? Please describe.

The current exporter automatically converts the logs into plog format which adds attribute prefixes to all the fields that break the existing dashboards and alerts for enterprises who are using the elastic search exporter. Please see the below debug logs from the Otel collector

Event received by otel

ObservedTimestamp: 2023-03-29 17:47:46.28812868 +0000 UTC
Timestamp: 2023-03-29 17:47:46.071000064 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Map({"message":"{\"host\":\"17.47.219.133\",\"user-identifier\":\"shaneIxD\",\"datetime\":\"29/Mar/2023:17:47:46\",\"method\":\"HEAD\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/1.1\",\"status\":\"500\",\"bytes\":28854,\"referer\":\"https://for.us/user/booperbot124\"}","source_type":"demo_logs"})
Attributes:
     -> protocol: Str(HTTP/1.1)
     -> host: Str(17.47.219.133)
     -> datetime: Str(29/Mar/2023:17:47:46)
     -> method: Str(HEAD)
     -> request: Str(/do-not-access/needs-work)
     -> status: Str(500)
     -> bytes: Double(28854)
     -> referer: Str(https://for.us/user/booperbot124)
     -> user-identifier: Str(shaneIxD)
Trace ID:
Span ID:
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "logging"}

Elastic view

  "_source": {
   "@timestamp": "2023-03-29T17:46:42.069999872Z",
   "Attributes.bytes": 39389,
   "Attributes.datetime": "29/Mar/2023:17:46:42",
   "Attributes.host": "48.247.3.152",
   "Attributes.method": "PATCH",
   "Attributes.protocol": "HTTP/1.1",
   "Attributes.referer": "https://some.net/controller/setup",
   "Attributes.request": "/observability/metrics/production",
   "Attributes.status": "550",
   "Attributes.user-identifier": "benefritz",
   "Body.message": "{\"host\":\"48.247.3.152\",\"user-identifier\":\"benefritz\",\"datetime\":\"29/Mar/2023:17:46:42\",\"method\":\"PATCH\",\"request\":\"/observability/metrics/production\",\"protocol\":\"HTTP/1.1\",\"status\":\"550\",\"bytes\":39389,\"referer\":\"https://some.net/controller/setup\"}",
   "Body.source_type": "demo_logs",
   "SeverityNumber": 0,
   "TraceFlags": 0
 }

Describe the solution you'd like

According to https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/model.go#L51 the elasticsearch exporter automatically converts the data into plog format before sending the event to Elastic. The only way we can handle this is to add ingest pipleine on Elastic to strip off the attribute field.

We would like to see a RAW format option like loki and kafka exporter where we can send the raw logs to elasticsearch.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter

The following encodings are valid only for logs.
raw: if the log record body is a byte array, it is sent as is. Otherwise, it is serialized to JSON. Resource and record attributes are discarded.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/lokiexporter

The following formats are supported:
logfmt: Write logs as logfmt lines.
json: Write logs as JSON objects. It is the default format if no hint is present.
raw: Write the body of the log message as string representation.

Describe alternatives you've considered

No alternatives found except for adding ingest pipelines on the Elastic

Additional context

No response

[github-actions]

Pinging code owners for exporter/elasticsearch: @JaredTan95. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

makes sense, I think we can support it in es exporter.

[raghu999]

Hi @JaredTan95,

Thank you for your response. It's great to hear that you think we can support this feature in the Elasticsearch exporter. We're eager to see this feature implemented.

Is there any update on the progress of this feature request? If there's anything we can do to help or contribute to its development, please feel free to let us know. We're excited to see this feature added to the project and are willing to assist in any way we can.

Looking forward to your response and the future improvements to the project.

[JaredTan95]

Hi @JaredTan95,

Thank you for your response. It's great to hear that you think we can support this feature in the Elasticsearch exporter. We're eager to see this feature implemented.

Is there any update on the progress of this feature request? If there's anything we can do to help or contribute to its development, please feel free to let us know. We're excited to see this feature added to the project and are willing to assist in any way we can.

Looking forward to your response and the future improvements to the project.

Feel free to contribute if you have time~

[ycombinator]

Hi @JaredTan95, I'd like to work on this issue unless you're already working on it. Thanks!

[JaredTan95]

Hi @JaredTan95, I'd like to work on this issue unless you're already working on it. Thanks!

It's yours~

[ycombinator]

@raghu999 @JaredTan95 I've implemented this feature, albeit via a more explicit setting name, mapping.omit_attributes_prefix, in #29619. Please let me know what you think. Thanks!

[raghu999]

This is a cleaner approach than the one I was planning to implement. Thanks @ycombinator

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/elasticsearch: @JaredTan95

See Adding Labels via Comments if you do not have permissions to add labels yourself.