Loki to Loki exporter not working

[trajano]

Component(s)

exporter/loki

What happened?

Description

I am trying to send data from promtail to loki via the OTEL collector. It's does not appear to be sending to LOKI

Steps to Reproduce

Configure a loki receiver and loki exporter
Send a message based on the example

curl -v -H "Content-Type: application/json" -XPOST -s "http://otelcol:3500/loki/api/v1/push" --data-raw   '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "'$(date +'%s')'000000000", "fizzbuzz3" ] ] }]}'

Expected Result

Debug message appears AND data appears in loki

Actual Result

Debug message appears but no data appears in loki.

Sending via

curl -v -H "Content-Type: application/json" -XPOST -s "http://loki:3100/loki/api/v1/push" --data-raw   '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "'$(date +'%s')'000000000", "fizzbuzz2" ] ] }]}'

Works and I receive the entry.

Collector version

v0.88.0

Environment information

Environment

OS: DietPi
Docker Swarm

OpenTelemetry Collector configuration

exporters:
  debug:
    verbosity: detailed
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: true
      job: true
receivers:
  loki:
    protocols:
      http:
    use_incoming_timestamp: true
processors:
  batch:
  attributes/loki:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: container
      - action: insert
        key: loki.format
        value: raw
service:
  pipelines:
    logs:
      receivers: [loki]
       # I tried without batch even attributes
      processors: [attributes/loki,batch]
      # I also tried without debug
      exporters: [debug,loki]

Log output

{"level":"info","ts":1699392758.861195,"msg":"LogsExporter","kind":"exporter","data_type":"logs","name":"debug","resource logs":1,"log records":1}
{"level":"info","ts":1699392758.8614204,"msg":"ResourceLog #0\nResource SchemaURL: \nScopeLogs #0\nScopeLogs SchemaURL: \nInstrumentationScope  \nLogRecord #0\nObservedTimestamp: 2023-11-07 21:32:38.731536162 +0000 UTC\nTimestamp: 2023-11-07 21:32:38 +0000 UTC\nSeverityText: \nSeverityNumber: Unspecified(0)\nBody: Str(fizzbuzz3)\nAttributes:\n     -> foo: Str(bar2)\n     -> loki.attribute.labels: Str(container)\nTrace ID: \nSpan ID: \nFlags: 0\n","kind":"exporter","data_type":"logs","name":"debug"}

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @trajano, are you seeing any errors in your loki backend that could explain the failure here? Or any other collector errors coming from the loki exporter?

Also, I noticed this in your config:

processors:
  batch:
  attributes/loki:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: container
      - action: insert
        key: loki.format
        value: raw

From the README, the loki.format variable is actually supposed to be a resource attribute, so it would be defined in a resource processor instead, and added to the pipeline there.

[trajano]

Checked, nothing in the logs.

FYI I did copy that block from
https://grafana.com/docs/opentelemetry/collector/send-logs-to-loki/loki-receiver/#use-opentelemetry-collector-to-send-logs-to-loki

[crobert-1]

Checking, but I did copy that block from https://grafana.com/docs/opentelemetry/collector/send-logs-to-loki/loki-receiver/#use-opentelemetry-collector-to-send-logs-to-loki

Oh, thanks for linking, you're right. We'll have to check with the loki code owners to see if one is wrong (maybe they both work?), and update documentation if necessary 👍

[crobert-1]

I'll have to defer to the Loki exporter code owners for now, I'm not very experienced here.

[trajano]

NP I did try

  resource/loki:
    attributes:
      - action: insert
        key: loki.format
        value: raw

And

    logs:
      receivers: [loki]
      processors: [resource/loki]
      exporters: [loki]

No luck. I wonder if there's a way to debug the exporter itself so I can tell whether it is trying to send something

[crobert-1]

One more thing: Could you try to run with the log level set to debug? Maybe there are some logs that will show.

[trajano]

Nope all it showed was this which was part of the tracing #29015

{
  "level": "debug",
  "ts": 1699396862.3295133,
  "caller": "[email protected]/compare.go:22",
  "msg": "string to non-string",
  "kind": "processor",
  "name": "filter/drop_actuator",
  "pipeline": "traces",
  "op": 0
}

I turned off the trace processors to silence that log but all I see is the info log for the message I send

[jangaraj]

IMHO it's config issue. Log MUST have some label(s) otherwise Loki can't index it.
Log example has only foo: bar attribute, but config is saying that container attribute will be label used by Loki. But that attribute is missing, so there is no label which can be used by Loki.

[trajano]

I had this originally

  attributes/loki:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: container

Shouldn't that add a label?
I readded so I have both

  resource/loki:
    attributes:
      - action: insert
        key: loki.format
        value: raw
  attributes/loki:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: container

and still no luck. There's no explicit log message either. I updated the OP to show the screenshot on Grafana

I also have

    default_labels_enabled:
      exporter: true
      job: true

I presume those would add them.

[jangaraj]

Try to use foo attribute as label because you don't have container attribute, pls
attributes/loki:
actions:
- action: insert
key: loki.attribute.labels
value: foo

[trajano]

  attributes/loki:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: foo
...
  pipelines:
    logs:
      receivers: [loki]
      processors: [attributes/loki]
      exporters: [debug,loki]

Still not showing anything on Grafana loki when I send

> curl -v -H "Content-Type: application/json" -XPOST -s "http://otelcol:3500/loki/api/v1/push" --data-raw   '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "'$(date +'%s')'000000000", "fizzbuzz3" ] ] }]}'

*   Trying 10.0.14.51:3500...
* Connected to otelcol (10.0.14.51) port 3500
> POST /loki/api/v1/push HTTP/1.1
> Host: otelcol:3500
> User-Agent: curl/8.4.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 100
>
} [100 bytes data]
< HTTP/1.1 204 No Content
< Date: Tue, 07 Nov 2023 23:26:07 GMT
<
* Connection #0 to host otelcol left intact

[jangaraj]

Debug output always, pls. And format them for humans, pls e.g.:

ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2023-11-07 21:32:38.731536162 +0000 UTC
Timestamp: 2023-11-07 21:32:38 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(fizzbuzz3)
Attributes:
     -> foo: Str(bar2)
     -> loki.attribute.labels: Str(container)
Trace ID: 
Span ID: 
Flags: 0

This doesn't make sense:

Attributes:
     -> foo: Str(bar2)
     -> loki.attribute.labels: Str(container)

It should be:

Attributes:
     -> foo: Str(bar2)
     -> loki.attribute.labels: Str(foo)

[trajano]

I sent the message based on the example

https://grafana.com/docs/loki/latest/reference/api/#examples-4

I just swapped a couple of values

[jangaraj]

Pls provide debug output. And format them for humans.

[trajano]

I got it working with the following:

  attributes/loki:
    actions:
    - action: insert
      key: loki.format
      value: raw
    - action: insert
      key: loki.attribute.labels
      value: exporter

Seems exporter is sent implicitly so I don't have to list it. However, all the mappings I did on promtail are lost. I am going to see if that loki.format would resolve it.

[trajano]

So trying out a message from promtail

2023-11-08T00:13:40.017Z        info    ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2023-11-08 00:13:40.017681217 +0000 UTC
Timestamp: 2023-11-08 00:13:38.688613504 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(whomami 2 requested)
Attributes:
     -> stream: Str(stdout)
     -> detected: Str(logstash)
     -> level: Str(INFO)
     -> logger: Str(net.trajano.simpleauth.RootController)
     -> service_name: Str(trajano_auth)
     -> loki.attribute.labels: Str(exporter)
Trace ID: 
Span ID: 
Flags: 0
        {"kind": "exporter", "data_type": "logs", "name": "debug"}

Got this

If it is raw I get

So seems to be a limitation then that if I do my initial labeling in promtail by the OTEL processes it those are gone unless I explicitly list it. Also it appears that the structured_metadata isn't transmitted either.

[jangaraj]

I think you have wrong expectation. Collector doesn't work like a "proxy" (receive, don't change anything and export). It is processing by default: receive -> convert to common OTLP format, so -> any exporter can convert it from common OTLP format into vendor specific format.

When you have:

Attributes:
     -> stream: Str(stdout)
     -> detected: Str(logstash)
     -> level: Str(INFO)
     -> logger: Str(net.trajano.simpleauth.RootController)
     -> service_name: Str(trajano_auth)

And you want to have all attributes as Loki labels, then you need:

    - action: insert
      key: loki.attribute.labels
      value: exporter, stream, detected, level, logger, service_name

[trajano]

Yeah I thought as much as I was going through it.

https://grafana.com/docs/opentelemetry/collector/send-logs-to-loki/loki-receiver/#use-opentelemetry-collector-to-send-logs-to-loki needs to put your comment on the page in it's own section.

The OpenTelemetry Collector does not work like a proxy where ... (the rest of your comment)

It should also note that Loki specific constructs like structured metadata is not part of OTEL and would be lost. Though I think OTEL should have a separate grouping for structured metadata on the logs. Or perhaps make it such that resource/... processor is the one that creates the labels and attribute/... processor creates structured meta data.

Same with the README

In the end I think its a documentation bug because

Let’s imagine you have your application running, and you would like to collect metrics, traces, and logs from it.

Using the OpenTelemetry Collector allows you to collect all these signals in one place, and control various backends that the individual signals send to.

makes it sound like a simple proxy, thus creating the expectation if I receive LOKI, I should be able to send it back out. Also that even if I receive LOKI I have to know what I get and parse it accordingly.

[mar4uk]

right, all labels you configured in promtail will be lost in the collector unless you explicitly promote them to loki labels.

Thank you for the feedback regarding docs. Agree docs should make it clear that collector is not a proxy

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

I'm closing, as I believe this has been answered.