631 loki output

[romanchechyotkin]

Loki

Added loki output plugin

Fixes #631

[DmitryRomanov]

Hello! Good job but it's better to split up messages and extract fields to labels.

Here is example:

There is sample of config for file.d:

pipelines:
  loki:
    settings:
      capacity: 1000
    input:
      type: http
      emulate_mode: "elasticsearch"
      address: ":9200"
    actions:
      - type: parse_es
      - type: json_decode
        field: message
    output:
      type: loki
      address: http://localhost:3100
      # you can suggest your structure of config
      timestamp_field: timestamp
      message_field: message
      # I don’t know the purpose of the stream yet
      stream:
        - label: test_label
          value: test_value

File with input messages (data.json):

{"index": {"_index": "index-main","_type": "span"}}
{"timestamp":"2024-08-01T10:31:55.665748976Z","k8s_container":"container","message":"Started"}
{"index": {"_index": "index-main","_type": "span"}}
{"timestamp":"2024-08-01T10:35:55.665748976Z","k8s_container":"container","message":"Stopped"}

send to file.d

curl "127.0.0.1:9200/input" -v -H 'Content-Type: application/json' -d @data.json

And file.d sends to loki:

{
  "streams": [
    {
      "stream": {
        "test_label": "test_value"
      },
      "values": [
          [ "17225143070000000", "Started", {"k8s_container": "container"} ],
          [ "17225143080000000", "Stopped", {"k8s_container": "container"} ]
      ]
    }
  ]
}

Feel free for ask any questions!

[romanchechyotkin]

Hello, thanks for feedback, yesterday i found out that there is loki-go client by Grafana, i tried to create my wheel as Loki HTTP client, may be rewrite using this library?
https://github.com/grafana/loki-client-go
I can see the the phrase DONT USE IT by authors, but this client is used by this library https://github.com/samber/slog-loki

What do you think, rewrite using library or make a little bit refactoring

[romanchechyotkin]

but what do you thnk about current logs output

[DmitryRomanov]

Hello, thanks for feedback, yesterday i found out that there is loki-go client by Grafana, i tried to create my wheel as Loki HTTP client, may be rewrite using this library? https://github.com/grafana/loki-client-go I can see the the phrase DONT USE IT by authors, but this client is used by this library https://github.com/samber/slog-loki

What do you think, rewrite using library or make a little bit refactoring

I think it's better to not use libraries (to avoid using dependencies) and make some fixes.

[DmitryRomanov]

but what do you thnk about current logs output

looks good to me. try searching by labels and others logql examples

[romanchechyotkin]

all logs

select where kind=normal22

[DmitryRomanov]

Here is example of logs from Promtail to Loki. We have fields app, container, node_name, pod and you can search by them. And our k8s input plugin adds those labels as well.