Logstash-loki plugin not sending logs to Loki (handshake_failure)

[FelixRodriguezJara]

Describe the bug
Logstash-loki output plugin is not sending logs to Loki endpoint. Thus, Loki is not indexing nor showing the logs in the Grafana console.

To Reproduce

1. Started logstash image with loki plugin installed:

FROM logstash:6.8.1
RUN bin/logstash-plugin install logstash-output-loki

2. Pipeline configuration is the following:

logstash.conf: |
    input {
        rabbitmq {
         [...]
        }
    }
    filter {
      mutate {
        add_field => {
         "cluster" => "nonprod-eu"
         }
      }
    }
    output {
        loki {
            url => "${LOKI_URL}"
            username => "${LOKI_USERNAME}"
            password => "${LOKI_PASSWORD}"
            tenant_id => "logstash-external-ti"
            # batch_size => 112640 #112.64 kilobytes
            batch_size => 256 #256 bytes - low for test
            retries => 5
            min_delay => 3
            max_delay => 500
            message_field => "message"
        }
        stdout { codec => rubydebug }
    }

3. Stdout prints traces like the following one:

{
    "container_name" => "/gitlab",
    "container_id" => "0da347b1ca346282bf1fa864e86ab6d13e92a19ee31b643f613f1bb61405362a",
    "@version" => "1",
    "source" => "stdout",
    "log" => "10.238.1.214 - - [11/Aug/2021:13:33:46 +0000] \"POST /api/v4/jobs/request HTTP/1.1\" 204 0 \"\" \"gitlab-runner 13.8.0 (13-8-stable; go1.13.8; darwin/amd64)\" -",
    "cluster" => "nonprod-eu",
    "message" => "test",
    "service" => "test-gitlab",
    "@timestamp" => 2021-08-11T13:33:49.687Z
}

3. Query: {service="test-gitlab"} and no results are shown.

4. We have tested using curl command on the loki URL and the logs flow in.

Expected behavior
Logs are sent to Loki and appear in Grafana.

Environment:

• Kubernetes (EKS on AWS)

[FelixRodriguezJara]

From time to time, we see errors like the following one, regarding handshake:

[2021-08-11T16:16:11,036][WARN ][logstash.outputs.loki    ] Failed to send batch, attempt: 1/10 {:error_inspect=>"#<OpenSSL::SSL::SSLError: Received fatal alert: handshake_failure>", :error=>#<OpenSSL::SSL::SSLError: Received fatal alert: handshake_failure>}

This is happening even when insecure_skip_verify is set to true in the loki output configuration.

[FelixRodriguezJara]

Handshake_error is gone after upgrading logastash to 6.8.18.

Thank you very much for your support.

[rickydjohn]

@FelixRodriguezJara, did you add any additional changes to the configuration to get this to work. I am experiencing a similar issue and my settings are pretty much identical. I still dont see the logs over in loki. I am using a much later version of logstash (docker.elastic.co/logstash/logstash:7.9.3)

[FelixRodriguezJara]

Hi @rickydjohn! I did not add any configuration changes, just upgraded to the latest version and it started working. If you have upgraded, perhaps it makes sense to check whether loki gateway or ingester is discarding any log events for any other reason.