Skip to content

Latest commit

 

History

History
84 lines (67 loc) · 2.91 KB

README.md

File metadata and controls

84 lines (67 loc) · 2.91 KB
Raw

eqp - elasticsearch query as prometheus metrics

Queries Elasticsearch for patterns and exposes them as metrics to prometheus

With this you can generate Alertmanager notifications based on log entries

configuration

eqp is configured using the below environment variables

Option Description Default
ELASTICSEARCH_HOST Elastic URL https://elasticsearch-es-http.elastic-system.svc:9200
ELASTICSEARCH_USER Elastic User elastic
ELASTICSEARCH_PASSWORD Elastic Pass changeme
FREQUENCY Frequency to check 20s
SKIP_VERIFY_TLS Skip TLS validation false
CONFIG_FILE File to read config from /etc/eqp.yaml

When running inside Kubernetes you can create a secret that works with the provided manifest as follows:

kubectl create secret generic elastic-user --from-literal password=changeme

configuration

an example configuration file is below. the example kubernetes deployment uses a ConfigMap to make this available

url: https://elasticsearch-es-http.elastic-system.svc:9200
username: elastic
password: changeme
insecure: false
frequency: 20s

matches:
- name: ErrorMessage
  pattern: .*[^.,_][Ee][Rr][Rr][Oo][Rr].*
  type: regexp
  index: log-syslog-serverlog-kubernetes*
- name: WarnMessage
  pattern: .*[Ww][Aa][Rr][Nn].*
  type: regexp
  index: log-syslog-serverlog-kubernetes*

Here's a PrometheusRule that triggers when a pattern is matched

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: eqp-rules
  namespace: kube-system
spec:
  groups:
  - name: eqp
    rules:
    - alert: LogAlert
      expr: eqp_log_scrape_matches > 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Pod {{ $labels.pod_name }} is matching pattern {{ $labels.pattern }}
        description: Check {{ $labels.pod_name }} in namespace {{ $labels.namespace }}

The below metrics are provided:

# HELP eqp_log_scrape_matches The total number of patterns matched
# TYPE eqp_log_scrape_matches gauge
eqp_log_scrape_matches{namespace="kafka",pattern="ruok",pod_name="kafka-zookeeper-0"} 11
eqp_log_scrape_matches{namespace="kafka",pattern="ruok",pod_name="kafka-zookeeper-1"} 11
eqp_log_scrape_matches{namespace="kafka",pattern="ruok",pod_name="kafka-zookeeper-2"} 11