[Feature] support prometheus-operator serviceMonitor and prometeusRules

CHANGELOG.md

@@ -3,8 +3,10 @@
 ## Breaking changes
 - When using HTTP POST 2, it is no longer necessary to pre-escape strings (should they contain control chars) from events in elastic search which are replaced by the jinja2 template.
 
+- [Kubernetes] [Breaking] reconfigure metrics to follow prometheus operator nomenclature. `metrics` value, now control the addition of metrics endpoint (command argument), the creation of a service to expose the metrics endpoint and the (optional) creation of prometheus-operator objects: serviceMonitor and prometheurRules to match implementations of other charts. The labels of the chart have been modified, so you'll need to uninstall and reinstall the chart for the upgrade to work. - [#902](https://github.com/jertel/elastalert2/pull/902) - @PedroMSantosD
+
 ## New features
-- None
+- [Kubernetes] Chart is now able to create a service for the metrics, and optional  prometheus-operator custom resources serviceMonitor and  prometheusRule. - [#902](https://github.com/jertel/elastalert2/pull/902) - @PedroMSantosD
 
 ## Other changes
 - Upgrade pylint 2.13.8 to 2.14.3, Upgrade sphinx 4.5.0 to 5.0.2 - [#891](https://github.com/jertel/elastalert2/pull/891) - @nsano-rururu

chart/elastalert2/README.md

@@ -96,5 +96,13 @@ The command removes all the Kubernetes components associated with the chart and
 | `tolerations`                                | Tolerations for deployment                                                                                                    | []                                                       |
 | `smtp_auth.username`                         | Optional SMTP mail server username. If the value is not empty, the smtp_auth secret will be created automatically.       | `NULL`                                                   |
 | `smtp_auth.password`                         | Optional SMTP mail server passwpord. This must be specified if the above field, `smtp_auth.username` is also specified.      | `NULL`                                                   |
-| `prometheusPort`                         | Optional TCP port to be used to expose prometheus metrics. if set: (1) it will pass the start parameter --prometheus_port to the command, (2) it will expose said TCP port on the POD and (3) It will add the pod annotation: prometheus.io/port: value to POD, for prometheus pod service discovery to pick the metrics      | `NULL`                                                   |
-| `prometheusScrapeAnnotations`                         | Optional Dict with the flags used by prometheus SD to know the scrape path and to keep the scrapted metrics. Note that this values are only rendered if prometheusPort is set | prometheusScrapeAnnotations: {prometheus.io/scrape: "true" prometheus.io/path: "/"}                                   |
+| `metrics`                         | enable elastalert prometheus endpoint, add prometheus.io annotations to pod and create a service pointing to the port for prometheus to scrape the metrics      | `false`                                                   |
+| `metrics.prometheusPort`                         | if "metrics" is set to true, CP port pod will expose prometheus metrics on.       | `8080`                                                   |
+| `metrics.prometheusPortName`                         | name of the port where metrics are exposed    | `http-alt`                                                   |
+| `metrics.prometheusScrapeAnnotations`                         | if metrics are enabled, annotations to add to the pod for prometheus configuration. prometheus.io/port is also added uring the prometheusPort and prometheusPortName values |  {prometheus.io/scrape: "true" prometheus.io/path: "/"}                                   |
+| `metrics.serviceMonitor.enabled`                         | If metrics are enabled, create a servicemonitor custom resource for prometheus-operator to detect and monitor the service with the merics endpoint  |  `false` |
+| `metrics.serviceMonitor.labels`                         | labels to add to the serviceMonitor object for prometheus-operator to detect and append it to your prometheus configuration, when deployed on a different namespas as the prometheus operator |  `{}` |
+| `metrics.serviceMonitor.metricRelabelings`                         | list of prometheus metric relabeling configs to aply to scrape. Example@ drop python_gc metrics or alter pod name |  `[]` |
+| `metrics.prometheusRule.enabled`                         | If metrics are enabled, create a prometheusRule custom resource for prometheus-operator to customise scrape configuration  |  `false` |
+| `metrics.prometheusRule.additionalLabels`                         | labels to add to the prometheusRule object for prometheus-operator to detect and append it to your prometheus configuration, when deployed on a different namespas as the prometheus operator |  `{}` |
+| `metrics.prometheusRule.rules`                         | group of rules to add to the prometheus configuration, example Alerting rules for pod down, or for file descriptors. Define as multiline Yaml string |  `[]` |

chart/elastalert2/templates/_labels.tpl

@@ -0,0 +1,18 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Kubernetes standard labels
+*/}}
+{{- define "common.labels.standard" -}}
+app.kubernetes.io/name: {{ include "common.names.name" . }}
+helm.sh/chart: {{ include "common.names.chart" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end -}}
+
+{{/*
+Labels to use on deploy.spec.selector.matchLabels and svc.spec.selector
+*/}}
+{{- define "common.labels.matchLabels" -}}
+app.kubernetes.io/name: {{ include "common.names.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end -}}

chart/elastalert2/templates/_names.tpl

@@ -0,0 +1,60 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "common.names.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "common.names.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "common.names.fullname" -}}
+{{- if .Values.fullnameOverride -}}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- if contains $name .Release.Name -}}
+{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+
+{{- define "common.names.servicename" -}}
+{{- $name := include "common.names.fullname" . | trunc 53 -}}
+{{- printf "%s-%s" $name "metrics" -}}
+{{- end -}}
+
+{{- define "common.names.configname" -}}
+{{- $name := include "common.names.fullname" . | trunc 53 -}}
+{{- printf "%s-%s" $name "config" -}}
+{{- end -}}
+
+{{/*
+Allow the release namespace to be overridden for multi-namespace deployments in combined charts.
+*/}}
+{{- define "common.names.namespace" -}}
+{{- if .Values.namespaceOverride -}}
+{{- .Values.namespaceOverride -}}
+{{- else -}}
+{{- .Release.Namespace -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+Create a fully qualified app name adding the installation's namespace.
+*/}}
+{{- define "common.names.fullname.namespace" -}}
+{{- printf "%s-%s" (include "common.names.fullname" .) (include "common.names.namespace" .) | trunc 63 | trimSuffix "-" -}}
+{{- end -}}

chart/elastalert2/templates/_tplvalues.tpl

@@ -0,0 +1,13 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Renders a value that contains template.
+Usage:
+{{ include "common.tplvalues.render" ( dict "value" .Values.path.to.the.Value "context" $) }}
+*/}}
+{{- define "common.tplvalues.render" -}}
+    {{- if typeIs "string" .value }}
+        {{- tpl .value .context }}
+    {{- else }}
+        {{- tpl (.value | toYaml) .context }}
+    {{- end }}
+{{- end -}}

chart/elastalert2/templates/config.yaml

@@ -2,12 +2,11 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: {{ template "elastalert.fullname" . }}-config
-  labels:
-    app: {{ template "elastalert.name" . }}
-    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
+  name: {{ template "common.names.configname" . }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+  {{- if .Values.commonLabels }}
+  {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+  {{- end }}
 data:
   elastalert_config: |-
     ---

chart/elastalert2/templates/deployment.yaml

@@ -3,33 +3,36 @@ kind: Deployment
 metadata:
   name: {{ template "elastalert.fullname" . }}
   labels:
+    {{- if .Values.commonLabels }}
+    {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+    {{- end }} 
     app: {{ template "elastalert.name" . }}
     chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
     release: {{ .Release.Name }}
     heritage: {{ .Release.Service }}
 spec:
   selector:
     matchLabels:
-      app: {{ template "elastalert.name" . }}
-      release: {{ .Release.Name }}
+      {{- include "common.labels.matchLabels" . | nindent 6 }}
   replicas: {{ .Values.replicaCount }}
   revisionHistoryLimit: {{ .Values.revisionHistoryLimit }}
   template:
     metadata:
       annotations:
         checksum/config: {{ include (print $.Template.BasePath "/config.yaml") . | sha256sum }}
         checksum/rules: {{ include (print $.Template.BasePath "/rules.yaml") . | sha256sum }}
-{{- if .Values.prometheusPort }}
-{{ toYaml .Values.prometheusScrapeAnnotations | indent 8 }}
-        prometheus.io/port: {{ .Values.prometheusPort | quote}}
+{{- if .Values.metrics.enabled  }}
+{{ toYaml .Values.metrics.prometheusScrapeAnnotations | indent 8 }}
+        prometheus.io/port: {{ .Values.metrics.prometheusPort | quote}}
 {{- end }}
 {{- if .Values.podAnnotations }}
 {{ toYaml .Values.podAnnotations | indent 8 }}
 {{- end }}
-      labels:
-        name: {{ template "elastalert.fullname" . }}-elastalert
-        app: {{ template "elastalert.name" . }}
-        release: {{ .Release.Name }}
+      labels: {{- include "common.labels.standard" . | nindent 8 }}
+        app.kubernetes.io/component: {{ .Values.appKubernetesIoComponent }}
+        {{- if .Values.commonLabels }}
+        {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+        {{- end }}
     spec:
 {{- if .Values.image.pullSecret }}
       imagePullSecrets:
@@ -44,10 +47,11 @@ spec:
       - name: elastalert
         image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
         imagePullPolicy: {{ .Values.image.pullPolicy }}
-{{- if .Values.prometheusPort }}
+{{- if .Values.metrics.enabled }}
         ports:
-        - containerPort: {{ .Values.prometheusPort }}
+        - containerPort: {{ .Values.metrics.prometheusPort }}
           protocol: TCP
+          name: {{ .Values.metrics.prometheusPortName }}
 {{- end }}
 {{- if .Values.securityContext }}
         securityContext:
@@ -58,13 +62,13 @@ spec:
 {{ toYaml .Values.command | indent 10 }}
 {{- end }}
 
-{{- if or .Values.args .Values.prometheusPort }}
+{{- if or .Values.args .Values.metrics.enabled }}
         args:
   {{- if .Values.args }}
 {{ toYaml .Values.args | indent 10 }}
   {{- end }}
-  {{- if .Values.prometheusPort }}
-  {{- $enableportlist := list "--prometheus_port" (.Values.prometheusPort | toString) }}
+  {{- if .Values.metrics.enabled }}
+  {{- $enableportlist := list "--prometheus_port" (.Values.metrics.prometheusPort | toString) }}
 {{ toYaml $enableportlist | indent 10 }}
   {{- end }}
 {{- end }}

chart/elastalert2/templates/podsecuritypolicy.yaml

@@ -3,11 +3,10 @@ apiVersion: policy/v1beta1
 kind: PodSecurityPolicy
 metadata:
   name: {{ template "elastalert.fullname" . }}
-  labels:
-    app: {{ template "elastalert.name" . }}
-    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+  {{- if .Values.commonLabels }}
+  {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+  {{- end }}
 spec:
   # Prevents running in privileged mode
   privileged: false

chart/elastalert2/templates/prometheusrule.yaml

@@ -0,0 +1,19 @@
+{{- if and .Values.metrics.enabled .Values.metrics.prometheusRule.enabled }}
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: {{ template "common.names.fullname" . }}
+  namespace: {{ default .Release.Namespace .Values.metrics.prometheusRule.namespace | quote }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+    {{- if .Values.metrics.prometheusRule.additionalLabels }}
+    {{- include "common.tplvalues.render" (dict "value" .Values.metrics.prometheusRule.additionalLabels "context" $) | nindent 4 }}
+    {{- end }}
+    {{- if .Values.commonLabels }}
+    {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+    {{- end }}
+  {{- if .Values.commonAnnotations }}
+  annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
+  {{- end }}
+spec:
+  {{- include "common.tplvalues.render" ( dict "value" .Values.metrics.prometheusRule.rules "context" $ ) | nindent 2 }}
+{{- end }}

chart/elastalert2/templates/role.yaml

@@ -3,11 +3,10 @@ apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: {{ template "elastalert.fullname" . }}
-  labels:
-    app: {{ template "elastalert.name" . }}
-    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+  {{- if .Values.commonLabels }}
+  {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+  {{- end }}
 rules:
   - apiGroups:
     - policy

chart/elastalert2/templates/rolebinding.yaml

@@ -3,11 +3,10 @@ apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
   name: {{ template "elastalert.fullname" . }}
-  labels:
-    app: {{ template "elastalert.name" . }}
-    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+  {{- if .Values.commonLabels }}
+  {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+  {{- end }}
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: Role

chart/elastalert2/templates/rules.yaml

@@ -2,11 +2,10 @@ apiVersion: v1
 kind: ConfigMap
 metadata:
   name: {{ template "elastalert.fullname" . }}-rules
-  labels:
-    app: {{ template "elastalert.name" . }}
-    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+  {{- if .Values.commonLabels }}
+  {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+  {{- end }}
 data:
 {{- range $key, $value := .Values.rules }}
 {{ $key | indent 2}}: |-

chart/elastalert2/templates/service.yaml

@@ -0,0 +1,43 @@
+{{- if .Values.metrics.enabled -}}
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "common.names.servicename" . | quote  }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+    app.kubernetes.io/component: {{ .Values.appKubernetesIoComponent}}
+    {{- if .Values.commonLabels }}
+    {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+    {{- end }}    
+  annotations:
+    {{- if .Values.commonAnnotations }}
+    {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
+    {{- end }}
+spec:
+  type: {{ .Values.metrics.service.type }}
+  {{- if and .Values.metrics.service.clusterIP (eq .Values.metrics.service.type "ClusterIP") }}
+  clusterIP: {{ .Values.metrics.service.clusterIP }}
+  {{- end }}
+  {{- if ne .Values.metrics.service.type "ClusterIP" }}
+  externalTrafficPolicy: {{ .Values.metrics.service.externalTrafficPolicy }}
+  {{- end }}
+  {{- if and .Values.metrics.service.loadBalancerIP (eq .Values.metrics.service.type "LoadBalancer") }}
+  loadBalancerIP: {{ .Values.metrics.service.loadBalancerIP }}
+  {{- end }}
+  {{- if and (eq .Values.metrics.service.type "LoadBalancer") .Values.metrics.service.loadBalancerSourceRanges }}
+  loadBalancerSourceRanges: {{- toYaml .Values.metrics.service.loadBalancerSourceRanges | nindent 4 }}
+  {{- end }}
+  ports:
+    - port: {{ .Values.metrics.prometheusPort }}
+      targetPort: {{ .Values.metrics.prometheusPort }}
+      protocol: TCP
+      name: {{ .Values.metrics.prometheusPortName }}
+      {{- if and (or (eq .Values.metrics.service.type "NodePort") (eq .Values.metrics.service.type "LoadBalancer")) .Values.metrics.service.nodePorts }}
+      nodePort: {{ .Values.metrics.service.nodePorts }}
+      {{- else if eq .Values.metrics.service.type "ClusterIP" }}
+      nodePort: null
+      {{- end }}
+
+  selector:
+    {{- include "common.labels.matchLabels" . | nindent 4 }}
+    app.kubernetes.io/component: {{ .Values.appKubernetesIoComponent }}
+{{- end }}

chart/elastalert2/templates/serviceaccount.yaml

@@ -3,11 +3,10 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: {{ include "elastalert.serviceAccountName" . }}
-  labels:
-    app: {{ template "elastalert.name" . }}
-    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+  {{- if .Values.commonLabels }}
+  {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
+  {{- end }}
   {{- with .Values.serviceAccount.annotations }}
   annotations:
     {{- toYaml . | nindent 4 }}