alertmanager-github-receiver

|

Not all alerts are an emergency. But, we want to track every one because alerts are always an actual problem. Either:

• an actual problem in the monitored system
• an actual problem in processes around the monitored system
• an actual problem with the alert itself

The alertmanager github receiver creates GitHub issues using Alertmanager webhook notifications.

Build

make docker DOCKER_TAG=repo/imageName

This will build the binary and push it to repo/imageName.

Setup

Create GitHub access token

The github receiver uses user access tokens to create issues in an existing repository.

Generate a new access token:

• Log into GitHub and visit https://github.com/settings/tokens
• Click the 'Generate new token' button
• Select the 'repo' scope and all subscopes of 'repo'

Because this access token has permission to create issues and operate on repositories the access token user can access, protect the access token as you would a password.

Start GitHub Receiver

To start the github receiver locally:

docker run -it measurementlab/alertmanager-github-receiver:latest
        -authtoken=$(GITHUB_AUTH_TOKEN) -org=<org> -repo=<repo>

Note: both the org and repo must already exist.

Configure Alertmanager Webhook Plugin

The Prometheus Alertmanager supports third-party notification mechanisms using the Alertmanager Webhook API.

Add a receiver definition to the alertmanager configuration.

- name: 'github-receiver-issues'
  webhook_configs:
  - url: 'http://localhost:9393/v1/receiver'

To publish a test notification by hand, try:

msg='{
  "version": "4",
  "groupKey": "fakegroupkey",
  "status": "firing",
  "receiver": "http://localhost:9393/v1/receiver",
  "groupLabels": {"alertname": "FoobarIsBroken"},
  "externalURL": "http://localhost:9093",
  "alerts": [
    {
      "labels": {"thing": "value"},
      "annotations": {"hint": "how to fix foobar"},
      "status": "firing",
      "startsAt": "2018-06-12T01:00:00Z",
      "endsAt": "2018-06-14T01:00:00Z"
    }
  ]
}'
curl -XPOST --data-binary "${msg}" http://localhost:9393/v1/receiver

Configuration

Auto close

If -enable-auto-close is specified, the program will close each issue as its corresponding alert is resolved. It searches for matching issues by filtering open issues on the value of -alertlabel and then matching issue titles. The issue title template can be overridden using -title-template-files for greater (or lesser) specificity. The default template is {{ .Data.GroupLabels.alertname }}, which sets the issue title to the alert name. The template is passed a Message as its argument.

Repository

If the alert includes a repo label, issues will be created in that repository, under the GitHub organization specified by -org. If no repo label is present, issues will be created in the repository specified by the -repo option.