Manual alert for routes and receivers test

[vpbarb]

Now I must wait for real alert to test if I configured routes or receivers correctly.
It would be great to add possibility of manual alert trigger to debug/test routes and receivers. It may be added on "Status" page as a form with fields for Status, Labels and Annotations (and maybe other fields from model.Alert). It may work via API method addAlerts.

[fabxc]

If you scroll to the bottom of the configuration in the web UI you should see a visualizer where you can enter label set, resembling alerts, and see where they end up.

The same editor is also available here: https://prometheus.io/webtools/alerting/routing-tree-editor/

So much for the routing aspect. Actually sending something to configured receivers is not part of that. But you can always send some alerts directly to the Alertmanager API via curl.

[acdha]

@fabxc Re: “But you can always send some alerts directly to the Alertmanager API via curl.” – what's the best way to get the right format payload for testing? I was trying to adjust the formatting for email notifications and it would have been really helpful if alertmanager either had a web UI to resend an active alert or simply to copy the last n alert payloads as JSON so you could tweak & replay.

[davidwebber]

I also wanted to test alert message formatting. Here's a sample curl command to trigger an alert. You can see the full tags of a message from prometheus by stopping the alertmanager and running "nc -k -l 9093" in it's place to grab incoming messages.

curl -H "Content-Type: application/json" -d '[{"labels":{"alertname":"TestAlert1"}}]' localhost:9093/api/v1/alerts

[r4j4h]

To add to @davidwebber 's curl example here's a Gist I found that describes some other fields, namely 'status: firing/resolved' that may enrich other's testing efforts. Also shout out to the wonderful netcat approach of grabbing messages! Just figured this may save some people some effort.

https://gist.github.com/cherti/61ec48deaaab7d288c9fcf17e700853a

[tangr]

for my using alertmanager version:0.8.0, the resolved alert is not tagged by status:resolved but endsAt.

just like:

#!/bin/bash

name=$RANDOM
url='http://localhost:9093/alertmanager/api/v1/alerts'

echo "firing up alert $name"

curl -XPOST $url -d '[{
"labels": {
"alertname": "$name",
"service": "my-service",
"severity": "warning",
"instance": "$name.example.net"
},
"annotations": {
"summary": "High latency is high!"
},
"generatorURL": "http://prometheus.int.example.net/<generating_expression>"
}]'

echo ""

echo "press enter to resolve alert"
read

echo "sending resolve"
curl -XPOST $url -d '[{
"status": "resolved",
"labels": {
"alertname": "$name",
"service": "my-service",
"severity":"warning",
"instance": "$name.example.net"
},
"annotations": {
"summary": "High latency is high!"
},
"generatorURL": "http://prometheus.int.example.net/<generating_expression>",
"startsAt": '"$(TZ=UTC date -d -1hour +"%FT%T.%3NZ")"',
"endsAt": '"$(TZ=UTC date -d -10mins +"%FT%T.%3NZ")"'
}]'

echo ""

[prologic]

I have a similar shell script:

#!/bin/bash

alerts='[
  {
    "labels": {
       "alertname": "instance_down",
       "instance": "example1"
     },
     "annotations": {
        "info": "The instance example1 is down",
        "summary": "instance example1 is down"
      }
  }
]'

URL="https://alertmanager.mydomain.com"

curl -XPOST -d"$alerts" $URL/api/v1/alerts

In general however it would be nice if we shipped a nice CLI tool to trigger alerts more easily. Other use-case might actually be via shell scripts or other processes.

[stuartnelson3]

This should be relatively simple to add to client/alert.go using the functionality provided in cli/client.go.

[sipian]

@stuartnelson3
Do correct me if I'm wrong but does'nt cli/alert_add.go add this feature in the amtool ?

[stuartnelson3]

It does. Perhaps this issue can be closed then?

[vpbarb]

As creator of this issue I will close it. Thanks, guys!