Use DeadLetterQueue for alarms on ScheduledLambda

[freenerd]

Right now ScheduledLambda alarms on every single error of a lambda function. I propose to change this to never alarm on errors but instead use a Dead Letter Queue and alarm whenever something lands in there. This will make ScheduledLambda alarms less noisy to transient errors, while still retaining the visibility into persistent problems.

https://docs.aws.amazon.com/lambda/latest/dg/dlq.html

[dputtick]

@freenerd would you be ok with making this configurable? I can think of some ScheduledLambdas I've written where I do want to get paged every time they fail

[npeternel]

I used a DLQ implementation for a daily Scheduled Lambda so that we are only alarmed on three failed invocations and it has worked well so far. The only downside is manually cleaning up the messages in the queue after resolving the errors.

Following up to the above comment, the Error Alarm should also be configurable - I currently manually delete the Error Alarm in favor of the DLQ.