Add lambdas to handle cloudwatch alarms

[JustinPihony]

Closes #2117

---

This change is 

[JustinPihony]

Note that the 3 lambda_function.py are the real code to review. The rest is needed to import requests in the python lambda. We could zip it up and add a note how to deploy if it would be easier, though. Or even how to upgrade

[JustinPihony]

Once a timeline to trigger an alarm is determined, the only thing that needs done is some code cleanup and to set the schedules for the cleanup and creation lambdas to trigger. Probably ok to run once every hour? In testing they run in < 15 seconds.

[JustinPihony]

OK, this is ready for review/install. Current status:

• cloudwatchAllInstances will iterate through all running instances in a region, turn on monitoring if not already, add a cpu idle watch to go off after 6 hours of idle, and check if the instance has been running for 5 or more days (re-notifying every 5th day thereafter)
• idleNotifier is the lambda that talks to Slack. The app I created talks to #ec2-idle-notification
• cleanOrphanCloudWatch is the lambda to search for all cloudwatch alarms which were generated by cloudwatchAllInstances (naming convention), but the instance is no longer running. I figure even if they are only stopped, then a new alarm will be added if they re-start.

I'm planning on setting up a cloudwatch rule to trigger every 30 minutes to run cloudWatchAllInstances and cleanOrphanCloudWatch. They are fairly light weight, so shouldn't be much burden, as well as make sure to notify/cleanup sooner than later. Additionally, I will have the lambdas check us-east-1, us-east-2, us-west-1, and us-west-2 (any others?).

If any of that sounds off then let me know and I can tweak. Otherwise I'll install into us-east-2 once approved - since that is where all other lambdas live.

Oh, and should we @here to get more attention to that channel, or we'll know it is important? It shouldn't be as chatty as the other automated channels

[REASY]

@JustinPihony so many files, is it expected? There is no way to bring dependencies via pip?

[JustinPihony]

Not that I found - this is the suggested route from an amazon official document. They suggest zipping up the contents (including the lambda handler) and uploading. I am open to other approaches, but I could not find anything other than this. I didn't pre-zip it because I wanted to let it all be zipped together, but we can zip and make a note on how to update if needed. I'm open to suggestions :)

…

On Tue, Oct 15, 2019 at 10:09 AM REASY ***@***.***> wrote: @JustinPihony <https://github.com/JustinPihony> so many files, is it expected? There is no way to bring dependencies via pip? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2222?email_source=notifications&email_token=AAML6X4FNMHMNKWZ3KKU723QOXFIHA5CNFSM4JAK3CKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBI4VEQ#issuecomment-542231186>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAML6X7RAA27NRJJFKGBQFDQOXFIHANCNFSM4JAK3CKA> .

-- Justin Pihony *Developer Support Manager, Lightbend, Inc.* [email protected] <https://www.lightbend.com>

[REASY]

@JustinPihony you're right. Now I know that people even upload the whole binary executable. I've learned something new today :D

[wrashid]

I am assuming we are filtering out small instances?

[JustinPihony]

No, we are not. Wouldn't it be better to keep an eye on anything? I figure we can easily add an ignore instance list if needed.

…

On Tue, Oct 15, 2019 at 6:41 PM Rashid Waraich ***@***.***> wrote: I am assuming we are filtering out small instances? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2222?email_source=notifications&email_token=AAML6X5KXRNELIVJAQJWLDDQOZBJPA5CNFSM4JAK3CKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBKON6I#issuecomment-542435065>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAML6X3G3462EFIRPVKZJBDQOZBJPANCNFSM4JAK3CKA> .

-- Justin Pihony *Developer Support Manager, Lightbend, Inc.* [email protected] <https://www.lightbend.com>

[wrashid]

ok, need to be able to at least exclude specific machines which need to be running all the time

[JustinPihony]

@wrashid Fixed via 455bc48

[JustinPihony]

For some reason not all instances are being picked up in us-east-2. I had only tested for one instance, so maybe it is not properly iterating through all instances. I'll debug tonight, see what's going on and update accordingly.

[JustinPihony]

test!