[AIRFLOW-5027] Get logs from CloudWatch for ECSOperators

[StevenReitsma]

My first Airflow PR so I hope I adhered to all the contribution guidelines. All comments welcome.

---

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-5027

Description

• Here are some details about my PR, including screenshots of any UI changes:

It's very useful to have the CloudWatch task logs of an ECS/SageMaker job available within Airflow. Currently, this is not possible for ECS tasks.

The SageMaker operator starts a job on AWS SageMaker. It already supports grabbing the CloudWatch logs of the (finished) job to the Airflow instance.

The ECS operator is very similar in that it starts a job on AWS ECS. It doesn't, however, support grabbing CloudWatch logs at the end of a job.

I've separated the existing log grabbing of the SageMaker hook to a separate logs hook and both the SageMaker hook and ECS operator now call that logs hook at the end of a job.

Tests

• My PR adds the following unit tests:

Added: tests.contrib.hooks.TestAwsLogsHook
Changed: tests.contrib.hooks.TestSageMakerHook
Changed: tests.contrib.sensors.TestSageMakerTrainingSensor

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

• Passes flake8

[potiuk]

@StevenReitsma please rebase and push so that Travis can build it. Travis had a problem whole day yesterday.

[StevenReitsma]

Could someone take a look at this PR?

[StevenReitsma]

PTAL @ashb

[ashb]

Few more small changes please.

[StevenReitsma]

Thanks for the reviews so far @ashb, PTAL :)

[StevenReitsma]

@ashb Made the change and Travis is green again. PTAL :)