[AIRFLOW-5218] less polling for AWS Batch status

[dazza-codes]

Jira

• My PR addresses the following [Airflow Jira]
  • https://issues.apache.org/jira/browse/AIRFLOW-5218
  • this PR is the smallest possible change that could help that issue

Description

• Here are some details about my PR, including screenshots of any UI changes:
  • a small increase in the backoff factor could avoid excessive polling
  • avoid the AWS API throttle limits for highly concurrent tasks
  • correct polling loop details
  • correct log messages, with more informative status details

See also:

• AWS batch spin-up time:
  • https://forums.aws.amazon.com/thread.jspa?messageID=897734
  • Depends on various scheduling intervals:
    • job scheduling interval
    • compute-environment scaling schedule interval
    • API calls to container registry (and container pulls)
  • bottom line, there are inconsistencies from 10-sec to 10-min spin-ups

Here are some quotes from that message forum link above (note that the forum comments are dated approx 2017 through to 2019 and improvements are released over time). AWS batch support says,

For example, if you submit a hundred jobs to AWS Batch, the Scheduler will transition all of these from SUBMITTED to RUNNABLE or PENDING in about a minute. RUNNABLE jobs should transition to STARTING and RUNNING fairly quickly assuming you have sufficient resources in your compute environment.

AWS batch support note that improvements are released over time,

For example, when we launched the service, we would schedule jobs every 1 minute. We now perform these operations every 10 seconds. It is however important to differentiate that this delay is seen only if there are no Submitted jobs in the JobQueue. Batch continues to transition and schedule jobs with no delay until the JobQueue has no Submitted or Runnable jobs. Thus, AWS Batch may take longer to schedule an individual job when Jobs are submitted infrequently. At scale, Batch can schedule jobs far more quickly.

But, when 100's of jobs require a scaling solution, the delays can be longer,

Further, it is important to note that the AWS Batch resource scaling decisions occur on a different frequency. Upon receiving your first job submission, AWS Batch will launch an initial set of compute resources. After this point Batch re-evaluates resource needs approximately every 10 minutes. By making scaling decisions less frequently, we avoid scenarios where AWS Batch would scale up too quickly and complete all RUNNABLE jobs, leaving a large number of unused instances with partially consumed billing hours.

Tests

• My PR does not need testing for this extremely good reason:
  • there are tests on the AWS BatchOperator already
  • this PR is an implementation detail in a private method
  • the change does not impact any public API

Commits

• My commits all reference Jira issues in their subject lines
  • just one commit

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • no new functionality
  • logs are more explicit than before

Code Quality

• Passes flake8

[dazza-codes]

Most of the CI builds are OK but one of them timed-out and I don't have access to restart it

• https://travis-ci.org/apache/airflow/builds/572411318

[dazza-codes]

Review requests addressed

• leaving one conversation to @ashb to consider/resolve
• considered application of black formatting, but skipped it (too many changes)