[AIRFLOW-5819] Update AWSBatchOperator default value

[domantasjurkus]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-XXX
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

• Here are some details about my PR, including screenshots of any UI changes:

The default value None throws an error, should be changed to {}.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

I am actually not sure why the issue was not captured in the existing test:

airflow/tests/contrib/operators/test_awsbatch_operator.py

Line 39 in 73bf718

self.batch = AWSBatchOperator(

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

[codecov-io]

Codecov Report

Merging #6473 into master will decrease coverage by 0.2%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #6473      +/-   ##
==========================================
- Coverage   83.82%   83.62%   -0.21%     
==========================================
  Files         635      635              
  Lines       36657    36716      +59     
==========================================
- Hits        30727    30702      -25     
- Misses       5930     6014      +84

Impacted Files	Coverage Δ
airflow/contrib/operators/awsbatch_operator.py	78.4% <100%> (ø)	⬆️
airflow/operators/postgres_operator.py	0% <0%> (-100%)	⬇️
airflow/kubernetes/volume_mount.py	44.44% <0%> (-55.56%)	⬇️
airflow/kubernetes/volume.py	52.94% <0%> (-47.06%)	⬇️
airflow/kubernetes/pod_launcher.py	45.25% <0%> (-46.72%)	⬇️
airflow/kubernetes/kube_client.py	33.33% <0%> (-41.67%)	⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py	70.14% <0%> (-28.36%)	⬇️
airflow/hooks/postgres_hook.py	92.85% <0%> (-3.64%)	⬇️
airflow/operators/subdag_operator.py	94.89% <0%> (-1.88%)	⬇️
airflow/jobs/backfill_job.py	89.9% <0%> (-1.53%)	⬇️
... and 34 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 58060d3...3eb4cb3. Read the comment docs.

[ashb]

You shouldn't do it like this. https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments

Instead in the method do

        self.array_properties = array_properties if array_properties is not None else {}

[domantasjurkus]

Thanks for pointing out, updated.

[kaxil]

Can you fix the failing tests too:

======================================================================
7) FAIL: test_execute_with_failures (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/operators/test_awsbatch_operator.py line 102 in test_execute_with_failures
      arrayProperties=None
    /usr/local/lib/python3.6/unittest/mock.py line 825 in assert_called_once_with
      return self.assert_called_with(*args, **kwargs)
    /usr/local/lib/python3.6/unittest/mock.py line 814 in assert_called_with
      raise AssertionError(_error_message()) from cause
   AssertionError: Expected call: submit_job(arrayProperties=None, containerOverrides={}, jobDefinition='hello-world', jobName='51455483-c62c-48ac-9b88-53a6a725baa3', jobQueue='queue')
   Actual call: submit_job(arrayProperties={}, containerOverrides={}, jobDefinition='hello-world', jobName='51455483-c62c-48ac-9b88-53a6a725baa3', jobQueue='queue')
   -------------------- >> begin captured logging << --------------------
   airflow.task.operators: INFO: Running AWS Batch Job - Job definition: hello-world - on queue queue
   airflow.task.operators: INFO: AWSBatchOperator overrides: {}
   airflow.task.operators: INFO: AWS Batch Job started: 
   airflow.task.operators: INFO: AWS Batch Job has failed executed
   --------------------- >> end captured logging << ---------------------
======================================================================
8) FAIL: test_execute_without_failures (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator)
----------------------------------------------------------------------
   Traceback (most recent call last):
    /usr/local/lib/python3.6/unittest/mock.py line 1183 in patched
      return func(*args, **keywargs)
    tests/contrib/operators/test_awsbatch_operator.py line 81 in test_execute_without_failures
      arrayProperties=None
    /usr/local/lib/python3.6/unittest/mock.py line 825 in assert_called_once_with
      return self.assert_called_with(*args, **kwargs)
    /usr/local/lib/python3.6/unittest/mock.py line 814 in assert_called_with
      raise AssertionError(_error_message()) from cause
   AssertionError: Expected call: submit_job(arrayProperties=None, containerOverrides={}, jobDefinition='hello-world', jobName='51455483-c62c-48ac-9b88-53a6a725baa3', jobQueue='queue')
   Actual call: submit_job(arrayProperties={}, containerOverrides={}, jobDefinition='hello-world', jobName='51455483-c62c-48ac-9b88-53a6a725baa3', jobQueue='queue')
   -------------------- >> begin captured logging << --------------------
   airflow.task.operators: INFO: Running AWS Batch Job - Job definition: hello-world - on queue queue
   airflow.task.operators: INFO: AWSBatchOperator overrides: {}
   airflow.task.operators: INFO: AWS Batch Job started: {'jobName': '51455483-c62c-48ac-9b88-53a6a725baa3', 'jobId': '8ba9d676-4108-4474-9dca-8bbac1da9b19'}
   airflow.task.operators: INFO: AWS Batch Job has been successfully executed: {'jobName': '51455483-c62c-48ac-9b88-53a6a725baa3', 'jobId': '8ba9d676-4108-4474-9dca-8bbac1da9b19'}
   --------------------- >> end captured logging << ---------------------
======================================================================
9) FAIL: test_init (tests.contrib.operators.test_awsbatch_operator.TestAWSBatchOperator)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/operators/test_awsbatch_operator.py line 56 in test_init
      self.assertEqual(self.batch.array_properties, None)
   AssertionError: {} != None