Fix more PodMutationHook issues for backwards compatibility

[kaxil]

This PR/commit

• Adds missing affinity from old POD
• Adds comprehensive tests to check pod_mutation_hook works well with both new and old PODs with various configs like volume, volumeMounts, Ports, affinity, tolerations etc
• Refactors various parts of k8s code

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[kaxil]

cc @aneesh-joseph Thanks for testing the last fix, can you please help me by testing this PR too -- Can you run your pod_mutations hooks with this PR, please (both pre 1.10.11 and 1.10.11 behaviors) ?

[aneesh-joseph]

cc @aneesh-joseph Thanks for testing the last fix, can you please help me by testing this PR too -- Can you run your pod_mutations hooks with this PR, please (both pre 1.10.11 and 1.10.11 behaviors) ?

sure, will test it out and slack you

[dimberman]

addresses #9827 as well

[dimberman]

│                                                                                      │
│ Traceback (most recent call last):                                                   │
│   File "/home/airflow/.local/bin/airflow", line 25, in <module>                      │
│     from airflow.configuration import conf                                           │
│   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line  │
│     from airflow.utils.log.logging_mixin import LoggingMixin                         │
│   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/__init__.py", │
│     from .decorators import apply_defaults as _apply_defaults                        │
│   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/decorators.py │
│     from airflow import settings                                                     │
│   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line  │
│     from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG  # NOQA F │
│   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/configuration.py",  │
│     with open(AIRFLOW_CONFIG, 'w') as f:                                             │
│ IsADirectoryError: [Errno 21] Is a directory: '/opt/airflow/airflow.cfg'             │
│ stream closed                                                                        │
│

that is causing k8s tests to fail (this only happens on the workers, not the scheduler)

[dimberman]

scheduler

│   config:                                                                            │
│     Type:      ConfigMap (a volume populated by a ConfigMap)                         │
│     Name:      airflow-airflow-config                                                │
│     Optional:  false

worker

   airflow-config:                                                                    │
│     Type:       EmptyDir (a temporary directory that shares a pod's lifetime)        │
│     Medium:                                                                          │
│     SizeLimit:  <unset>

for some reason the workers are mounting the cfg as an emptydir

[dimberman]

@kaxil so the tasks are completing correctly but then the pod goes into a crash loop and I'm not sure why. Will confer with you in the morning.

[dimberman]

@kaxil everything is working except those random mongo_hook tests. Can you take a look? After that we just need to refactor.

At the very least k8s stuff is WAY better tested now :)

[kaxil]

@kaxil everything is working except those random mongo_hook tests. Can you take a look? After that we just need to refactor.

At the very least k8s stuff is WAY better tested now :)

Aneesh has reported that pod_mutation_hook isn't working for pre-1.10.11

[kaxil]

A big thanks and a shoutout to @aneesh-joseph for helping us with testing that both pre-1.10.11 and 1.10.11 behaviours are maintained for 1.10.12. 🚀

[kaxil]

Please also refer to c230156 for all the changes in 1.10.12