[AIRFLOW-6014] - handle pods which are preempted and deleted by kuber…

[atrbgithub]

…netes but not restarted

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title.

Description

• Here are some details about my PR, including screenshots of any UI changes:
  This PR addresses the issue of when a pod is Preempted during the creation phase and due to pods having the following in the spec restartPolicy: Never The pod is never restarted and ends up as a queued task within Airflow until the scheduler is restarted.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  Unsure if it is possible to simulate this scenario.

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[atrbgithub]

@dimberman would it be possible to take a look at this?

[kaxil]

cc @dimberman

[inytar]

We are running a fork of Airflow with a similar fix, really hoping this gets merged on upstream!

[dimberman]

LGTM. I put in a fix for a new namespace awareness and if tests pass I will merge. Apologies for the delay this one fell through the cracks :/

[codecov-io]

Codecov Report

Merging #6606 into master will decrease coverage by 0.25%.
The diff coverage is 89.85%.

@@            Coverage Diff             @@
##           master    #6606      +/-   ##
==========================================
- Coverage   86.96%   86.71%   -0.26%     
==========================================
  Files         915      915              
  Lines       44188    44201      +13     
==========================================
- Hits        38429    38328     -101     
- Misses       5759     5873     +114     

Impacted Files	Coverage Δ
airflow/executors/kubernetes_executor.py	56.63% <0.00%> (-0.36%)	⬇️
airflow/models/taskinstance.py	94.70% <0.00%> (-0.28%)	⬇️
airflow/providers/google/cloud/hooks/base.py	96.27% <97.29%> (+0.37%)	⬆️
...rflow/providers/google/cloud/operators/bigquery.py	91.37% <100.00%> (-0.12%)	⬇️
...roviders/google/cloud/operators/gcs_to_bigquery.py	92.95% <100.00%> (+22.36%)	⬆️
...viders/google/cloud/operators/kubernetes_engine.py	97.18% <100.00%> (-0.08%)	⬇️
...oviders/google/cloud/utils/credentials_provider.py	91.83% <100.00%> (-1.39%)	⬇️
airflow/providers/qubole/operators/qubole.py	87.69% <100.00%> (-0.88%)	⬇️
airflow/utils/process_utils.py	76.53% <100.00%> (+3.27%)	⬆️
airflow/kubernetes/volume_mount.py	44.44% <0.00%> (-55.56%)	⬇️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a54512...4151877. Read the comment docs.

[boring-cyborg]

Awesome work, congrats on your first merged pull request!