[AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming

[KevinYang21]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title.
  • https://issues.apache.org/jira/browse/AIRFLOW-4956

Description

• Here are some details about my PR, including screenshots of any UI changes:

If there's an exception in LocalTaskJob, e.g. DB connectivity error, the job will not wait before attempting the next heartbeat, causing the job to spam retry attempts and task logging.

This PR will unify the usage of BaseJob.heartbeat() so that exceptions are handled all inside the method instead of caller.

This PR also replaces local_task_job_heartbeat_fail metric with lower(JobClassName)_heartbeat_failure metrics so all jobs can have StatsD metrics on heartbeat failures.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  tests/jobs/test_local_task_job.py. test_heartbeat_failed_fast

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[ashb]

This will also include the stack trace of the exception - was that intended?

[KevinYang21]

Ya, already printing it previously and just a small touch up to use log.exception( also removed the redundant str(e) in the draft).

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@3ee2dcb). Click here to learn what that means.
The diff coverage is 75%.

@@            Coverage Diff            @@
##             master    #5589   +/-   ##
=========================================
  Coverage          ?   79.06%           
=========================================
  Files             ?      489           
  Lines             ?    30681           
  Branches          ?        0           
=========================================
  Hits              ?    24259           
  Misses            ?     6422           
  Partials          ?        0

Impacted Files	Coverage Δ
airflow/jobs/base_job.py	87.05% <0%> (ø)
airflow/jobs/local_task_job.py	90% <100%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ee2dcb...1be47ef. Read the comment docs.

[KevinYang21]

@ashb Updated accordingly, PTAL 🙏

[ashb]

Looks good once you add example to updating, or do something to maintain the existing metric format.

[KevinYang21]

@ashb Updated with backward compatible approach, PTAL

[ashb]

(Will merge once we fix master tests)

[KevinYang21]

@ashb seems like recent two commits are passing master CI. Is master CI fixed already?