[AIRFLOW-5385] spark hook does not work on spark 2.3/2.4

[tooptoop4]

• Description above provides context of the change
• Commit message contains [AIRFLOW-5385] or [AIRFLOW-5385] for document-only changes
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.
  In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
  In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
  In case of backwards incompatible changes please leave a note in UPDATING.md.
  Read the Pull Request Guidelines for more information.

[tooptoop4]

@potiuk this is same as #6911 but using standard assert

[codecov-io]

Codecov Report

Merging #6976 into master will decrease coverage by 0.19%.
The diff coverage is 85.71%.

@@            Coverage Diff            @@
##           master    #6976     +/-   ##
=========================================
- Coverage   84.74%   84.54%   -0.2%     
=========================================
  Files         679      679             
  Lines       38495    38504      +9     
=========================================
- Hits        32622    32553     -69     
- Misses       5873     5951     +78

Impacted Files	Coverage Δ
airflow/contrib/hooks/spark_submit_hook.py	84.61% <85.71%> (+2.54%)	⬆️
airflow/kubernetes/volume_mount.py	44.44% <0%> (-55.56%)	⬇️
airflow/kubernetes/volume.py	52.94% <0%> (-47.06%)	⬇️
airflow/kubernetes/pod_launcher.py	45.25% <0%> (-46.72%)	⬇️
airflow/kubernetes/refresh_config.py	50.98% <0%> (-23.53%)	⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py	78.75% <0%> (-20%)	⬇️
airflow/jobs/backfill_job.py	91.88% <0%> (+0.28%)	⬆️
airflow/hooks/postgres_hook.py	94.36% <0%> (+1.4%)	⬆️
airflow/hooks/dbapi_hook.py	91.73% <0%> (+1.65%)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 991b34d...46ecb36. Read the comment docs.

[potiuk]

Looks good. You just need to rebase :)

[tooptoop4]

@potiuk conflict resolved, pls merge

[potiuk]

Thanks @tooptoop4 !