spark-on-k8s sensor logs - properly pass defined namespace to pod log call

[bbenzikry]

This is a follow up to #10023 - it seems that passing the defined namespace to the log call was missed in one of the rebases done during the PR.
Without this fix, logging will fail when the k8s connection uses a different namespace than the one SparkApplication(s) are actually submitted to.

[mik-laj]

Can you add tests to prevent regression?

[bbenzikry]

Hi @mik-laj , I changed the code to get both the pod name and namespace from the returned spec, so it should already be covered with existing tests as create_namespaced_custom_object will throw if namespaces aren't equivalent.

[ashb]

E           AssertionError: Expected call: get_pod_logs('spark_pi-driver')
E           Actual call: get_pod_logs('spark-pi-2020-02-24-1-driver', namespace='default')

You need to update the unittests.

[bbenzikry]

@ashb updated

[kaxil]

Can you please rebase your PR on latest Master since we have applied Black and PyUpgrade on Master.

It will help if your squash your commits into single commit first so that there are less conflicts.

[bbenzikry]

@kaxil Done, sorry for only getting to it now

[github-actions]

The PR should be OK to be merged with just subset of tests as it does not modify Core of Airflow. The committers might merge it or can add a label 'full tests needed' and re-run it to run all tests if they see it is needed!