Make K8sPodOperator backwards compatible

[dimberman]

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[ashb]

Just discussed with Daniel: We're going to add (back) a airflow/kubernetes/volume_mount.py to "core" airflow that contains something like:

from airflow.providers.cncf.kubernetes.backcompat.volume_mount import VolumeMount

warnings.warn(
    "This module is deprecated. Please use `kubernetes.client.models.V1Volume`.",
    DeprecationWarning,
    stacklevel=2,
)

We may need to wrap that import in a `warnings.catch_warnings, otherwise the warning will be reported from the wrong context (we ideally want the file:lineno of the warning to be the import line in the user's DAG):

import warnings

with warnings.catch_warnings():
    from airflow.providers.cncf.kubernetes.backcompat.volume_mount import VolumeMount

warnings.warn(
    "This module is deprecated. Please use `kubernetes.client.models.V1Volume`.",
    DeprecationWarning,
    stacklevel=2,
)

This means the old imports continue to work, but the bulk of the compat code lives in the provider.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[potiuk]

Love it!

[potiuk]

Great job @dimberman and @ashb . I think we are nicely handling all cases :). I think we might merge it just in time to release new backports :)

[github-actions]

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!

[ashb]

Waaaaait -- just noticed that 1.10.x still has an airflow/kubernetes/pod.py etc. What is this going to do?

[potiuk]

Yeah. They should be removed after rename. However Interesting thing is that the package installed correctly (prepare backport providers job in CI does it).

[ashb]

Yeah, pip doesn't really care if two distributions install the same file -- it'll just happily overwrite the existing file.

Which is fine right until you come to uninstall things.

This brings up an interesting point. What is the behaviour when upgrading from 1.10.x with backport provider to 2.0.0 with normal provider. Yes pip will complain about version conflicts, but that still isn't an error (especially given many people will be on an older pip still)

[potiuk]

From this respect good @ashb ! the backport provider package only contains airflow.providers files. We do not add airflow/kubernetes.

Just double-checked and installed both airflow 1.10.12 and backport cncf,kubernetes from pcakges and seem to be all ok.

This is what you get in 'backwards_compat_converters.py in the backport package:

from typing import List

from kubernetes.client import models as k8s

from airflow.exceptions import AirflowException
from airflow.kubernetes.pod import Port, Resources
from airflow.kubernetes.pod_runtime_info_env import PodRuntimeInfoEnv
from airflow.kubernetes.volume import Volume
from airflow.kubernetes.volume_mount import VolumeMount


def _convert_kube_model_object(obj, old_class, new_class):
    convert_op = getattr(obj, "to_k8s_client_obj", None)
    if callable(convert_op):
        return obj.to_k8s_client_obj()

[ashb]

What is this rename() achieving then? What dist does/should it appear in?

[ashb]

Force-pushed back to the previous commit.

[potiuk]

But we should not I believe. This import should be changed in the backport provider as well.

[potiuk]

So the proper fix will be to remove the classes from backcompat and add rename for that one as well.

[potiuk]

the kubernetes_pod.py was excluded from the previous rename, but pod_runtime_info can be changed in all classes.

[potiuk]

@dimberman ^^ have you seen this discussion ?

[kaxil]

same here

[dimberman]

Is there a setting on black we can create for this or some form of a static check that automatically fixes this?

[ashb]

No, black doesn't look at doc comments.

pydocstyle might.