PodRuntimeInfoEnv is not working in 1.10.11

[dimon222]

Apache Airflow version: 1.10.11

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.17.0

What happened:
PodRuntimeInfoEnv object is not working in Airflow 1.10.11. It might be result of some recent (past 1 year?) refactoring of kubernetes code or so.

Exception thrown is this

Invalid value for `field_path`, must not be `None`

What you expected to happen:
It's able to initialize and produce respective KubernestPodOperator with it.

How to reproduce it:
Create PodRuntimeInfoEnv object, try to plug it in KubernetesPodOperator. Kubernetes Pod Operator never gets submitted and stacktrace pops.

It happens 100% of times. I know for sure that it worked perfectly fine on Airflow 1.10.9. So might be in 1.10.10 or 1.10.11 there was introduced regression.

After some investigation it seem to bring me to this place -

airflow/airflow/kubernetes/pod_runtime_info_env.py

Lines 50 to 52 in 44d4ae8

	field_ref=k8s.V1ObjectFieldSelector(
	self.field_path
	)

As you might see, field_path is first argument. However, if I look at official kubernetes-python code, the first argument in V1ObjectFieldSelector constructor is api_version.
https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_object_field_selector.py#L45
Since its clearly not provided in here, we end up with exception related to field_path value:
https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_object_field_selector.py#L103

Possible solutions could be:

1. Pass hardcoded api_version as well, this way ensuring that in case Kubernetes API changes, its remembered which version it complies to

                field_ref=k8s.V1ObjectFieldSelector(
                    api_version='v1',
                    field_path=self.field_path
                )

2. Instead of using positional arg, use kwarg in V1ObjectFieldSelector

                field_ref=k8s.V1ObjectFieldSelector(
                    field_path=self.field_path
                )

For people who might already be noticing issue right now the simplest approach would be inheritance that would overwrite or monkeypatch the PodRuntimeInfoEnv class (specifically its method to_k8s_client_obj) and then use with it one of above mentioned solutions.

[potiuk]

@kaxil @dimberman -> I keep my 🤞 that it's already fixed in 1.10.12rc3

[kaxil]

Yup this one is fixed or atleast I think we have

[potiuk]

@dimon222 - there was a major Kubernetes refactor in 1.10.11 that introduced some errors that 1.10.12 fixes. We are going to mark 1.10.11 as "yanked" so that it will be skipped by default.

[dimon222]

It doesn't seem to be fixed, because I still see the incorrect instantiation in both master and 1.10.12rc3 branches
https://github.com/apache/airflow/blob/1.10.12rc3/airflow/kubernetes/pod_runtime_info_env.py#L49
https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_runtime_info_env.py#L50

[kaxil]

Will try it out in a bit.

[kaxil]

If it was a regression since 1.10.10 and there is a workaround available this will be fixed in 1.10.13 -- but if not we will create a new rc4 for it.

[potiuk]

OH NO!!!!!!!

[potiuk]

Just tested rc3 😢

[kaxil]

@dimon222 can you please test out the patch in #10478 :)