-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-5873] KubernetesPodOperator fixes and test #6524
Conversation
1332ace
to
2068a35
Compare
Now just waiting for the master one to pass :) |
- `security_context` was missing from docs of `KubernetesPodOperator` - `KubernetesPodOperator` kwarg `in_cluster` erroneously defaults to False in comparison to `default_args.py`, also default `do_xcom_push` was overwritten to False in contradiction to `BaseOperator` - `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) - `kubernetes/pod.py`: classes do not have `__slots__` so they would accept arbitrary values in `setattr` - Reduce amount of times the pod object is copied before execution
Codecov Report
@@ Coverage Diff @@
## master #6524 +/- ##
=========================================
Coverage ? 83.62%
=========================================
Files ? 645
Lines ? 37291
Branches ? 0
=========================================
Hits ? 31186
Misses ? 6105
Partials ? 0
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really nice set of fixes for KubernetesPodOperator
@@ -40,6 +40,8 @@ class Resources(K8SModel): | |||
:param limit_gpu: Limits for GPU used | |||
:type limit_gpu: int | |||
""" | |||
__slots__ = ('request_memory', 'request_cpu', 'limit_memory', 'limit_cpu', 'limit_gpu') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice!
- `security_context` was missing from docs of `KubernetesPodOperator` - `KubernetesPodOperator` kwarg `in_cluster` erroneously defaults to False in comparison to `default_args.py`, also default `do_xcom_push` was overwritten to False in contradiction to `BaseOperator` - `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) - `kubernetes/pod.py`: classes do not have `__slots__` so they would accept arbitrary values in `setattr` - Reduce amount of times the pod object is copied before execution
- `security_context` was missing from docs of `KubernetesPodOperator` - `KubernetesPodOperator` kwarg `in_cluster` erroneously defaults to False in comparison to `default_args.py`, also default `do_xcom_push` was overwritten to False in contradiction to `BaseOperator` - `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) - `kubernetes/pod.py`: classes do not have `__slots__` so they would accept arbitrary values in `setattr` - Reduce amount of times the pod object is copied before execution (cherry picked from commit cf38ddc)
- `security_context` was missing from docs of `KubernetesPodOperator` - `KubernetesPodOperator` kwarg `in_cluster` erroneously defaults to False in comparison to `default_args.py`, also default `do_xcom_push` was overwritten to False in contradiction to `BaseOperator` - `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) - `kubernetes/pod.py`: classes do not have `__slots__` so they would accept arbitrary values in `setattr` - Reduce amount of times the pod object is copied before execution (cherry picked from commit cf38ddc)
- `security_context` was missing from docs of `KubernetesPodOperator` - `KubernetesPodOperator` kwarg `in_cluster` erroneously defaults to False in comparison to `default_args.py`, also default `do_xcom_push` was overwritten to False in contradiction to `BaseOperator` - `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) - `kubernetes/pod.py`: classes do not have `__slots__` so they would accept arbitrary values in `setattr` - Reduce amount of times the pod object is copied before execution (cherry picked from commit cf38ddc)
- `security_context` was missing from docs of `KubernetesPodOperator` - `KubernetesPodOperator` kwarg `in_cluster` erroneously defaults to False in comparison to `default_args.py`, also default `do_xcom_push` was overwritten to False in contradiction to `BaseOperator` - `KubernetesPodOperator` kwarg `resources` is erroneously passed to `base_operator`, instead should only go to `PodGenerator`. The two have different syntax. (both on `master` and `v1-10-test` branches) - `kubernetes/pod.py`: classes do not have `__slots__` so they would accept arbitrary values in `setattr` - Reduce amount of times the pod object is copied before execution (cherry picked from commit cf38ddc)
Make sure you have checked all steps below.
Jira
Description
This PR implements the same as #6523 but is adapted to
master
branch:KubernetesPodOperator
kwargresources
is erroneously passed tobase_operator
, instead should only go toPodGenerator
. The two have different syntax. (both onmaster
andv1-10-test
branches)resources
passed to PodGenerator should bek8s.V1ResourceRequirements
, which is now handled inKubernetesPodOperator
kubernetes/pod.py
:Resources
does not have__slots__
so accepts arbitrary values insetattr
(not present on either branch https://github.com/apache/airflow/blame/50343040ff4679e32e01f138ead80bc4bcef4b47/airflow/contrib/operators/kubernetes_pod_operator.py#L166-L171)KubernetesPodOperator
kwargin_cluster
erroneously defaults to False in comparison todefault_args.py
, also defaultdo_xcom_push
was overwritten to False in contradiction toBaseOperator
v1-10-test
is behindmaster
with KubernetesPodOperator fixes and refactors (will not be addressed in this PR,)move kubernetes folder one level up fromhttps://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L21/contrib
fixhttps://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L90xcom_push
todo_xcom_push
Tests
contrib/operators/test_kubernetes_pod_operator.py