-
Notifications
You must be signed in to change notification settings - Fork 299
Commit
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: 070f48de7b90eb55b36452e050719a6e | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,128 @@ | ||
:py:mod:`airflow.providers.cncf.kubernetes.callbacks` | ||
===================================================== | ||
|
||
.. py:module:: airflow.providers.cncf.kubernetes.callbacks | ||
Module Contents | ||
--------------- | ||
|
||
Classes | ||
~~~~~~~ | ||
|
||
.. autoapisummary:: | ||
|
||
airflow.providers.cncf.kubernetes.callbacks.ExecutionMode | ||
airflow.providers.cncf.kubernetes.callbacks.KubernetesPodOperatorCallback | ||
|
||
|
||
|
||
|
||
Attributes | ||
~~~~~~~~~~ | ||
|
||
.. autoapisummary:: | ||
|
||
airflow.providers.cncf.kubernetes.callbacks.client_type | ||
|
||
|
||
.. py:data:: client_type | ||
.. py:class:: ExecutionMode | ||
Bases: :py:obj:`str`, :py:obj:`enum.Enum` | ||
|
||
Enum class for execution mode. | ||
|
||
.. py:attribute:: SYNC | ||
:value: 'sync' | ||
|
||
|
||
|
||
.. py:attribute:: ASYNC | ||
:value: 'async' | ||
|
||
|
||
|
||
|
||
.. py:class:: KubernetesPodOperatorCallback | ||
`KubernetesPodOperator` callbacks methods. | ||
|
||
Currently, the callbacks methods are not called in the async mode, this support will be added | ||
in the future. | ||
|
||
.. py:method:: on_sync_client_creation(*, client, **kwargs) | ||
:staticmethod: | ||
|
||
Invoke this callback after creating the sync client. | ||
|
||
:param client: the created `kubernetes.client.CoreV1Api` client. | ||
|
||
|
||
.. py:method:: on_pod_creation(*, pod, client, mode, **kwargs) | ||
:staticmethod: | ||
|
||
Invoke this callback after creating the pod. | ||
|
||
:param pod: the created pod. | ||
:param client: the Kubernetes client that can be used in the callback. | ||
:param mode: the current execution mode, it's one of (`sync`, `async`). | ||
|
||
|
||
.. py:method:: on_pod_starting(*, pod, client, mode, **kwargs) | ||
:staticmethod: | ||
|
||
Invoke this callback when the pod starts. | ||
|
||
:param pod: the started pod. | ||
:param client: the Kubernetes client that can be used in the callback. | ||
:param mode: the current execution mode, it's one of (`sync`, `async`). | ||
|
||
|
||
.. py:method:: on_pod_completion(*, pod, client, mode, **kwargs) | ||
:staticmethod: | ||
|
||
Invoke this callback when the pod completes. | ||
|
||
:param pod: the completed pod. | ||
:param client: the Kubernetes client that can be used in the callback. | ||
:param mode: the current execution mode, it's one of (`sync`, `async`). | ||
|
||
|
||
.. py:method:: on_pod_cleanup(*, pod, client, mode, **kwargs) | ||
:staticmethod: | ||
|
||
Invoke this callback after cleaning/deleting the pod. | ||
|
||
:param pod: the completed pod. | ||
:param client: the Kubernetes client that can be used in the callback. | ||
:param mode: the current execution mode, it's one of (`sync`, `async`). | ||
|
||
|
||
.. py:method:: on_operator_resuming(*, pod, event, client, mode, **kwargs) | ||
:staticmethod: | ||
|
||
Invoke this callback when resuming the `KubernetesPodOperator` from deferred state. | ||
|
||
:param pod: the current state of the pod. | ||
:param event: the returned event from the Trigger. | ||
:param client: the Kubernetes client that can be used in the callback. | ||
:param mode: the current execution mode, it's one of (`sync`, `async`). | ||
|
||
|
||
.. py:method:: progress_callback(*, line, client, mode, **kwargs) | ||
:staticmethod: | ||
|
||
Invoke this callback to process pod container logs. | ||
|
||
:param line: the read line of log. | ||
:param client: the Kubernetes client that can be used in the callback. | ||
:param mode: the current execution mode, it's one of (`sync`, `async`). | ||
|
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
:py:mod:`airflow.providers.cncf.kubernetes.decorators` | ||
====================================================== | ||
|
||
.. py:module:: airflow.providers.cncf.kubernetes.decorators | ||
Submodules | ||
---------- | ||
.. toctree:: | ||
:titlesonly: | ||
:maxdepth: 1 | ||
|
||
kubernetes/index.rst | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
:py:mod:`airflow.providers.cncf.kubernetes.decorators.kubernetes` | ||
================================================================= | ||
|
||
.. py:module:: airflow.providers.cncf.kubernetes.decorators.kubernetes | ||
Module Contents | ||
--------------- | ||
|
||
|
||
Functions | ||
~~~~~~~~~ | ||
|
||
.. autoapisummary:: | ||
|
||
airflow.providers.cncf.kubernetes.decorators.kubernetes.kubernetes_task | ||
|
||
|
||
|
||
.. py:function:: kubernetes_task(python_callable = None, multiple_outputs = None, **kwargs) | ||
Kubernetes operator decorator. | ||
|
||
This wraps a function to be executed in K8s using KubernetesPodOperator. | ||
Also accepts any argument that DockerOperator will via ``kwargs``. Can be | ||
reused in a single DAG. | ||
|
||
:param python_callable: Function to decorate | ||
:param multiple_outputs: if set, function return value will be | ||
unrolled to multiple XCom values. Dict will unroll to xcom values with | ||
keys as XCom keys. Defaults to False. | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
:py:mod:`airflow.providers.cncf.kubernetes.executors` | ||
===================================================== | ||
|
||
.. py:module:: airflow.providers.cncf.kubernetes.executors | ||
Submodules | ||
---------- | ||
.. toctree:: | ||
:titlesonly: | ||
:maxdepth: 1 | ||
|
||
kubernetes_executor/index.rst | ||
kubernetes_executor_types/index.rst | ||
kubernetes_executor_utils/index.rst | ||
local_kubernetes_executor/index.rst | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,166 @@ | ||
:py:mod:`airflow.providers.cncf.kubernetes.executors.kubernetes_executor` | ||
========================================================================= | ||
|
||
.. py:module:: airflow.providers.cncf.kubernetes.executors.kubernetes_executor | ||
.. autoapi-nested-parse:: | ||
|
||
KubernetesExecutor. | ||
|
||
.. seealso:: | ||
For more information on how the KubernetesExecutor works, take a look at the guide: | ||
:doc:`/kubernetes_executor` | ||
|
||
|
||
|
||
Module Contents | ||
--------------- | ||
|
||
Classes | ||
~~~~~~~ | ||
|
||
.. autoapisummary:: | ||
|
||
airflow.providers.cncf.kubernetes.executors.kubernetes_executor.KubernetesExecutor | ||
|
||
|
||
|
||
|
||
Attributes | ||
~~~~~~~~~~ | ||
|
||
.. autoapisummary:: | ||
|
||
airflow.providers.cncf.kubernetes.executors.kubernetes_executor.base_version | ||
airflow.providers.cncf.kubernetes.executors.kubernetes_executor.ARG_NAMESPACE | ||
airflow.providers.cncf.kubernetes.executors.kubernetes_executor.ARG_MIN_PENDING_MINUTES | ||
airflow.providers.cncf.kubernetes.executors.kubernetes_executor.KUBERNETES_COMMANDS | ||
|
||
|
||
.. py:data:: base_version | ||
.. py:data:: ARG_NAMESPACE | ||
.. py:data:: ARG_MIN_PENDING_MINUTES | ||
.. py:data:: KUBERNETES_COMMANDS | ||
:value: () | ||
|
||
|
||
|
||
.. py:class:: KubernetesExecutor | ||
Bases: :py:obj:`airflow.executors.base_executor.BaseExecutor` | ||
|
||
Executor for Kubernetes. | ||
|
||
.. py:attribute:: RUNNING_POD_LOG_LINES | ||
:value: 100 | ||
|
||
|
||
|
||
.. py:attribute:: supports_ad_hoc_ti_run | ||
:type: bool | ||
:value: True | ||
|
||
|
||
|
||
.. py:method:: clear_not_launched_queued_tasks(session = NEW_SESSION) | ||
Clear tasks that were not yet launched, but were previously queued. | ||
|
||
Tasks can end up in a "Queued" state when a rescheduled/deferred operator | ||
comes back up for execution (with the same try_number) before the | ||
pod of its previous incarnation has been fully removed (we think). | ||
|
||
It's also possible when an executor abruptly shuts down (leaving a non-empty | ||
task_queue on that executor), but that scenario is handled via normal adoption. | ||
|
||
This method checks each of our queued tasks to see if the corresponding pod | ||
is around, and if not, and there's no matching entry in our own | ||
task_queue, marks it for re-execution. | ||
|
||
|
||
.. py:method:: start() | ||
Start the executor. | ||
|
||
|
||
.. py:method:: execute_async(key, command, queue = None, executor_config = None) | ||
Execute task asynchronously. | ||
|
||
|
||
.. py:method:: sync() | ||
Synchronize task state. | ||
|
||
|
||
.. py:method:: get_task_log(ti, try_number) | ||
Return the task logs. | ||
|
||
:param ti: A TaskInstance object | ||
:param try_number: current try_number to read log from | ||
:return: tuple of logs and messages | ||
|
||
|
||
.. py:method:: try_adopt_task_instances(tis) | ||
Try to adopt running task instances that have been abandoned by a SchedulerJob dying. | ||
|
||
Anything that is not adopted will be cleared by the scheduler (and then become eligible for | ||
re-scheduling) | ||
|
||
:return: any TaskInstances that were unable to be adopted | ||
|
||
|
||
.. py:method:: cleanup_stuck_queued_tasks(tis) | ||
Handle remnants of tasks that were failed because they were stuck in queued. | ||
|
||
Tasks can get stuck in queued. If such a task is detected, it will be marked | ||
as `UP_FOR_RETRY` if the task instance has remaining retries or marked as `FAILED` | ||
if it doesn't. | ||
|
||
:param tis: List of Task Instances to clean up | ||
:return: List of readable task instances for a warning message | ||
|
||
|
||
.. py:method:: adopt_launched_task(kube_client, pod, tis_to_flush_by_key) | ||
Patch existing pod so that the current KubernetesJobWatcher can monitor it via label selectors. | ||
|
||
:param kube_client: kubernetes client for speaking to kube API | ||
:param pod: V1Pod spec that we will patch with new label | ||
:param tis_to_flush_by_key: TIs that will be flushed if they aren't adopted | ||
|
||
|
||
.. py:method:: end() | ||
Shut down the executor. | ||
|
||
|
||
.. py:method:: terminate() | ||
Terminate the executor is not doing anything. | ||
|
||
|
||
.. py:method:: get_cli_commands() | ||
:staticmethod: | ||
|
||
Vends CLI commands to be included in Airflow CLI. | ||
|
||
Override this method to expose commands via Airflow CLI to manage this executor. This can | ||
be commands to setup/teardown the executor, inspect state, etc. | ||
Make sure to choose unique names for those commands, to avoid collisions. | ||
|
||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
:py:mod:`airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types` | ||
=============================================================================== | ||
|
||
.. py:module:: airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types | ||
Module Contents | ||
--------------- | ||
|
||
.. py:data:: ADOPTED | ||
:value: 'adopted' | ||
|
||
|
||
|
||
.. py:data:: KubernetesJobType | ||
.. py:data:: ALL_NAMESPACES | ||
:value: 'ALL_NAMESPACES' | ||
|
||
|
||
|
||
.. py:data:: POD_EXECUTOR_DONE_KEY | ||
:value: 'airflow_executor_done' | ||
|
||
|
||
|