Handle green threads for IO tasks

[victorjourne]

Celery offers the capability to concurrently run green threads like gevent or eventlet for IO bound tasks.

I didn't manage to setup an Airflow project which follow this idea.
From docker compose airflow 2.5.0 and adding those env var to airflow conf AIRFLOW__CELERY__POOL : "gevent", a simple dynamic mapping dag takes a while to execute and tasks status in the result backend (here postgres) are not updated.

I would appreciate a lot to know how do you tackle such issue, about Airflow compatibility with green threads ?

simple dag :

from datetime import datetime

from airflow import DAG
from airflow.decorators import task

with DAG(dag_id="simple_mapping",
        catchup=False,
        start_date=datetime(2022, 3, 4),
        max_active_tasks=200) as dag:

    @task
    def add_one(x: int):
        return x + 1

    @task
    def sum_it(values):
        total = sum(values)
        print(f"Total was {total}")

    xlist = list(range(25))
    added_values = add_one.expand(x=xlist)

    sum_it(added_values)

[potiuk]

A number of people reported them working so there is nothing "generic" that we can help with. Maybe except giving you some leads where you should furrther investigate and debug it.

There are some problems reported with gevent monkeypatching that might not work in some cases (the nature is it rather subtle and convoluted): #8212 (you will find workarounds there by patching local settings) and there is a possible fix coming in #28283.

But this would be just a guesswork - you need to look at your logs and find out what exactly does not work for your and investigate it - if you report enough details and evidences and logs showing the problem maybe someone here will be able to help you.

BTW. Do not expect that docker-compose will just work out of the box in all cases. Docker Compose has a number of limitations and you might have hit some of them. This Docker compose is just there to "play with". If you want something more serious, there are no "ready to use recipes" to any problems you might have - Docker Compose is known to require high level of expertise and understanding what's going on under the hood if you want to make it production -ready and optimised.

As mentioned in the note:

This procedure can be useful for learning and exploration. However, adapting it for use in real-world situations can be complicated. Making changes to this procedure will require specialized expertise in Docker & Docker Compose, and the Airflow community may not be able to help you.

For that reason, we recommend using Kubernetes with the Official Airflow Community Helm Chart when you are ready to run Airflow in production.

Again - If you report the problems you had and result of your investigations (how did you debug it, what did you found and what exactly does not work, we might be able here to help much more efficiently to solve your problems but without it, it's just a guesswork - you need to really understand what's not working if you reach out for help in case of such problems.

[victorjourne]

Thanks a lot for the quick answer.
I turned this discussion into the issue #28380 ; I am kind to investigate and try hot patches to make airflow greater for green threads!
About Docker compose, I prefer this option for debugging. Do you use a Kubernetes cluster to develop the Airflow core ?

[potiuk]

I like your curiosity.

We use Breeze and local virtualenv. Breeze uses a completely different (and fully managed by the Breeze Python wrapping code) set of docker-compose files stored here: https://github.com/apache/airflow/tree/main/scripts/ci/docker-compose but all details of that are hidden from the contributor. Also we control every aspect of it and we developed a python wrapper that limits the ways you can use it as a core developer. And it only uses Local Executor by default. Celery executor tests are run with integration tests: https://github.com/apache/airflow/blob/main/TESTING.rst#airflow-integration-tests with celery integration enabled. It's very tightly controlled and we do not expect anyone to modify those docker-compose files unless new development features are added (with PR, reviews, automated tests and many gates).

One of the reasons is precisely the fact that the moment you modify the docker-compose files (which is every time you want to make any change) - you take over the responsibility for what it brings. And it's next to impossible to prepare a "generic" docker compose that you - as a user - might have an expectation that it will work for any combination of configuration parameters. What works for you might not work for 1000s of other users easily - that's why we don't even claim our docker-compose for quickstart will handle many options. We even explicitly say it won't.

You can read more about Breeze here: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst - but it has nothing to do with the "quick-start" docker compose. Quick Start docker compose is just this - quick start to get things up and running.