-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Celery green threads incompatibility #28380
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! |
As I explained in Slack, it's lilely fixed in #28283 . Provisionally, clossing, until you test and confirm it is not fixed by it. |
@potiuk, I still have the issue. Nevertheless, monkey patch warning have disappeared.
I am quite sure now that the celery broker or backend are responsible. The result backend database are not updated when some tasks finished, contrary to metadata database... What do you think? |
This is a good news. At least it allows us to merge this change because clearly the warnings are removed.
There are many things that could have caused the behaviour - but without any evidences, I can only guess, and I have no prior similar experience to base it on. As discussed on Slack the "Celery" Executor is not supported in Breeze as a "working" feature (Issue opened #28412 . It might well be some issue in the way how things are configured in Breeze. If you would like to explore it further and try to investigate how to implement it, that woudl be awesome. But this is not a high priority, really and I would prefer someone (like you) who is a celery user can do some more investigation how to make it works in Breeze. This happend in the past and I would like more of the contirbutors to spend their time in improving our dev env - seems that someone who understand what gevent is and wants to use it has enough incentive to do more investigation. Happy to help with it but need more evidences and some deeper debugging from your side if we are to progress there. I am happy to get ideas bounced off me and to help to understand ins-outs of Breeze to guide such a person. Slack discussion in #breeze channel on slack is likely best place for that. |
After testing many configurations about celery backend, the solution I found is the combination of :
Thus, the whole issue about celery green threads is related to the way of airflow calls the result backend. There is something blocking the celery workers to stop. I would go deeper in the code, but I am quite surprised that I don't see many users suffering from this issue, since it is a quite common pattern to concurrently call IO tasks with green threads. To achieve that, do you guys use |
Apache Airflow version
2.5.0
What happened
Celery offers the capability to concurrently run green threads like
gevent
oreventlet
for IO bound tasks.I didn't manage to setup a simple Airflow project which follow this basic idea, as I pointed out in this discussion.
It could be followed in those logs files :
The log of a task : dag_id=simple_mapping_run_id=scheduled__2022-12-14T12 10 17.314020+00 00_task_id=add_one_map_index=3_attempt=1.log
The log of the worker : airflow-worker.log
What you think should happen instead
Instead, the result backend database should be updated as soon as a task completes, in order to quickly let other tasks to run.
First, I have suspected the
Postgres
result backend to be the culprit since it is not clear that Psychopg manage concurrent writings withgevent
.But after seeing worker logs warning identical than #8164 about the gevent monkey-patching , I have a doubt.
How to reproduce
Strictly follow docker compose airflow 2.5.0 instructions.
Add to
airflow-common-env
the env varAIRFLOW__CELERY__POOL: 'gevent'
.Launch from Airflow UI this simple DAG:
Operating System
Ubuntu 22.04.1 LTS
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
Docker Engine - Community - Version: 20.10.18
Docker Compose version v2.10.2
Anything else
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: