Replies: 3 comments 4 replies
-
I seriously doubt this is the case - this is an extremely wild and unfounded guess. It's likely an error in your DAG, but I think we need more informationand reproducible case showing exactly what causes it to be sure. I suggest you do more experimentation with it and trying to provide more information, logs, circumstances, also experimenting with decreasing the number of schedulers to 1 and running it for a while - if you can make a fully reproducible case that can help with diagnosing and result of your analysis, it would be much better. Converting it into discussion. I am also not going to respond as I am going for 2 weeks vacation, but I hope others will. |
Beta Was this translation helpful? Give feedback.
-
We noticed a similar issue, however we are not running multi-scheduler. We are only running one at the same time (however because kubernetes deploys in some cases we might have several schedulers. The issue just randomly happened, at some point the scheduler started crashing. There was a dynamic task which had a taskinstance with map_index I'll be trying to reproduce this next week, but it might be our scheduler got killed at an unfurtunate time. Some help on where to look in the code, or some ideas on how to reproduce would be appreciated :) |
Beta Was this translation helpful? Give feedback.
-
@potiuk we had this happen 2 extra times over the weekend. I am not sure how this can happen looking into the code, everything should be run nicely into a transaction but what we see is the following:
The scheduler keeps on crashing trying to change the I can think of the following fix. When there is a As to how this might happen I have no clue yet. |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
Other Airflow 2 version
What happened
We noticed that tasks using the dynamic task mapping api crashes the scheduler with the error
We have 2 schedulers so my guess is that both try to schedule the dynamically mapped task
What you think should happen instead
it should be not crash the scheduler
How to reproduce
run a large dynamically mapped task in multi-scheduler environment
Operating System
linux
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions