-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
on_failure_callback is not called when task is manually marked as failed #38935
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
I saw that in version 1.9.0, there is already a different type of exception for this case.
|
This behaviour is documented in the code airflow/airflow/models/taskinstance.py Lines 2594 to 2597 in 5f6f4a5
So it's not a bug |
Yes, I saw that. So there is no intention to change this behavior? |
There are no plans to change the behaviour at the moment. Also, I don't know why you would want the callback to run when you intentionally failed a task |
In my case, I use Airflow as an orchestrator to run Spark applications (spark-sql and spark-submit). When a task has failed or is marked as failed, in addition to notifying that the task has failed, I need to make sure that the corresponding step has been canceled in Spark. Most of the time, when a task actually fails, the cancel step operation is unnecessary, but this is not the case when a task is marked as failed. I don't think it's such a rare scenario, plus similar issues have been opened before. I believe this behavior change would be very welcome in future Airflow releases. |
Anyway, I managed to solve my problem overwriting the
PS: The |
I think it might be worth it if you can create a feature request and explain how necessary this issue is, the use cases because I see it, that it's not required because the user is performing a manual action on the UI, what's the need to run the callback when same callback can be a task instead. closing since it's not a bug |
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.7.1
What happened?
on_failure_callback is not called when task is terminated externally.
A similar issue was reported in #25297 and fixed in #29743.
I saw that the code change to call the on_failure_callback in this cases is no longer present on version 2.7.1.
After which on_failure_callback is no longer called when SIGTERM is received
As you can see from the screenshots bellow the function was never called, the logs are never printed.
What you think should happen instead?
on_failure_callback should be called when task fails including when the task was manually marked as failed.
How to reproduce
Create a dag task with an on_failure_callback configured, start and then mark the task as failed. You can see from the log that the function was never called.
Operating System
debian
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: