Dynamic task mapping does not correctly handle depends_on_past #28296
Labels
affected_version:2.4
Issues Reported for 2.4
area:core
area:dynamic-task-mapping
AIP-42
kind:bug
This is a clearly a bug
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
Using Airflow 2.4.2.
I've got a task that retrieves some filenames, which then creates dynamically mapped tasks to move the files, one per task.
I'm using a similar task across multiple DAGs. However, task mapping fails on some DAG runs: it inconsistently happens per DAG run, and some DAGs do not seem to be affected at all. These seem to be the DAGs where no task was ever mapped, so that the mapped task instance ended up in a Skipped state.
What happens is that multiple files will be found, but only a single dynamically mapped task will be created. This task never starts and has map_index of -1. It can be found under the "List instances, all runs" menu, but says "No Data found." under the "Mapped Tasks" tab.
When I press the "Run" button when the mapped task is selected, the following error appears:
The previous task has run however. No errors appeared in my Airflow logs.
What you think should happen instead
The appropriate amount of task instances should be created, they should correctly resolve the
depends_on_past
check and then proceed to run correctly.How to reproduce
This DAG reliably reproduces the error for me. The first set of mapped tasks succeeds, the subsequent ones do not.
Operating System
Amazon Linux 2
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: