AWS ECS Logging very slow when lots of logging leading to task failure

[smsm1-ito]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.28.0 is affected

apache-airflow-providers-amazon==8.27.0 is not affected

Apache Airflow version

2.10.2

Operating System

Debian 12 bookworm

Deployment

Other Docker-based deployment

Deployment details

No response

What happened

After upgrading to Airflow 2.10.2 longer running ECS tasks with significant logging started failing. The logs would still be slowly appearing on Airflow, yet the ECS Task had completed. If the logging took more than an hour more than the task, then the ECS task in Airflow would fail with an error that the ECS Task was missing. This is due to the older tasks disappearing within ECS (Fargate).

Looking at the changes I came across https://github.com/apache/airflow/pull/41515/files which added a 0.1 second sleep if the timestamps were the same. On looking further at the logs of the tasks that were failing, there were 2 log times. One which was getting significantly later than the other from the application.

On rolling back the amazon provider to the previous version and still using Airflow 2.10.2 the issue went away.

Linked tickets #41515 #40875

What you think should happen instead

Logging should be submitted in a timely manner.

Could we go for a much shorter delay such as 0.001 seconds?

How to reproduce

Have an ECS Task that has a lot more logging than the time it takes to run the task.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[smsm1-ito]

I've created a merge request for this: #42449