Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AwsTaskLogFetcher missing logs #41515

Merged
merged 1 commit into from
Aug 17, 2024
Merged

Conversation

vincbeck
Copy link
Contributor

Resolves #40875.

As mentioned in #40875, in some cases EcsRunTaskOperator have missing logs in the log group specified through awslogs_group. When awslogs_group is provided, the operator pull logs from awslogs_group (where the ECS task logs are) and log them so that the user can see them in the UI. The issue is, somehow (and I really cannot explain why), when multiple logs from awslogs_group have the same timestamp, only one of them (the first one) is logged. An example to better understand:

log_events = self._get_log_events(continuation_token)
for log_event in log_events:
    self.logger.info(self.event_to_str(log_event))

If log_events has 2 events with the same timestamp, self.logger.info(self.event_to_str(log_event)) will run twice but will log once.

Adding a delay between the logging operators "fix" the issue.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon-aws AWS/Amazon - related issues labels Aug 15, 2024
@vincbeck vincbeck marked this pull request as draft August 15, 2024 21:37
@vincbeck vincbeck marked this pull request as ready for review August 15, 2024 21:59
@eladkal eladkal merged commit e92cf81 into apache:main Aug 17, 2024
55 checks passed
Artuz37 pushed a commit to Artuz37/airflow that referenced this pull request Aug 19, 2024
molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Aug 20, 2024
@vincbeck vincbeck deleted the vincbeck/ecs branch September 20, 2024 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EcsRunTaskOperator log output in UI does not contain all events
2 participants