[AIRFLOW-6522] Clear log file to fix duplication in S3TaskHandler #7120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:logging
provider:amazon-aws
The same task instance (including try number) can be run on a worker when using a sensor in "reschedule" mode. Accordingly, this clears the local log file when re-initializing the logger so that the old log lines aren't uploaded again when the logger is closed.
rconroy293
force-pushed
the
AIRFLOW-6522-s3-task-handler-log-duplication
branch
from
a040930
to
1fc4738
Compare
ashb
approved these changes
Jan 11, 2020
The other option I considered was downloading the existing log from S3 into the file first (i.e. on initialization of the logger). If that succeeds, it can just overwrite with the local file. If it fails, it can use the existing append technique. Not sure your thoughts on that @ashb |
|
Awesome work, congrats on your first merged pull request! |
kaxil
pushed a commit
that referenced
this pull request
Jan 23, 2020
… in S3TaskHandler (#7120) The same task instance (including try number) can be run on a worker when using a sensor in "reschedule" mode. Accordingly, this clears the local log file when re-initializing the logger so that the old log lines aren't uploaded again when the logger is closed. (cherry picked from commit 88608ca)
potiuk
pushed a commit
that referenced
this pull request
Jan 26, 2020
… in S3TaskHandler (#7120) The same task instance (including try number) can be run on a worker when using a sensor in "reschedule" mode. Accordingly, this clears the local log file when re-initializing the logger so that the old log lines aren't uploaded again when the logger is closed. (cherry picked from commit 88608ca)
kaxil
pushed a commit
that referenced
this pull request
Jan 26, 2020
… in S3TaskHandler (#7120) The same task instance (including try number) can be run on a worker when using a sensor in "reschedule" mode. Accordingly, this clears the local log file when re-initializing the logger so that the old log lines aren't uploaded again when the logger is closed. (cherry picked from commit 88608ca)
kaxil
pushed a commit
that referenced
this pull request
Feb 3, 2020
… in S3TaskHandler (#7120) The same task instance (including try number) can be run on a worker when using a sensor in "reschedule" mode. Accordingly, this clears the local log file when re-initializing the logger so that the old log lines aren't uploaded again when the logger is closed. (cherry picked from commit 88608ca)
galuszkak
pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
… in S3TaskHandler (apache#7120) The same task instance (including try number) can be run on a worker when using a sensor in "reschedule" mode. Accordingly, this clears the local log file when re-initializing the logger so that the old log lines aren't uploaded again when the logger is closed.
tylr-mk
added a commit
to guardian/amigo
that referenced
this pull request
Apr 21, 2020
While here add a `files/` dir to the module. This allows the subdirectories to be called in the ansible patch module itself because it will search relative to this path. Actual patch: Taken from: apache/airflow#7120 Related to: https://issues.apache.org/jira/browse/AIRFLOW-6522
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The same task instance (including try number) can be run on a worker
when using a sensor in "reschedule" mode. This PR clears the
local log file when re-initializing the logger so that the old log
lines aren't uploaded again when the logger is closed on subsequent runs.
Issue link: AIRFLOW-6522
[AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID** For document-only changes commit message can start with
[AIRFLOW-XXXX].In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.