[AIRFLOW-2516] Fix mysql deadlocks #6988
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #6988 +/- ##
=========================================
Coverage ? 85.03%
=========================================
Files ? 707
Lines ? 39361
Branches ? 0
=========================================
Hits ? 33472
Misses ? 5889
Partials ? 0
Continue to review full report at Codecov.
|
kaxil
reviewed
Jan 1, 2020
|
Not yet @kaxil -> that's why it's still Draft . But I provided the users with patched versions of jobs.py/scheduled_job.py for 1.9, 1.10.6, 1.10.3 and asked them to test it. See https://issues.apache.org/jira/browse/AIRFLOW-2516?focusedCommentId=17006364&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17006364 and https://issues.apache.org/jira/browse/AIRFLOW-4498?focusedCommentId=17006370&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17006370 They tested it before for 1.9 and 1.10.6 so I expect they will come back after NewYear's holidays. |
|
And I keep my fingers crossed that it's going to help 🤞 |
|
🤞 |
Deadlocks were occuring in mysql when task_instance was modified by two queries at the same time. One query used state as selection criteria and updated it in the same query where second query just updated the state for the same table. The first query locked state index first and primary index afterwards, the second query locked primary index first and state afterwards - leading to deadlocks. This change splits the first query into two independent ones. First query makes select FOR UPDATE and selects all the task instances to act on (this will lock primary index only) and second updates all affected task instances. Note that performance impact for that is neglectable because this query is only run once every scheduler loop and the second part of it (looping through task instances) will only happen in case there are some manually modified DagRun states - so it is only run to correct some wrong states of DagRun. This should happen very infrequently.
potiuk
force-pushed
the
AIRFLOW-2516-fix-mysql-deadlocks
branch
from
8084255
to
706b8cc
Compare
boring-cyborg
bot
added
the
area:Scheduler
|
@kaxil @ashb @mik-laj @nuclearpinguin -> It's 10 days without the deadlock for our customer https://issues.apache.org/jira/browse/AIRFLOW-2516 so it looks like the problem is solved. Please approve and i will merge it and cherry-pick to 1.10.8 |
kaxil
approved these changes
Jan 13, 2020
mik-laj
approved these changes
Jan 13, 2020
potiuk
added a commit
that referenced
this pull request
Jan 21, 2020
Deadlocks were occuring in mysql when task_instance was modified by two queries at the same time. One query used state as selection criteria and updated it in the same query where second query just updated the state for the same table. The first query locked state index first and primary index afterwards, the second query locked primary index first and state afterwards - leading to deadlocks. This change splits the first query into two independent ones. First query makes select FOR UPDATE and selects all the task instances to act on (this will lock primary index only) and second updates all affected task instances. Note that performance impact for that is neglectable because this query is only run once every scheduler loop and the second part of it (looping through task instances) will only happen in case there are some manually modified DagRun states - so it is only run to correct some wrong states of DagRun. This should happen very infrequently. (cherry picked from commit 1a52182)
kaxil
pushed a commit
that referenced
this pull request
Jan 22, 2020
Deadlocks were occuring in mysql when task_instance was modified by two queries at the same time. One query used state as selection criteria and updated it in the same query where second query just updated the state for the same table. The first query locked state index first and primary index afterwards, the second query locked primary index first and state afterwards - leading to deadlocks. This change splits the first query into two independent ones. First query makes select FOR UPDATE and selects all the task instances to act on (this will lock primary index only) and second updates all affected task instances. Note that performance impact for that is neglectable because this query is only run once every scheduler loop and the second part of it (looping through task instances) will only happen in case there are some manually modified DagRun states - so it is only run to correct some wrong states of DagRun. This should happen very infrequently. (cherry picked from commit 1a52182)
kaxil
pushed a commit
that referenced
this pull request
Jan 23, 2020
Deadlocks were occuring in mysql when task_instance was modified by two queries at the same time. One query used state as selection criteria and updated it in the same query where second query just updated the state for the same table. The first query locked state index first and primary index afterwards, the second query locked primary index first and state afterwards - leading to deadlocks. This change splits the first query into two independent ones. First query makes select FOR UPDATE and selects all the task instances to act on (this will lock primary index only) and second updates all affected task instances. Note that performance impact for that is neglectable because this query is only run once every scheduler loop and the second part of it (looping through task instances) will only happen in case there are some manually modified DagRun states - so it is only run to correct some wrong states of DagRun. This should happen very infrequently. (cherry picked from commit 1a52182)
potiuk
added a commit
that referenced
this pull request
Jan 26, 2020
Deadlocks were occuring in mysql when task_instance was modified by two queries at the same time. One query used state as selection criteria and updated it in the same query where second query just updated the state for the same table. The first query locked state index first and primary index afterwards, the second query locked primary index first and state afterwards - leading to deadlocks. This change splits the first query into two independent ones. First query makes select FOR UPDATE and selects all the task instances to act on (this will lock primary index only) and second updates all affected task instances. Note that performance impact for that is neglectable because this query is only run once every scheduler loop and the second part of it (looping through task instances) will only happen in case there are some manually modified DagRun states - so it is only run to correct some wrong states of DagRun. This should happen very infrequently. (cherry picked from commit 1a52182)
kaxil
pushed a commit
that referenced
this pull request
Jan 26, 2020
Deadlocks were occuring in mysql when task_instance was modified by two queries at the same time. One query used state as selection criteria and updated it in the same query where second query just updated the state for the same table. The first query locked state index first and primary index afterwards, the second query locked primary index first and state afterwards - leading to deadlocks. This change splits the first query into two independent ones. First query makes select FOR UPDATE and selects all the task instances to act on (this will lock primary index only) and second updates all affected task instances. Note that performance impact for that is neglectable because this query is only run once every scheduler loop and the second part of it (looping through task instances) will only happen in case there are some manually modified DagRun states - so it is only run to correct some wrong states of DagRun. This should happen very infrequently. (cherry picked from commit 1a52182)
galuszkak
pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
Deadlocks were occuring in mysql when task_instance was modified by two queries at the same time. One query used state as selection criteria and updated it in the same query where second query just updated the state for the same table. The first query locked state index first and primary index afterwards, the second query locked primary index first and state afterwards - leading to deadlocks. This change splits the first query into two independent ones. First query makes select FOR UPDATE and selects all the task instances to act on (this will lock primary index only) and second updates all affected task instances. Note that performance impact for that is neglectable because this query is only run once every scheduler loop and the second part of it (looping through task instances) will only happen in case there are some manually modified DagRun states - so it is only run to correct some wrong states of DagRun. This should happen very infrequently.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Deadlocks were occuring in mysql when task_instance was modified
by two queries at the same time. One query used state as selection
criteria and updated it in the same query where second query just
updated the state for the same table. The first query locked state
index first and primary index afterwards, the second query locked
primary index first and state afterwards - leading to deadlocks.
This change splits the first query into two independent ones.
First query makes select FOR UPDATE and selects all the task
instances to act on (this will lock primary index only)
and second updates all affected task instances.
Note that performance impact for that is neglectable because this
query is only run once every scheduler loop and the second part
of it (looping through task instances) will only happen in case
there are some manually modified DagRun states - so it is only
run to correct some wrong states of DagRun. This should happen
very infrequently.
Issue link: AIRFLOW-2516
[AIRFLOW-NNNN], where AIRFLOW-NNNN = JIRA ID*(*) For document-only changes, no JIRA issue is needed, commit message is
[AIRFLOW-XXXX].In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.