Fix race condition in agent Traceflow controller #5954
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tnqn
added
area/ops/traceflow
tnqn
force-pushed
the
fix-traceflow-race
branch
from
c9d8f13
to
97be1dd
Compare
hangyan
previously approved these changes
Feb 1, 2024
antoninbas
previously approved these changes
Feb 1, 2024
| c.runningTraceflowsMutex.Unlock() | ||
| // This may happen if a Traceflow is assigned with a tag that was just released from an old Traceflow but | ||
| // the agent hasn't processed the deletion event of the old Traceflow yet. | ||
| if ok && tfState.name != traceflowName { |
There was a problem hiding this comment.
can we see a different TF with the same name before we process the deletion?
There was a problem hiding this comment.
It's possible in theory. If we create a TF, delete it, then recreate a new one with the same name. It's possible that the same tag is assigned to it. If agent processes the creation of the 2nd TF first, it will encounter this.
But this shouldn't happen with antctl which appends random string to the name.
To totally prevent it, we should use UID to identify TF.
There was a problem hiding this comment.
I might misunderstand your question. Now I feel you are asking in which case we will see the current bug happen?
It's due to the concurrent workers: we will receive the deletion of old TF before the creation of new TF. But after they are dispatched to workers, one worker may run faster than another. Then we end of with processing creation first.
There was a problem hiding this comment.
No, you understood the question correctly.
Yes I was thinking we could store the UID in the Traceflow "state", and use something like tfState.UID != tf.UID here. We could probably keep indexing the Traceflow state map on the name. Would more changes be required?
gran-vmv
previously approved these changes
Feb 2, 2024
It may happen that a Traceflow is assigned with a tag that was just released from an old Traceflow but the controller hasn't processed the deletion event of the old Traceflow yet. Previously the controller skipped starting new Traceflow if the tag was already being used, which caused the Traceflow to timeout. The commit adds a check when determining whether it should start a Traceflow. If the tag is associated with another Traceflow, it will clean it up then start a new trace for the current one. It also fixes a bug in cleanupTraceflow, which might uninstall flows for another Traceflow if the tag is reassigned. Signed-off-by: Quan Tian <[email protected]>
tnqn
dismissed stale reviews from luolanzone, gran-vmv, antoninbas, and hangyan
via
1ef41eb
tnqn
force-pushed
the
fix-traceflow-race
branch
from
97be1dd
to
1ef41eb
Compare
|
/skip-all |
1 similar comment
|
/skip-all |
tnqn
added a commit
to tnqn/antrea
that referenced
this pull request
Mar 6, 2024
It may happen that a Traceflow is assigned with a tag that was just released from an old Traceflow but the controller hasn't processed the deletion event of the old Traceflow yet. Previously the controller skipped starting new Traceflow if the tag was already being used, which caused the Traceflow to timeout. The commit adds a check when determining whether it should start a Traceflow. If the tag is associated with another Traceflow, it will clean it up then start a new trace for the current one. It also fixes a bug in cleanupTraceflow, which might uninstall flows for another Traceflow if the tag is reassigned. Signed-off-by: Quan Tian <[email protected]>
tnqn
added a commit
to tnqn/antrea
that referenced
this pull request
Mar 6, 2024
It may happen that a Traceflow is assigned with a tag that was just released from an old Traceflow but the controller hasn't processed the deletion event of the old Traceflow yet. Previously the controller skipped starting new Traceflow if the tag was already being used, which caused the Traceflow to timeout. The commit adds a check when determining whether it should start a Traceflow. If the tag is associated with another Traceflow, it will clean it up then start a new trace for the current one. It also fixes a bug in cleanupTraceflow, which might uninstall flows for another Traceflow if the tag is reassigned. Signed-off-by: Quan Tian <[email protected]>
tnqn
added a commit
to tnqn/antrea
that referenced
this pull request
Mar 25, 2024
It may happen that a Traceflow is assigned with a tag that was just released from an old Traceflow but the controller hasn't processed the deletion event of the old Traceflow yet. Previously the controller skipped starting new Traceflow if the tag was already being used, which caused the Traceflow to timeout. The commit adds a check when determining whether it should start a Traceflow. If the tag is associated with another Traceflow, it will clean it up then start a new trace for the current one. It also fixes a bug in cleanupTraceflow, which might uninstall flows for another Traceflow if the tag is reassigned. Signed-off-by: Quan Tian <[email protected]>
tnqn
added a commit
that referenced
this pull request
Mar 26, 2024
It may happen that a Traceflow is assigned with a tag that was just released from an old Traceflow but the controller hasn't processed the deletion event of the old Traceflow yet. Previously the controller skipped starting new Traceflow if the tag was already being used, which caused the Traceflow to timeout. The commit adds a check when determining whether it should start a Traceflow. If the tag is associated with another Traceflow, it will clean it up then start a new trace for the current one. It also fixes a bug in cleanupTraceflow, which might uninstall flows for another Traceflow if the tag is reassigned. Signed-off-by: Quan Tian <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It may happen that a Traceflow is assigned with a tag that was just released from an old Traceflow but the controller hasn't processed the deletion event of the old Traceflow yet. Previously the controller skipped starting new Traceflow if the tag was already being used, which caused the Traceflow to timeout.
The commit adds a check when determining whether it should start a Traceflow. If the tag is associated with another Traceflow, it will clean it up then start a new trace for the current one.
It also fixes a bug in cleanupTraceflow, which might uninstall flows for another Traceflow if the tag is reassigned.
Fixes #5760
Fixes #5609