Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: don't clean up old offloaded records during save. Fixes: #13220 #13286

Merged
merged 1 commit into from
Aug 31, 2024

Conversation

shuangkun
Copy link
Member

don't clean up old records when save

Fixes #13220

Motivation

Modifications

Verification

@agilgur5 agilgur5 changed the title fix: don't clean up old records when save. Fixes: #13220 fix: don't clean up old offloaded records when save. Fixes: #13220 Jul 1, 2024
@agilgur5 agilgur5 changed the title fix: don't clean up old offloaded records when save. Fixes: #13220 fix: don't clean up old offloaded records during save. Fixes: #13220 Jul 1, 2024
Copy link
Contributor

@imliuda imliuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have encoutered the same issue, and the delete operation will cause many dead locks when there are many slow queries. I agree this this change.

@agilgur5 agilgur5 added area/gc Garbage collection, such as TTLs, retentionPolicy, delays, and more area/controller Controller issues, panics labels Jul 9, 2024
if err != nil {
return "", err
}
logCtx.WithField("rowsAffected", rowsAffected).Debug("Deleted offloaded nodes")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense what you're saying. So, I notice that Save() can be called in either of these cases. Just want to confirm that the normal deletion method will take care of deleting the old Archived Workflows in either of those cases?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don‘t take care of.
We already have a periodic cleanup mechanism, so I think we can remove the cleanup during the save phase, because it does sometimes delete things that should not be deleted.

Copy link
Contributor

@juliev0 juliev0 Aug 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand. But I'm just wondering if you tested both of these cases: if packer.IsTooLargeError(err) as well as alwaysOffloadNodeStatus. Those are the ones being called where Save() is being called here, so I'm just paranoid and want to make sure we'd still be calling Delete() in the periodic job.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, both cases have passed the test.

@juliev0 juliev0 merged commit b026a0f into argoproj:main Aug 31, 2024
30 checks passed
@agilgur5 agilgur5 added the area/offloading Node status offloading label Aug 31, 2024
Joibel pushed a commit to pipekit/argo-workflows that referenced this pull request Sep 19, 2024
Joibel pushed a commit that referenced this pull request Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/controller Controller issues, panics area/gc Garbage collection, such as TTLs, retentionPolicy, delays, and more area/offloading Node status offloading
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Stop a workflow during offload causes upper: no more rows in this result set
4 participants