Stop a workflow during offload causes upper: no more rows in this result set

[shuangkun]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

When the workflow updates the nodestatus to the database and deletes the old records, if someone stops the workflow during the update, a conflict will occur. However, reapply cannot parse woc.orig at this time, resulting in upper: no more rows in this result set

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Any large workflow

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[shuangkun]

Maybe we don't need to delete oldRecord when offload sace, because there is periodic cleanup.

[jswxstw]

Maybe we don't need to delete oldRecord when offload sace, because there is periodic cleanup.

uid is the primary key, use insert on update instead?

[shuangkun]

This is similar to canceling the cleanup of old records in the offload save phase. Dehydrate and update are very close. If you insert after update, you need to save the offloadVersion generated in the Dehydrate phase.

[jswxstw]

Sorry, I misunderstood this issue, I thought it was a issue caused by workflow archiving before.

[imliuda]

@shuangkun I have a doubt with this, when deleting old records, there is a updatedat filter, that means your reapplyUpdate() run at least OFFLOAD_NODE_STATUS_TTL time, right? And have you noticed there are any slow queries?

	rs, err := wdc.session.SQL().
		DeleteFrom(wdc.tableName).
		Where(db.Cond{"clustername": wdc.clusterName}).
		And(db.Cond{"uid": uid}).
		And(db.Cond{"version <>": version}).
		And(wdc.oldOffload()).
		Exec()

[shuangkun]

Sometimes, the workflow may not be updated for many cycles, which is longer than OFFLOAD_NODE_STATUS_TTL. If the old version is deleted and the workflow is changed at this time, a conflict will occur and the workflow cannot be restored.