redo apply panic: "insert events should be emitted after other kinds of events in the same transaction"

[fubinzh]

What did you do?

1. create mysql sink changefeed with redo log on
2. run gotpc workload
3. After 30m, pause changefeed and run redo apply

What did you expect to see?

redo apply should succeed.

What did you see instead?

redo apply failed with panic:

[2024/05/13 04:58:14.678 +00:00] [INFO] [worker.go:94] [\"Transaction dmlSink worker exits as canceled\"] [changefeedID=.] [workerID=11], stderr: panic: insert events should be emitted after other kinds of events in the same transaction\n\ngoroutine 434 [running]:\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x0?, 0x0?, {0x0?, 0x0?, 0xc00015af00?})\n\tgo.uber.org/[email protected]/zapcore/entry.go:196 +0x54\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0018fe1a0, {0x0, 0x0, 0x0})\n\tgo.uber.org/[email protected]/zapcore/entry.go:262 +0x3ec\ngo.uber.org/zap.(*Logger).Panic(0xc006eda630?, {0x525a3c8?, 0x40c9a74?}, {0x0, 0x0, 0x0})\n\tgo.uber.org/[email protected]/logger.go:285 +0x51\ngithub.com/pingcap/log.Panic({0x525a3c8?, 0xc00006b7a0?}, {0x0?, 0xc0017e46c0?, 0x0?})\n\tgithub.com/pingcap/[email protected]/global.go:54 +0x85\ngithub.com/pingcap/tiflow/pkg/applier.(*updateEventSplitter).checkEventOrder(0xc0010ad180?, 0x5d61178?)\n\tgithub.com/pingcap/tiflow/pkg/applier/redo.go:654 +0x4f\ngithub.com/pingcap/tiflow/pkg/applier.(*updateEventSplitter).readNextRow(0xc003558200, {0x5d61178, 0xc001046f00})\n\tgithub.com/pingcap/tiflow/pkg/applier/redo.go:694 +0xa8\ngithub.com/pingcap/tiflow/pkg/applier.(*RedoApplier).consumeLogs(0xc0013d2120, {0x5d61178, 0xc001046f00})\n\tgithub.com/pingcap/tiflow/pkg/applier/redo.go:210 +0x42a\ngithub.com/pingcap/tiflow/pkg/applier.(*RedoApplier).Apply.func3()\n\tgithub.com/pingcap/tiflow/pkg/applier/redo.go:741 +0x1f\ngolang.org/x/sync/errgroup.(*Group).Go.func1()\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56\ncreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96, ExitCode: 2"] [stack="github.com/pingcap/test-infra/caselib/pkg/host.(*TiCDCHost).Redo\n\tgithub.com/pingcap/test-infra/caselib/pkg/host/ticdc.go:107\ngithub.com/pingcap/test-infra/caselib/pkg/steps.(*redo).Execute\n\tgithub.com/pingcap/test-infra/caselib/pkg/steps/redo.go:28\ngithub.com/pingcap/test-infra/caselib/pkg/steps.withRecover\n\tgithub.com/pingcap/test-infra/caselib/pkg/steps/step.go:24\ngithub.com/pingcap/test-infra/caselib/pkg/steps.(*Serial).Execute\n\tgithub.com/pingcap/test-infra/caselib/pkg/steps/step.go:45\nmain.main\n\ttest-infra/caselib/ticdc/main.go:112\nruntime.main\n\truntime/proc.go:267"]

Versions of the cluster

[release-version=v8.1.0] [git-hash=faabbc0bae6d77ed6d8cac005bfc31fe8b075b63]

[fubinzh]

/label affects-8.1

[fubinzh]

/severity major

[kennedy8312]

/type regression

[ti-chi-bot]

@kennedy8312: The label(s) type/regression cannot be applied, because the repository doesn't have them.

In response to this:

/type regression

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[fubinzh]

Regression issue introduced by #11027