after print "rollback checkpoint" in log, unistore folder size grows

[lance6716]

What did you do?

tiflow/dm/syncer/checkpoint.go

Lines 715 to 727 in 6308967

	logger.Info("rollback checkpoint", zap.Stringer("from", from), zap.Stringer("to", point.FlushedMySQLLocation()))
	// schema changed
	if err := schemaTracker.DropTable(table); err != nil {
	logger.Warn("failed to drop table from schema tracker", log.ShortError(err))
	}
	if point.ti != nil {
	// TODO: Figure out how to recover from errors.
	if err := schemaTracker.CreateSchemaIfNotExists(schemaName); err != nil {
	logger.Error("failed to rollback schema on schema tracker: cannot create schema", log.ShortError(err))
	}
	if err := schemaTracker.CreateTableIfNotExists(table, point.ti); err != nil {
	logger.Error("failed to rollback schema on schema tracker: cannot create table", log.ShortError(err))
	}

mock the logic of rollback checkpoint

	for i := 0; i < 10000; i++ {
		err = tracker.DropTable(tbl)
		c.Assert(err, IsNil)

		err = tracker.CreateSchemaIfNotExists("test")
		c.Assert(err, IsNil)
		err = tracker.CreateTableIfNotExists(tbl, oriTi)
		c.Assert(err, IsNil)
	}

during the test

# lance @ LAPTOP-AK9BOS6J in ~/go/src/github.com/pingcap/tiflow on git:630896740
 x [19:38:27]
$ du -h /tmp/tidb-unistore-temp944770732
16M     /tmp/tidb-unistore-temp944770732/kv
16M     /tmp/tidb-unistore-temp944770732

# lance @ LAPTOP-AK9BOS6J in ~/go/src/github.com/pingcap/tiflow on git:630896740
 x [19:38:42]
$ du -h /tmp/tidb-unistore-temp944770732
104M    /tmp/tidb-unistore-temp944770732/kv
104M    /tmp/tidb-unistore-temp944770732

# lance @ LAPTOP-AK9BOS6J in ~/go/src/github.com/pingcap/tiflow on git:630896740
 x [19:40:22]
$ du -h /tmp/tidb-unistore-temp944770732
195M    /tmp/tidb-unistore-temp944770732/kv
195M    /tmp/tidb-unistore-temp944770732

So if the task is resumed for many times (which means the task has a resumable error), the size of unistore folder will grow.

What did you expect to see?

the size of /tmp/tidb-unistore-temp944770732 has an upper-bound

What did you see instead?

No response

Versions of the cluster

v5.3.1

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[lance6716]

since we found the first and most severe problem in #5334 , I think changing implementation of schema tracker is less urgent.

[lance6716]

	for i := 0; i < 10000; i++ {
		err = tracker.Exec(ctx, "test", "DROP TABLE t")
		c.Assert(err, IsNil)

		err = tracker.Exec(ctx, "", "CREATE DATABASE IF NOT EXISTS test")
		c.Assert(err, IsNil)
		err = tracker.Exec(ctx, "test", "CREATE TABLE t (c varchar(20000)) charset=utf8")
		c.Assert(err, IsNil)
	}

same effects

[lance6716]

after some discussion, this space may be occupied by DDL history job and we have no way to clean it. Maybe the fix is that we close schema tracker and remove the unistore folder after the task is paused.

Or we think about #5334

[GMHDBJD]

Maybe the fix is that we close schema tracker

looks good to me

[lance6716]

after some discussion, this space may be occupied by DDL history job and we have no way to clean it. Maybe the fix is that we close schema tracker and remove the unistore folder after the task is paused.

Or we think about #5334

just created 500 tables in a task (ADMIN SHOW DDL JOBS has 535 rows), the disk usage is 257M...

we should adjust some default options

https://github.com/pingcap/tidb/blob/f6ce33761b7c4672c98199cdd194029dc5ef546a/store/mockstore/unistore/config/config.go#L141

[lance6716]

Maybe the fix is that we close schema tracker

looks good to me

Just found that operate-schema should let DM-master directly modify the table checkpoint after this 😂

plan to forward to DM-worker to reuse router and parser