[Dynamic Regions] many sst files left in the import directory after br restore

[glorv]

Bug Report

What version of TiKV are you using?

nightly

What operating system and CPU are you using?

Steps to reproduce

I used br to restore about 500GB data into a fresh cluster with partition-raft-kv enabled. After br finished I found there were about 1200 sst filed under the import dir with a total size about 55GB. And after 24h, these sst files were still not cleaned up.

And when I restart the cluster I could see a lot of ingest sst operations in the tikv log. This also make the restart much longer than expected before the foreground qps is stable.

What did you expect?

No or little sst files should be left for a long time after br restore.

What did happened?

[glorv]

@overvenus @tonyxuqqi @SpadeA-Tang Do you have any suggestions?
Also cc @3pointer

[overvenus]

As discussed offline, one possible solution to this issue is that TiKV can periodically (or triggered by ingest sst event) manual-flush tablet so that flushed index can be advanced in time, as described in #14914

[tonyxuqqi]

As discussed offline, one possible solution to this issue is that TiKV can periodically (or triggered by ingest sst event) manual-flush tablet so that flushed index can be advanced in time, as described in #14914

@overvenus The problem is if there's no memtable, then flush will not help.

[overvenus]

As discussed offline, one possible solution to this issue is that TiKV can periodically (or triggered by ingest sst event) manual-flush tablet so that flushed index can be advanced in time, as described in #14914

@overvenus The problem is if there's no memtable, then flush will not help.

It does not matter whether there is a memtable or not, once (synchronously) manual-flushed is done, we know all changes that are less or equal to the modified index are persisted to disk.