Try to recover automatically from the background error about sst corrupt

[caipengbo]

When the SST file corrupts, which is an unrecoverable error for the rocksdb, then rocksdb will go into read-only mode(https://github.com/facebook/rocksdb/wiki/Background-Error-Handling). Only restart rocksdb to try to recover from the error.

When does sst file corruption occur? The error message looks like this:

1. Corruption: Corrupt or unsupported format_version: 1005 in /tmp/kvrocks/data/db/000038.sst
2. Corruption: Bad table magic number: expected 9863518390377041911, found 9863518390377041912 in /tmp/kvrocks_db/data/db/000038.sst
3. Corruption: block checksum mismatch: stored = 3308200672, computed = 51173877, type = 4 in /tmp/kvrocks_db/data/db/000038.sst offset 0 size 15715

The cause of the error is usually a hardware issue or a problem with the network or cloud disk (when using the cloud disk).

The most common place we see this error is when a file is generated by Compaction or Flush and the Version applies the result.

In this case, the result of the compaction is not actually applied, so we can ignore the error and avoid restarting the rocksdb.

Tikv introduces this check when sst file corruption occurs, you can refer to:

• storage: support automatic sst corruption recovery tikv/tikv#10578
• storage: support automatic sst corruption check and deletion tikv/tikv#10961

Let's try it on Kvrocks:

1. Extract the sst file from the background error message
2. Determine if it is a living file
3. If not, we ignore the error and force recovery from the background error

For the rocksdb error message, before the rocksdb v7.10.2, the error message was imperfect and we could only recover from a limited number of errors. Thanks to this PR facebook/rocksdb#11009, the error message is enriched and we can recover from more scenarios.

[git-hulk]

@caipengbo Thanks for your solution, some users did suffer from this issue even though it's a low probability. For test cases, it's good for me, since it's hard to mock without hijacking the rocksdb.

[mapleFU]

I'm not sure it can be used. Seems that myrocks didn't apply the rule: https://github.com/facebook/mysql-5.6/blob/fb-mysql-8.0.28/storage/rocksdb/event_listener.cc#L116

TiKV has replicas, and it scan transfer the leader to replica, so I think there could be something different. I'll take a look into rocksdb later, does any one else use this pattern to recover from Corrupt?

[caipengbo]

Seems that myrocks didn't apply the rule

Yes, myrocks will abort. I've looked into myrocks and other systems before and only found tikv with a special check.

I'm not sure it can be used.

In the past, we had a NIC failure in production, which caused this error to become more frequent. So we added this mechanism, and we will monitor the background error when it appears, and furthermore, we will archive the error file so that we can locate the problem.

Since then, there have been a few similar errors, all of which have been recovered safely. @mapleFU

[mapleFU]

So the problem here is that we have background network IO, which might cause error much more-frequently than disk io error?
This idea looks ok, but I prefer a default disabled config for this. Because it might cause some unexpected behavior.

[caipengbo]

IMHO, it will not affect the correctness of the data. The possible effect is that users tend to overlook background errors they encounter. HDYT @git-hulk @PragmaTwice

[git-hulk]

It should be fine since this only impacts the io error scenario, and the instance also would be broken if got those errors.

[mapleFU]

LGTM