Defend against data inconsistency and improve its troubleshooting

[lysu]

Description

Currently, some new features may introduce data inconsistency bugs, and we only can find them after admin check or indexLookup, following DML maybe write more inconsistent data after reading old inconsistent records, it's better to have a mechanism to report the inconsistent bug before laid down data.

On the other hand, it's also hard for customers/supporters to troubleshoot data inconsistency problems. we often need multiple rounds of communication with customers to collect information, and even worse, sometimes we cannot identify the root cause due to lack of some useful information.

This issue tracks our progress in defending against data inconsistency and improve its troubleshooting.

Sub Tasks

Task 1: Defending against data inconsistency

Development

1) Assertion in Prewrite Mutation

• Add a new key error type AssertionFailed kvproto#815
• Update kvproto for optimizing assertion for pessimistic transactions kvproto#824
• txn: Do assertions on data for mutations to reduce potential data inconsistency tikv/tikv#10977
• txn: Support check existence without getting value during pessimistic lock requests tikv/tikv#11516
• txn_types: Add assertion field to Mutation type tikv/tikv#11518
• Support setting assertions and receiving assertion errors for Prewrite requests tikv/client-go#311 client-go part
• *: Support assertion for DMLs #28313
• store: Update unistore for assertion #29902
• Use a special type instead of FlagsOp, so that it's less likely to write wrong code.

2) Check Mutation for single row changes

• *: Check mutations for single-row changes #27920 Check there are not mutations that shouldn't exist, i.e. {mutations generated} is a subset of {mutations needed}
• *: Check mutations for single-row changes #27920 Check handle consistency: handles in index and row mutations match
• tables: check handle consistency #28465 Handles in index values are consistent with those in rows
• *: add a sysvar to enable mutation checker #28663 Enablement mechanism
• *: Check index number in txn #29453 Check the constraint of numbers of mutations
• *: Improve errors for mutation checker #30434

Test

• Ensure it can report the error when there is something wrong
  • Inject the old bug in the previous issue to see whether it can report errors like desired
• Ensure it doesn't misreport where everything is ok
  • Write unit test to ensure code snippet, add test hook to support check mutation value in test
  • Retest exists write path test case
  • Add more multiple stmt in one txn test case + check mutation by hook
• Ensure it doesn't have performance regression
  • Sysbench/TPCC to ensure small txn doesn't have regression
  • BigTxn test case to ensure memory cause doesn't increase a lot (+1 Byte per key)

Task 2: Improve data inconsistency troubleshooting

1) Improve consistent check fail error & log

Development

• executor: enhance data inconsistency report #27388
  • Unify different check fail reporter, ensure error message contain necessary info like db/table/index (optional readable row identifier)
  • Request MVCC info for unmatched record and index and dump them in log

Test

• Check can be added in Task1's test case to ensure error messages or logs have enough information

2) Improve MVCC fetch API

Task 3: Enhance data inconsistency recover

• Support recover a region in tables
• Speed up admin recover

[tisonkun]

Hi @lysu , is there a design document for this effort or an issue to create a design?

Also it looks like a lot of work to do, would you like to take a look at developing in a feature branch?

[ekexium]

There is a design doc but is internal and in Chinese.

We are trying to develop at a separate feature branch pingcap:ft-data-inconsistency.