design: add a proposal to support a command 'ADMIN RESTORE TABLE table_id' to speed up recover faulty dropped table. #7383
Conversation
…e_id' to speed up recover faulty dropped table.
|
|
||
| Let's take a look what `DROP TABLE` statement does. `DROP TABLE` statement first remove the dropping table meta data from the coresponding database meta data. After the schemas are synced by all the TiDB instances, in `worker.deleteRange`, TiDB will insert a delete range that construct from the first row key to end row key of the dropping table into the table `mysql.gc_delete_range`. At most `max(gcDefaultRunInterval, gcLifeTimeKey)` time later, the GC worker will delete the table data finally. | ||
|
|
||
| The meta data of the table is not really deleted, the meta key format is `Table:table_id`, as long as we can find out the id of the dropped table, we can recover the table information. The `admin show ddl jobs` statement can retrive the table id: |
There was a problem hiding this comment.
We need a better way to find table id. The Admin show ddl jobs is not a convenient way and it only shows the last 10 jobs.
There was a problem hiding this comment.
Maybe we can support a new command like admin show tableid by table_name, but it is another issue, by the way, "ADMIN" "SHOW" "DDL" "JOBS" NUM now can show the last NUMBER jobs.
There was a problem hiding this comment.
There could be multiple tables with the same name. For example:
create t;
drop t;
create t;
drop t;
.......There was a problem hiding this comment.
Return the latest one? Or an id list.
There was a problem hiding this comment.
I guess we could return multiple rows. One id for each row with the create/delete timestamp.
|
|
||
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. | ||
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it), if it is exists, return `ErrTableExists` error. | ||
| 3. Secondly, find out whether the delete-range of the dropped table is still in the `mysql.gc_delete_range`, if not, means that the GC worker is cleanup the data, we can not restore the table successfully, return a error to the client. If it is still there, we remove the record in the `mysql.gc_delete_range` table, if we successfully remove, continue to step 4, otherwise we return a error to the client to indicate the command can not execute safely. |
There was a problem hiding this comment.
The range will not be deleted until the delete-range command is finished. So I guess we should check the GC time instead of checking mysql.gc_delete_range.
|
@CaitinChen PTAL |
|
|
||
| ## Abstract | ||
|
|
||
| The proposal proposes to support `ADMIN RESTORE TABLE table_id` command, to restore the table that is dropped by faulty operation. |
There was a problem hiding this comment.
-> This proposal proposes to support the ADMIN RESTORE TABLE table_id command, to restore the table that is dropped by a faulty operation.
|
|
||
| ## Background | ||
|
|
||
| At present, if we drop the table in production environment, and realize the operations is faulty immediately. Before we support the proposed command, we can only [Reading Data From History Versions](https://pingcap.com/docs/op-guide/history-read/) to rescue the disaster. But it needs to read all the data in the storage, it spend too much time for our purpose that just restore the dropped table. |
There was a problem hiding this comment.
-> At present, if we drop the table in the production environment, we will realize whether the operation is faulty immediately. Before we support the proposed command, we can only read data from history versions to relieve the disaster. But it needs to read all the data in the storage and it takes too much time to just restore the dropped table.
There was a problem hiding this comment.
Note:
- sb. spend some time in doing sth.
- it takes sb. some time to do sth.
- rescue + sb. 营救某人
- relieve + disaster
|
|
||
| ## Proposal | ||
|
|
||
| We can add a new command `ADMIN RESTORE TABLE table_id` to just make the dropped table be public again. If the data is not deleted by GC worker, this command can work. So it is better to enlarge the gc life time with `update mysql.tidb set variable_value='30h' where variable_name='tikv_gc_life_time';`, before we executing the statement. And the table and the original table data can be restore in a few seconds, it is a lot faster than before. It also can reduce the complexity of the operations, dissolves the artificial operation error. |
There was a problem hiding this comment.
-> We can add a new command ADMIN RESTORE TABLE table_id to just make the dropped table public again. If the data is not deleted by the GC worker, this command can work. So it is better to enlarge the GC life time with update mysql.tidb set variable_value='30h' where variable_name='tikv_gc_life_time';, before we execute the statement. The table and the original table data can be restored in a few seconds and it is a lot faster than before. It also can reduce the complexity of the operations and dissolve the artificial operation error.
|
|
||
| ## Rationale | ||
|
|
||
| Let's take a look what `DROP TABLE` statement does. `DROP TABLE` statement first remove the dropping table meta data from the coresponding database meta data. After the schemas are synced by all the TiDB instances, in `worker.deleteRange`, TiDB will insert a delete range that construct from the first row key to end row key of the dropping table into the table `mysql.gc_delete_range`. At most `max(gcDefaultRunInterval, gcLifeTimeKey)` time later, the GC worker will delete the table data finally. |
There was a problem hiding this comment.
-> Let's take a look at the workflow of the DROP TABLE statement. The DROP TABLE statement first removes the dropping table meta data from the corresponding database meta data. After the schemas are synced by all the TiDB instances, in worker.deleteRange, TiDB will insert a deleted range of the first row key to the end row key of the dropping table into the table mysql.gc_delete_range. At most max(gcDefaultRunInterval, gcLifeTimeKey) time later, the GC worker will delete the table data finally.
|
|
||
| Let's take a look what `DROP TABLE` statement does. `DROP TABLE` statement first remove the dropping table meta data from the coresponding database meta data. After the schemas are synced by all the TiDB instances, in `worker.deleteRange`, TiDB will insert a delete range that construct from the first row key to end row key of the dropping table into the table `mysql.gc_delete_range`. At most `max(gcDefaultRunInterval, gcLifeTimeKey)` time later, the GC worker will delete the table data finally. | ||
|
|
||
| The meta data of the table is not really deleted, the meta key format is `Table:table_id`, as long as we can find out the id of the dropped table, we can recover the table information. The `admin show ddl jobs` statement can retrive the table id: |
There was a problem hiding this comment.
-> The meta data of the table is not really deleted. The meta key format is Table:table_id. As long as we can find out the ID of the dropped table, we can recover the table information. The admin show ddl jobs statement can retrieve the table ID:
| ## Implementation | ||
|
|
||
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. | ||
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it), if it is exists, return `ErrTableExists` error. |
There was a problem hiding this comment.
-> 2. Before we start to do this job, we need to check if the table name already exists (created a new table with the same table name after you drop it). If it exists, return the ErrTableExists error.
|
|
||
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. | ||
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it), if it is exists, return `ErrTableExists` error. | ||
| 3. Secondly, find out whether the delete-range of the dropped table is still in the `mysql.gc_delete_range`, if not, means that the GC worker is cleanup the data, we can not restore the table successfully, return a error to the client. If it is still there, we remove the record in the `mysql.gc_delete_range` table, if we successfully remove, continue to step 4, otherwise we return a error to the client to indicate the command can not execute safely. |
There was a problem hiding this comment.
-> 3. Secondly, find out whether the delete-range of the dropped table is still in the mysql.gc_delete_range. If not, it means that the GC worker has cleaned up the data. In this situation, we cannot restore the table successfully but return an error to the client. If it is still there, we remove the record in the mysql.gc_delete_range table. If we successfully remove the record, continue to Step 4, otherwise we return an error to the client to indicate the command cannot be executed safely.
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. | ||
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it), if it is exists, return `ErrTableExists` error. | ||
| 3. Secondly, find out whether the delete-range of the dropped table is still in the `mysql.gc_delete_range`, if not, means that the GC worker is cleanup the data, we can not restore the table successfully, return a error to the client. If it is still there, we remove the record in the `mysql.gc_delete_range` table, if we successfully remove, continue to step 4, otherwise we return a error to the client to indicate the command can not execute safely. | ||
| 4. Remove the delete-range record in the `mysql.gc_delete_range` table, to prevent GC worker deleting the real data. |
There was a problem hiding this comment.
- Remove the delete-range record in the
mysql.gc_delete_rangetable, to prevent the GC worker deleting the real data.
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it), if it is exists, return `ErrTableExists` error. | ||
| 3. Secondly, find out whether the delete-range of the dropped table is still in the `mysql.gc_delete_range`, if not, means that the GC worker is cleanup the data, we can not restore the table successfully, return a error to the client. If it is still there, we remove the record in the `mysql.gc_delete_range` table, if we successfully remove, continue to step 4, otherwise we return a error to the client to indicate the command can not execute safely. | ||
| 4. Remove the delete-range record in the `mysql.gc_delete_range` table, to prevent GC worker deleting the real data. | ||
| 5. Use the previous table meta infomation of the table_id to insert the meta data into the schema meta data, like what `Meta.CreateTable` does. And make the table infomation state to be `model.StatePublic`, then the restore is finished after the schema is synced by all the TiDB instances. |
There was a problem hiding this comment.
-> 5. Use the previous table meta information of the table_id to insert the meta data into the schema meta data, like what Meta.CreateTable does. And set the table information state to model.StatePublic, then the restoration will be finished after the schema is synced by all the TiDB instances.
| 3. Secondly, find out whether the delete-range of the dropped table is still in the `mysql.gc_delete_range`, if not, means that the GC worker is cleanup the data, we can not restore the table successfully, return a error to the client. If it is still there, we remove the record in the `mysql.gc_delete_range` table, if we successfully remove, continue to step 4, otherwise we return a error to the client to indicate the command can not execute safely. | ||
| 4. Remove the delete-range record in the `mysql.gc_delete_range` table, to prevent GC worker deleting the real data. | ||
| 5. Use the previous table meta infomation of the table_id to insert the meta data into the schema meta data, like what `Meta.CreateTable` does. And make the table infomation state to be `model.StatePublic`, then the restore is finished after the schema is synced by all the TiDB instances. | ||
| 6. If the command is canceled or rollbacked, and the delete-range record is already removed, we need to insert it into the `mysql.gc_delete_range` again, like what `Drop Table` does in `worker.finishDDLJob`. |
There was a problem hiding this comment.
-> 6. If the command is canceled or rollbacked, and the delete-range record is already removed, we need to insert it into mysql.gc_delete_range again, like what Drop Table does in worker.finishDDLJob.
|
@CaitinChen @shenli PTAL |
|
|
||
| ## Background | ||
|
|
||
| At present, if we drop the table in production environment, we will realize whether the operation is faulty immediately. Before we support the proposed command, we can only [reading data from history versions](https://pingcap.com/docs/op-guide/history-read/) to relieve the disaster. But it needs to read all the data in the storage and it takes too much time to just restore the dropped the table. |
There was a problem hiding this comment.
reading data from history versions -> read data from history versions
|
|
||
| ## Proposal | ||
|
|
||
| We can add a new command `ADMIN RESTORE TABLE table_id` to just make the dropped table public again. If the data is not deleted by GC worker, this command can work. So it is better to enlarge the GC life time with `update mysql.tidb set variable_value='30h' where variable_name='tikv_gc_life_time';`, before we executing the statement. The table and the original table data can be restored in a few seconds and it is a lot faster than before. It also can reduce the complexity of the operations and dissolves the artificial operation error. |
There was a problem hiding this comment.
before we executing -> before we execute
dissolves -> dissolve
|
|
||
| Let's take a look at the workflow of the `DROP TABLE` statement. The `DROP TABLE` statement first removes the dropping table meta data from the coresponding database meta data. After the schemas are synced by all the TiDB instances, in `worker.deleteRange`, TiDB will insert a deleted range of the first row key to the end row key of the dropping table into the table `mysql.gc_delete_range`. At most `max(gcDefaultRunInterval, gcLifeTimeKey)` time later, the GC worker will delete the table data finally. | ||
|
|
||
| The meta data of the table is not really deleted. The meta key format is `Table:table_id`. As long as we can find out the ID of the dropped table, we can recover the table information. The `admin show ddl jobs` statement can retrive the table ID: |
|
|
||
| As you can see, if we can restore the table before the GC worker deletes the table data, we can restore the table completely. If the table data is deleted, we can only restore an empty table. | ||
|
|
||
| Before we run the `ADMIN RESTORE TABLE table_id`, you must: |
There was a problem hiding this comment.
Please remove "the".
Note: If sth. is enclosed by backquotes, it can be considered as a proper noun like "China". We don't say "the China" (when "China" is a country name); likewise, we don't add "the" before XXX.
| As you can see, if we can restore the table before the GC worker deletes the table data, we can restore the table completely. If the table data is deleted, we can only restore an empty table. | ||
|
|
||
| Before we run the `ADMIN RESTORE TABLE table_id`, you must: | ||
| * Ensure that there is no any GC tasks is running, you can figure out this by using TiDB logs and metrics. |
There was a problem hiding this comment.
-> Ensure that no GC task is running and then you can figure this by using TiDB logs and metrics.
Note: "There is no any case study based on this" is ungrammatical in standard English. "There is no case study based on this" is fine. "There is not any case study based on this" means "There is no case study of any kind based on this." See https://www.google.com/search?q=is+no+any&oq=is+no+any&aqs=chrome..69i57j0l5.2966j0j8&sourceid=chrome&ie=UTF-8
|
|
||
| Before we run the `ADMIN RESTORE TABLE table_id`, you must: | ||
| * Ensure that there is no any GC tasks is running, you can figure out this by using TiDB logs and metrics. | ||
| * Enlarge the `tikv_gc_life_time` to a sufficient value. |
There was a problem hiding this comment.
Enlarge the tikv_gc_life_time -> Increase tikv_gc_life_time
| ## Implementatio | ||
|
|
||
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. | ||
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it). If it exists, return `ErrTableExists` error. |
There was a problem hiding this comment.
-> 2. Before we start to do this job, we need to check if the table name already exists (created a new table with the same table name after you drop it). If it exists, return the ErrTableExists error.
Note: "sth. exists" is right; "sth. is exists" is wrong.
|
|
||
| It's a new command and will not lead to compatibility issues. | ||
|
|
||
| ## Implementatio |
|
|
||
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. | ||
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it). If it exists, return `ErrTableExists` error. | ||
| 3. Secondly, find out whether the delete-range of the dropped table is still in the `mysql.gc_delete_range`. If not, it means that the GC worker has cleanup the data. In this situation, we cannot restore the table successfully but return an error to the client. If it is still there, we remove the record in the `mysql.gc_delete_range` table. If we successfully remove the record, continue to step 4, otherwise we return an error to the client to indicate the command cannot execute safely. |
There was a problem hiding this comment.
"Secondly" can be removed because you have used an ordered list here. This paragraph is Step 3.
-> Find out whether the delete-range of the dropped table is still in mysql.gc_delete_range. If not, it means that the GC worker has cleaned up the data. In this situation, we cannot restore the table successfully but return an error to the client. If it is still there, we remove the record in the mysql.gc_delete_range table. If we successfully remove the record, continue to Step 4; otherwise, we return an error to the client to indicate the command cannot be executed safely.
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. | ||
| 2. Before we start to do this job, we need to check if the table name is already exists(created a new table with the same table name after you drop it). If it exists, return `ErrTableExists` error. | ||
| 3. Secondly, find out whether the delete-range of the dropped table is still in the `mysql.gc_delete_range`. If not, it means that the GC worker has cleanup the data. In this situation, we cannot restore the table successfully but return an error to the client. If it is still there, we remove the record in the `mysql.gc_delete_range` table. If we successfully remove the record, continue to step 4, otherwise we return an error to the client to indicate the command cannot execute safely. | ||
| 4. Use the previous table meta infomation of the table_id to insert the meta data into the schema meta data, like what `Meta.CreateTable` does. And set the table information state to `model.StatePublic`, then the restoration will be finished after the schema is synced by all the TiDB instances. |
There was a problem hiding this comment.
-> Use the previous table meta information of the table_id to insert the meta data into the schema meta data, like what Meta.CreateTable does. And set the table information state to model.StatePublic, then the restoration will be finished after the schema is synced by all the TiDB instances.
|
@CaitinChen PTAL |
|
@shenli PTAL |
|
|
||
| ## Implementation | ||
|
|
||
| 1. `ADMIN RESTORE TABLE table_id` will enqueue a new DDL job to TiDB general DDL queue. |
There was a problem hiding this comment.
Could we put it to the beginning of the queue?
There was a problem hiding this comment.
Job in the general queue will be handled quickly, maybe treat it as a normal job is ok.
shenli
added
status/LGT2
What problem does this PR solve?
Add a proposal to
docs/design, proposes to supportADMIN RESTORE TABLE table_id.Check List
Tests
Code changes
Side effects
Related changes