Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

will not quit if fail to prepare, like disk space is full #13

Open
july2993 opened this issue Mar 23, 2020 · 0 comments
Open

will not quit if fail to prepare, like disk space is full #13

july2993 opened this issue Mar 23, 2020 · 0 comments

Comments

@july2993
Copy link
Collaborator

run:
./tpccgen all --lightning-ip 172.16.5.70 --importer-ip 172.16.5.69 --data-dir /data2/tpcc_order_line --tidb-ip 172.16.5.70 --tidb-port 4000 --deploy-dir /data1/deploy --db tpcc2 --threads 40 --warehouse 14336 > out

It will not quit if fail to prepare, like disk space is full.
should quit if failed to generate CSV but not continue to restore.

load to warehouse in warehouse 10340

load to stock in warehouse 10340

execute prepare failed, err load stock at warehouse 10241 failed write /data2/tpcc_order_line/tpcc2.stock.0.csv: no space left on device

execute prepare failed, err load stock at warehouse 10354 failed write /data2/tpcc_order_line/tpcc2.stock.33.csv: no space left on device

execute prepare failed, err load stock at warehouse 10294 failed write /data2/tpcc_order_line/tpcc2.stock.13.csv: no space left on device

execute prepare failed, err load stock at warehouse 10287 failed write /data2/tpcc_order_line/tpcc2.stock.6.csv: no space left on device

and start restore:

Skip preparing checking. Please load CSV data into database and check later.

Finished

#============
generate csv files finished
#============
#============
start tikv-importer
#============
/bin/ssh 172.16.5.69 sh /data1/deploy/scripts/stop_importer.sh

lightning log( fail checksum ?):

[2020/03/23 11:35:38.679 +08:00] [INFO] [restore.go:506] [progress] [files="1281/1441 (88.9%)"] [tables="8/9 (88.9%)"] [speed(MiB/s)=84.36864763279368] [state=writing] [remaining=7m30s]
[2020/03/23 11:36:45.720 +08:00] [WARN] [util.go:115] ["compute remote checksum failed but going to try again"] [table=`tpcc2`.`stock`] [query="ADMIN CHECKSUM TABLE `tpcc2`.`stock`"] [retryCnt=0] [error="Error 9005: Region is unavailable"]
[2020/03/23 11:36:45.720 +08:00] [WARN] [util.go:106] ["compute remote checksum retry start"] [table=`tpcc2`.`stock`] [query="ADMIN CHECKSUM TABLE `tpcc2`.`stock`"] [retryCnt=1]
[2020/03/23 11:37:33.689 +08:00] [INFO] [restore.go:1510] ["remote checksum completed"] [table=`tpcc2`.`stock`] [takeTime=4m51.909715449s] []
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:602] ["restore table failed"] [table=`tpcc2`.`stock`] [takeTime=1h1m48.731959817s] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:697] ["restore all tables data failed"] [takeTime=1h1m55.013744875s] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:277] ["run failed"] [step=2] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [INFO] [restore.go:475] ["everything imported, stopping periodic actions"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:283] ["the whole procedure failed"] [takeTime=1h1m55.029686236s] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:114] ["tables failed to be imported"] [count=1]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [restore.go:116] [-] [table=`tpcc2`.`stock`] [status=checksum] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"]
[2020/03/23 11:37:33.691 +08:00] [ERROR] [main.go:75] ["tidb lightning encountered error"] [error="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)"] [errorVerbose="checksum mismatched remote vs local => (checksum: 3775979478809315642 vs 6530941926594385115) (total_kvs: 2062344285 vs 2062344286) (total_bytes:426414213518 vs 426414213563)\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*TableRestore).compareChecksum\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:1452\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*TableRestore).postProcess\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:1044\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*TableRestore).restoreTable\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:747\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).restoreTables.func1\n\t/Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:601\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant