leveldb/table: corruption on data-block

[gorilla2112]

System information

Geth version: 0.9.41
OS & Version: Windows10

Expected behavior

connection to blockchain

Actual behavior

Error:
eth/downloader/downloader.go:274] Synchronisation failed: leveldb/table: corruption on data-block (pos=0): checksum mismatch, want=0x8f40a97e got=0x792b89e6 [file=4762443.ldb]

Steps to reproduce the behavior

geth --rpc

[timjp87]

I have the same problem, but here it seems to be related to the ARMv7hf architecture.
On my ODROID-C1+ running Arch Linux ARM I built geth 1.5.0 unstable from source and fast sync to a mounted USB stick. After some time I get the error listed above. It's not always on the same block. Sometimes after 300K blocks some times after 700K blocks. I tried syncing to the SD Card and get the same error. I used a different file system on the USB stick and get the same error. When I plug the stick into my desktop PC running Arch Linux (x86_64) and build 1.5.0-unstable from the same version of Go and everything it syncs just fine to the USB stick. As of today I also have a ODROID-C2 running Arch Linux ARM when I build for aarch64 and try to sync to the same USB stick it works just fine. On my Galaxy Nexus (armv7hf) I have an Arch Linux ARM chroot created via the Linux Deploy app. I could not reproduce it there, but the main problem is that wifi connection seems to go away when the phone is in deep sleep. I only managed to fast sync 200K blocks to it over night. I have yet another ODROID-C1 (non-plus), which has the same CPU as the C1+, where I will try to reproduce it.
EDIT: ODRIOD C1 syncing to the SD Card has the same issue.

I0525 23:16:57.385415 core/blockchain.go:751] imported 1 receipt(s) (0 ignored) in 1.993011ms. #129025 [03bb06dc… / 03bb06dc…]
F0525 23:17:00.034115 core/database_util.go:281] failed to store header into database: leveldb/table: corruption on data-block (pos=950885): checksum mismatch, want=0xd74db9c6 got=0x1993e768 [file=000137.ldb]
F0525 23:17:00.061065 core/database_util.go:296] failed to store block body into database: leveldb/table: corruption on data-block (pos=950885): checksum mismatch, want=0xd74db9c6 got=0x1993e768 [file=000137.ldb]
F0525 23:17:00.120144 core/blockchain.go:696] failed to write log blooms:

[uros09]

It is not limited to arm and windows. I am trying to sync on Ubuntu Linux server 15.04 with geth --fast console 3 days and always get this error on different blocks. On the other hand on other machine geth sync as it should. I tried to change hard disk, bought fresh new SSD and error is here again. I don't know what to do.I have celeron 2 core CPU and 4GB of RAM. I have 128 GB SSD.
Error:
F0531 07:28:40.245157 core/database_util.go:267] failed to store last fast block's hash into database: leveldb/table: corruption on data-block (pos=1579129): checksum mismatch, want=0x17457fdb got=0xa7ab44c8 [file=004558.ldb]

CPU and OS:
model name : Intel(R) Celeron(R) CPU G460 @ 1.80GHz
Linux Ubuntu 15.04 server kernel:3.19.0-15-generic

After that error I can only close hang ssh connection, when I press key nothing happens. Just ctrl+c give me new prompt on which I can't type anything...

Last parts of log in chaindata after another attempt after removing chain data are:

08:57:40.185549 table@compaction L1·1 -> L2·6 S·14MiB Q·10503753
08:57:40.282924 table@build created L2@8406 N·19356 S·2MiB "\x97S\xe8..gv\x01,v5153539":"\x9a;o..F\x14\x01,v5750866"
08:57:40.431193 table@build created L2@8407 N·19263 S·2MiB "\x9a;r..2\xb4;,v11996":"\x9d";..\xf53\x87,v10262999"
08:57:40.572721 table@build created L2@8408 N·19239 S·2MiB "\x9d";..3\x87\x01,v10263000":"\xa0\x0f\xeb..\xac\xbfZ,v3668708"
08:57:40.676895 table@build created L2@8409 N·19546 S·2MiB "\xa0\x0f\xeb..\xbfZ\x01,v3668709":"\xa3\r\xec..\xbfn1,v7654478"
08:57:40.812549 table@build created L2@8410 N·19266 S·2MiB "\xa3\r\xec..n1\x01,v7654479":"\xa5\xf6\xe1..\x91\x91R,v10051025"
08:57:40.923261 table@build created L2@8411 N·19217 S·2MiB "\xa5\xf6\xe1..\x91R\x01,v10051026":"\xa8\xea\xc1..\x05\x84),v5479304"
08:57:40.998337 table@build created L2@8412 N·19425 S·2MiB "\xa8\xea\xc1..\x84)\x01,v5479305":"\xab\xebL..\xcc,U,v7194244"
08:57:41.004745 table@build created L2@8413 N·555 S·60KiB "\xab\xebL..,U\x01,v7194245":"\xac\x05\x89..+\x1d\xba,v780458"
08:57:41.005191 version@stat F·[2 62 334] S·767MiB[23MiB 101MiB 642MiB] Sc·[0.50 1.01 0.64]
08:57:41.008010 table@compaction committed F+1 S+55KiB Ke·0 D·0 T·822.404277ms
08:57:41.008601 table@remove removed @8271
08:57:41.009606 table@remove removed @7351
08:57:41.010526 table@remove removed @7352
08:57:41.011399 table@remove removed @7353
08:57:41.012235 table@remove removed @7354
08:57:41.013068 table@remove removed @7355
08:57:41.013890 table@remove removed @7356
08:57:41.014491 table@compaction L1·1 -> L2·7 S·12MiB Q·10509837
08:57:41.107308 table@build created L2@8414 N·19329 S·2MiB "\xa8\xea\xc1..\x84)\x01,v5479305":"\xab\xd4\xe0..\x8f1\x96,v7067561"
08:57:41.238741 table@build created L2@8415 N·19354 S·2MiB "\xab\xd4\xe0..1\x96\x01,v7067562":"\xae\xc3$..>\xaf\x01,v10151494"
08:57:41.408585 table@build created L2@8416 N·18938 S·2MiB "\xae\xc3$..\xfe\x84%,v8387362":"\xb1\x9c\xa6..\x97\xb1\v,v4733503"
08:57:41.520908 table@build created L2@8417 N·19185 S·2MiB "\xb1\x9c\xa6..\xb1\v\x01,v4733504":"\xb4\x88<..\xaf\x00\x01,v3904484"
08:57:41.558971 table@build error I·88689 "leveldb/table: corruption on data-block (pos=1294670): checksum mismatch, want=0xa52f51f7 got=0xa88a235c [file=007360.ldb]"
08:57:41.559000 table@build exiting (corruption detected)
08:57:41.559013 table@build revert @8414
08:57:41.559493 table@build revert @8415
08:57:41.559998 table@build revert @8416
08:57:41.560411 table@build revert @8417

[thomas-shirley]

I'm also experiencing this issue, running on a rPi 2 - chaindata writing to a USB mount.

Geth version 1.3.3
Go version 1.4

F0630 22:54:51.446400 3865 database_util.go:362] failed to store block body into database: leveldb/table: corruption on data-block (pos=2028406): checksum mismatch, want=0x4b0adb2e got=0x38d5ebdd [file=015275.ldb]
F0630 22:54:51.448207 3865 blockchain.go:972] failed to write log blooms: mipmap write fail for: 1108696: leveldb/table: corruption on data-block (pos=2028406): checksum mismatch, want=0x4b0adb2e got=0x38d5ebdd [file=015275.ldb]
F0630 22:54:51.448685 3865 blockchain.go:972] failed to write log blooms: mipmap write fail for: 1108698: leveldb/table: corruption on data-block (pos=2028406): checksum mismatch, want=0x4b0adb2e got=0x38d5ebdd [file=015275.ldb]
F0630 22:54:51.448821 3865 database_util.go:306] failed to store number to hash mapping into database: leveldb/table: corruption on data-block (pos=2028406): checksum mismatch, want=0x4b0adb2e got=0x38d5ebdd [file=015275.ldb]

[thomas-shirley]

This error is still present in version: 1.4.9-stable-b7e3dfc5

[thu-traub-old]

I also got the error (Version 1.4.10 and 1.4.11).
Geth ist very unstable on windows.
Sometimes it crahes after a few seconds, sometimes after 30 min.

[gosuto-inzasheru]

Would like to report experiencing this issue too. Seems to happen after starting/stopping geth a couple of times using --fast but still before getting to the current block.

My geth version:

Geth
Version: 1.4.15-stable
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.6.2
OS: linux

[fjl]

We have moved to a newer Version of leveldb in geth 1.5.0.

[thiagodelgado111]

I'm having the "same" issue with geth 1.5.7 when I run geth --fast

Info

Geth
Version: 1.5.7-stable
Git Commit: da2a22c384a9b621ec853fe4b1aa651d606cf42b
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.7.4
OS: darwin

Details

F0124 13:41:36.198735 core/database_util.go:325] failed to store last fast
block's hash into database: leveldb/table: corruption on data-block
(pos=1543271): checksum mismatch, want=0xcb6520b5 got=0xf19752ee

[buxx]

Same problem with last stable (making sync or import will produce error):

err="leveldb/table: corruption on data-block (pos=783749): checksum mismatch, want=0xe2cae6af got=0xcc97e598 [file=027201.ldb]"

[hrissan]

Getting similar problem with gets 1.6.6 on Mac
INFO [07-01|13:39:08] Starting peer-to-peer node instance=Geth/v1.6.6-stable-10a45cb5/darwin-amd64/go1.8.3
....
INFO [07-01|23:27:19] Imported new state entries count=23 flushed=33 elapsed=117.812ms processed=7734624 pending=2279 retry=0 duplicate=38699 unexpected=114453
CRIT [07-01|23:27:19] Failed to write log blooms err="mipmap write fail for: 2628576: leveldb/table: Writer: keys are not in increasing order: "mipmap-log-bloom-\x00\x00\x00\x00\x00\x00\xc3P&\xe8\xf0\x01[M\x85\x04\x00\x00\x00", "mipiap-log-bloom-\x00\x00\x00\x00\x00\x00\xc3P&\xe8\xf0\x01<M\x85\x04\x00\x00\x00""

This is after I deleted the blockchain because node could not sync for another problem...
Deleted blockchain again. Started sync. After almost a day (on crappy ADSL connection)

elapsed=236.740ms processed=9723275 pending=601 retry=0 duplicate=47879 unexpected=132572
CRIT [07-02|15:29:52] Failed to store block receipts err="leveldb/table: Writer: keys are not in increasing order: "receipts-\xcf\x04\xb8T\xd2\xf4\xf7N\x95\xa2a\xc8\x15\xe5\xb0\xd4\xe8+oP{Ѡ\xeb\x13\xd8\xd9\xd0?F\xf1\x01\xf1>a\x06\x00\x00\x00", "raceipts-\xcf\xc7\x0e\x12\xda\xfa\x90ү\xc7/d\x05]Ţ\xf1\xf5\xfb&\xb6\xeb<P3~\x17B\xe7\x90F\x01\xb4\x99]\x06\x00\x00\x00""
CRIT [07-02|15:29:52] Failed to store header err="leveldb/table: Writer: keys are not in increasing order: "receipts-\xcf\x04\xb8T\xd2\xf4\xf7N\x95\xa2a\xc8\x15\xe5\xb0\xd4\xe8+oP{Ѡ\xeb\x13\xd8\xd9\xd0?F\xf1\x01\xf1>a\x06\x00\x00\x00", "raceipts-\xcf\xc7\x0e\x12\xda\xfa\x90ү\xc7/d\x05]Ţ\xf1\xf5\xfb&\xb6\xeb<P3~\x17B\xe7\x90F\x01\xb4\x99]\x06\x00\x00\x00""
CRIT [07-02|15:29:52] Failed to store transactions err="leveldb/table: Writer: keys are not in increasing order: "receipts-\xcf\x04\xb8T\xd2\xf4\xf7N\x95\xa2a\xc8\x15\xe5\xb0\xd4\xe8+oP{Ѡ\xeb\x13\xd8\xd9\xd0?F\xf1\x01\xf1>a\x06\x00\x00\x00", "raceipts-\xcf\xc7\x0e\x12\xda\xfa\x90ү\xc7/d\x05]Ţ\xf1\xf5\xfb&\xb6\xeb<P3~\x17B\xe7\x90F\x01\xb4\x99]\x06\x00\x00\x00""

Any ideas? (Checked SSD smart status - no relocated sectors added...)

[maz-net-au]

Windows 8.1 having the same issue using --fast and also when not. Cannot sync. Cannot use Ethereum.

Geth/v1.6.6-stable-10a45cb5/windows-amd64/go1.8.3

CRIT [07-02|20:58:35] Failed to store block body err="leveldb/table: corruption on data-block (pos=1020486): checksum mismatch, want=0xd897cf55 got=0xa9b3f8d2 [file=035242.ldb]"

[space4092]

same issue CRIT [07-12|00:11:22|core/database_util.go:329] Failed to store last fast block's hash err="leveldb/table: corruption on data-block (pos=759124): checksum mismatch, want=0x7fe9fb78 got=0x29fa1df7 [file=126096.ldb]"

[micahcatlin]

Same issue appears for me,

INFO [07-20|22:51:57] Imported new state entries               count=384  flushed=269 elapsed=10.590ms  processed=378684 pending=8547  retry=0   duplicate=74 unexpected=219
CRIT [07-20|22:51:57] Failed to store block total difficulty   err="leveldb/table: corruption on data-block (pos=1923214): checksum mismatch, want=0x805ab052 got=0x452150f6 [file=002298.ldb]"

Built from source on MacOS Sierra at commit:

commit ab5646c532292b51e319f290afccf6a44f874372
Author: Felix Lange <[email protected]>
Date:   Tue Jul 11 16:32:36 2017 +0200

    params: v1.6.7 stable

Is there any other state I could collect that would aid the developers in debugging?

[buxx]

@gorilla2112 Any motivation about close this issue ?

[baaa]

why is this issue closed? am i missing something? or doing something terribly wrong? Still getting this problem with Geth/v1.6.7-stable-ab5646c5/windows-amd64/go1.8.3 cannot sync and use geth or any other ethererum tool with or without --fast

tried deleting blockchain multiple times...

err="leveldb/table: corruption on data-block (pos=1722061): checksum mismatch, want=0x885061ff got=0x2f084e61 [file=102005.ldb]"

[activequant]

In my case, I deleted the corrupt file (such as 102005.ldb) and restarted geth --fast. It then continued syncing. I am not sure if this is how it should be, I can just hope that the leveldb files are sort of stream like and it will resync everything that was in the deleted file (102005.ldb) on start. Any confirmation from anyone that this is a good solution?

[andrewgouin]

Still getting this issue with geth 1.7.3 on Ubuntu 17.10 (on HDD) and on the same machine with macOS High Sierra (on SSD), and Windows 10 (on a different SSD) using geth --syncmode fast. It is a i7-2600K with 24GB RAM, Gigabyte GA-Z68XP-UD3. When on IPv6 it will hang and not continue syncing on a random block each time after deleting geth/chaindata/*. On IPv4 it will keep syncing but with plenty of the same DB write error: leveldb/table: corruption on data-block, retrieved hash chain is invalid , and block body download canceled (requested) warnings along the way. I have tried removing geth/chaindata/* many times to start over but the errors are the same. I have also tried through a VPN. On a AWS EC2 instance (t2.micro) it is syncing fine without any of these errors/warnings.

[andrewgouin]

I found my problem to be the power supply. I tried different memory sticks first which had no effect, and then unplugged the extra hard drives and it worked better. I was pulling too much power from my 600w supply. Recently added an RX 580 and another SSD. Upgraded to an 850w and it is syncing much faster now without issues.

[anewbug]

Getting the same error on Ubuntu 14.04, Geth – Weir (v1.7.3):

Failed to store last header's hash err="leveldb/table: Writer: keys are not in increasing order

[udikantz]

I was able to solve the problem on windows after 5 days trying to synch.
Final steps for solution involves:

1. disabling windows defender
2. disable windows firewall completely
3. disable windows malware service: tutorial
4. setting computer to never sleep , and also make sure hardisk never sleep in control panel/power options: tutorial
5. I was running geth manually with the following command: geth --fast --cache 1024 --datadir "F:\somedir"

*Synch took about 5 hours on a new ssd / 100Mbit connection

Not sure which specific action made me successfully synch,

I have done all the above after many days of trying to synch the damn node, I even bought new SSD and it did not help.

Hope you find my solution to work for you lost souls.
If it helped you, then feel free to help me pay for my unwanted new ssd drive:

btc: 12y5cQN8RcmUgatn71UbD6eMWhHGCHu8fE
eth: 0x4860812ebda3e94154708d9fabd7e86451d6af16

[winsvega]

have the same issue geth 1.8.0
Ubuntu 16

CRIT [01-04|00:06:14] Failed to store last header's hash       err="leveldb/table: corruption on data-block (pos=2093038): checksum mismatch, want=0x0 got=0x8ee806e1 [file=095550.ldb]"

seriously it is 5th time I am trying to sync geth and no luck. another issue again. geth is not syncing.

just deleting the file leads to this issue:
#14435

[micahcatlin]

There are some comments earlier in the thread that suggest this error might come up for all classes of memory corruption. Having a bad memory module or a noisy/overloaded power supply, or a weakly-connected SATA cable might all be the culprit -- but it's probably some kind of extremely rare intermittent hardware failure, even on a machine that otherwise "seems okay".

Flipping one bit in a billion won't impair an MPEG movie, but it will make a checksum fail.

[winsvega]

there are no cables. ssd is integrated into motherboard. the power supply uses a battery.
such issues with syncing could not be happening 5 times in a row.

[Rhnr]

@winsvega did you get a solution for this?

I have the same problem .. tried resetting the complete setup for more than 3 times now, but landing on the same issue again n again ...

Looks like a very common problem, but no where I can find a clean solution .. :(

Not sure why this ticket is closed!

[winsvega]

I heard the go team has fixed the light client again. so try syncing with the geth light client option enabled.

my solution was to switch to the parity client. as they are focusing their efford on faster blockchain syncing.
so I have parity client full sync without warp. (parity wart does not work as well)
Parity full sync on SSD takes about 3 days.

[tinybike]

Is there a workaround / solution for this bug? I get this error when attempting to fast sync geth on mainnet:

> CRIT [04-27|12:24:55] Failed to store header                   err="leveldb/table: corruption on data-block (pos=1616511): checksum mismatch, want=0xfa33e75a got=0xd192f7ea [file=039166.ldb]"

Edited to add -- I'm running the latest master version of geth on Xubuntu 16.04:

Geth
Version: 1.8.7-unstable
Git Commit: 1da33028ce88c4365d99471977098f4911fd38fa
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.9.4
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.9

[winsvega]

no. either light mode or full resync.
ask go team, maybe they implemented bd repair method of some kind

[tinybike]

I did a full resync, and everything worked this time... very mysterious.

[gnorbsl]

I have the same issue, deleting the corrupted file only works for a bit before the next file is corrupted. This issue is now two years old and it doesn't seem like this will be fixed anytime soon.
CRIT [12-22|07:45:51.610] Failed to store last header's hash err="leveldb/table: corruption on data-block (pos=1188369): checksum mismatch, want=0x2a050638 got=0x91a88b22 [file=135617.ldb]"

Why can't the node just delete the file and try a resync from there on instead of crashing?

[strusty]

this problem is definitely still present.

[industral]

get the same with 1.8.27 stable and 1.9 unstable, e.g.

CRIT [05-21|21:09:21.895] Failed to store hash to number mapping err="leveldb/table: corruption on data-block (pos=91219): checksum mismatch, want=0x52086fd8 got=0x17a37886 [file=490521.ldb]"

[strusty]

the real bug here is the error message saying there is a bug. this is a subtle hardware problem and can be the motherboard, CPU, memory or SSD drive. do try on different hardware, worked for me.

…

On Wed, May 22, 2019, 4:30 AM Alex Ivasyuv ***@***.***> wrote: get the same with 1.8.27 stable and 1.9 unstable, e.g. CRIT [05-21|21:09:21.895] Failed to store hash to number mapping err="leveldb/table: corruption on data-block (pos=91219): checksum mismatch, want=0x52086fd8 got=0x17a37886 [file=490521.ldb]" — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2568?email_source=notifications&email_token=AAFJBNDPMNSBDN7NGZ43VMDPWR5IVA5CNFSM4CD2BUO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV5NMMY#issuecomment-494589491>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFJBNCTLWCLFW6YPGX4LIDPWR5IVANCNFSM4CD2BUOQ> .

[industral]

didn't point, that it's mainnet. running on official docker. current size of chain ~ 130GB synced ~88%.
And when error occurred, docker process exit :(

[vijay055]

I got this error today after downloading blockheaders for 2 weeks , I have 1 T SSD, occupied nearly 400GB now. :-(
Failed to write header into disk err="leveldb/table: corruption on data-block (pos=384824320): checksum mismatch, want=0x0 got=0xf0423001 [file=592568.ldb]"

ran geth prior to latest Geth/v1.9.24-stable-cc05b050/windows-amd64/go1.15.5 got the error
then upgrade to latest , still error

any help pls

[cmtatro]

Not sure if this helps any one, You need to delete the data at the cache dir. Somewhere in the sync the program broke, or had a partial download. Clearing your data dir which you can find with geth dumpconfig. You can find it in the AppData/local on windows. This will cause a resync, I did not trying clearing the invalid block the block that says [file=nameoffile.ldb] and everything after it, but in theory doing that should work.

[holiman]

You need to delete the data at the cache dir. Somewhere in the sync the program broke, or had a partial download. Clearing your data dir which you can find with geth dumpconfig. You can find it in the AppData/local on windows.

A simpler way is to do geth removedb (can be combined e.g. geth --goerli removedb). That will prompt you whether to remove the 'regular' db, the 'ancient' db and the 'light' db. You can then choose to let the 'ancient' remain, and the next time you sync, you don't actually have to download ~120Gb of block data, only the state.

[qizhou]

It is likely that the memory is corrupted if the disk SMART reports no corruption. Here are my check steps (on Linux):

1. Check disk corruption event using SMART report
   $ sudo smartctl /dev/nvme0n1 -a
   ...
   SMART/Health Information (NVMe Log 0x02)
   Critical Warning: 0x00
   Temperature: 40 Celsius
   Available Spare: 100%
   Available Spare Threshold: 10%
   Percentage Used: 14%
   Data Units Read: 261,813,125 [134 TB]
   Data Units Written: 434,890,103 [222 TB]
   Host Read Commands: 6,800,895,678
   Host Write Commands: 1,705,371,809
   Controller Busy Time: 21,899
   Power Cycles: 14
   Power On Hours: 3,642
   Unsafe Shutdowns: 4
   Media and Data Integrity Errors: 0
   ...

(Different disk may report different formats, and mine is 970 evo pro). The report tells that no integrity errors are found, so it is unlikely the corruption issue from disk.

2. Replace your memory or lower your memory clock. I recently upgraded my PC from 32G (4x8G) to 64G (4x16G), and found the frequent data corruption issue (constantly reproducible via geth import ....). After changing the memory back, no corruption is reported by "get import".

Further, if you want more reliable memory, you could always choose ECC ones.