Try to recover a rocksdb database corrupted from out of disk

[BenTuras]

[ #1134963 a87d…cba2 ]---[ 6 blk/s | 17 tx/s | 1000624 gas/s //··· 0/9 peers, #1163825, 0+28778 queued ···// mem: 0 bytes db, 305 KiB chain, 23 MiB queue, 933 KiB sync ]
thread 'IO Worker #0' panicked at 'State DB commit failed.: SimpleString("IO error: /Users/benverloop/.parity/906a34e69aec8c0d/v5.3-sec-archive/state/083169.log: No space left on device")', ../src/libcore/result.rs:709
2016-03-17 22:43:40 Stage 3 block verification failed for #1135048 (3f39…c272)
Error: Block(UnknownParent(c3b2cd8f13dd3d14da48fb75226942d7b0248eec09da357cfcfccae3154c4cd4))
2016-03-17 22:43:41 Stage 3 block verification failed for #1135176 (b7ab…247c)
Error: Block(UnknownParent(c627fdc2d59d37104fcfcbdf12310a179239aea8ffbf80926b81e621458f3452))
2016-03-17 22:43:41 Error writing node table file: Error { repr: Os { code: 28, message: "No space left on device" } }

OK, disk full. I cleaned up and restarted parity -j, but no luck:

$ parity -j
2016-03-17 22:46:02 Error parsing node table file: SyntaxError("EOF While parsing value", 1, 1)
2016-03-17 22:46:03 Could not add node: AddressResolve(Some(Error { repr: Custom(Custom { kind: Other, error: StringError("failed to lookup address information: nodename nor servname provided, or not known") }) }))
2016-03-17 22:46:03 Starting Parity/v1.0.0-beta-570d2c2-20160315/x86_64-macos-/rustc1.7.0
2016-03-17 22:46:03 Configured for Frontier/Homestead using Ethash engine
2016-03-17 22:46:03 Public node URL: enode://2803d587f21360eb406f139761ebc307884e78deca572af1688b946f4ed6fcc41401b1986bc3c5a03eb7828c1dd5c14a74ab635549e735fcd892f3e553452153@0.0.0.0:30303
thread 'IO Worker #1' panicked at 'Error binding UDP socket: Error { repr: Os { code: 48, message: "Address already in use" } }', ../src/libcore/result.rs:709
thread '' panicked at 'called Result::unwrap() on an Err value: "PoisonError { inner: .. }"', ../src/libcore/result.rs:688
thread '

' panicked at 'Error opening state db: Corruption: checksum mismatch', util/src/journaldb/archivedb.rs:53
thread '' panicked at 'called Result::unwrap() on an Err value: NotifyError::Closed(..)', ../src/libcore/result.rs:688
stack backtrace:
1: 0x103559b98 - sys::backtrace::tracing::imp::write::h891a4a9fcda67dccF4t
2: 0x10355bdef - panicking::default_handler::_::closure.41940
3: 0x10355b86f - panicking::default_handler::hde0cb232bc545756U3x
4: 0x103542174 - sys_common::unwind::begin_unwind_inner::haeb4c6adcf4eff3cF7s
5: 0x10354261e - sys_common::unwind::begin_unwind_fmt::hab9b7efabf14110aL6s
6: 0x103559197 - rust_begin_unwind
7: 0x10357f380 - panicking::panic_fmt::h9f005ee999496200zYL
8: 0x1031920d6 - util..io..service..IoService<util..network..host..NetworkIoMessage<service..SyncMessage>>::drop.32352::h3c862c332df42e92
9: 0x103185fb7 - service::ClientService::start::h730bc13b4bc2d20eVne
10: 0x1030c0e79 - Configuration::execute_client::hc5d64e574a586d19G4a
11: 0x1030ca7b8 - main::h723a5ed6478c069e1ab
12: 0x10355b4a2 - sys_common::unwind::try::try_fn::h5317933650159713495
13: 0x103559138 - __rust_try
14: 0x10355b345 - rt::lang_start::hcf1bb0740bf735368Vx
thread panicked while panicking. aborting.
Illegal instruction: 4

I did notice the Address already in use error message, so I'll restart my macbook and report back

[gavofyork]

rocksdb error. we'll look into potential options.

[BenTuras]

Correct me if I am wrong, but there doesn't seem to be an utility to fix the checksum mismatch that I have. My only(?) option is to start all over again?

[guanqun]

@BenTuras Hi, sorry to hijack this a little bit. How much disk space does parity swallow in the end? I'm asking this because I also have a near-full disk and I don't want to start it over again...

[MagaTailor]

I finished syncing recently and the two directories created by parity amount to something like 4.2 GB @ 1.3 million blocks. (--pruning fast)

Besides, log files can grow to very large sizes.

[gavofyork]

merged.

[ricardobeat]

@gavofyork what was merged here?