Error: Error(Block(InvalidReceiptsRoot(Mismatch

[stone212]

Before filing a new issue, please provide the following information.

• Parity Ethereum version: 2.3.9 downgraded from 2.5.5
• Operating system: Linux
• Installation: binary
• Fully synchronized: no (that is the problem)
• Network: Private Blockchain
• Restarted: yes

I had a blockchain that worked well. I upgraded to Parity 2.4.x and there were many problems. Reference: #10617

I have a new miner running 2.3.9 and one by one I downgraded nodes and each node synced well. But one node did not. This is the error:

2019-09-29 07:28:43 UTC Stage 5 block verification failed for #11942 (0xf7aa…553f)
Error: Error(Block(InvalidReceiptsRoot(Mismatch { expected: 0x45bef3c8f45af0174311babea84ed659c7380a7249e2fe1ac5d7749f2918a383, found: 0xedfcd71c38d5ae9a55bb70d5f92220bc319036b760b0b6b16c841ef6e4d019ce })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
2019-09-29 07:28:43 UTC 
Bad block detected: Error(Block(InvalidReceiptsRoot(Mismatch { expected: 0x45bef3c8f45af0174311babea84ed659c7380a7249e2fe1ac5d7749f2918a383, found: 0xedfcd71c38d5ae9a55bb70d5f92220bc319036b760b0b6b16c841ef6e4d019ce })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
RLP: f90283f90212a024a29d4ef3eb552d8724e0f6e0237c767eaafa1341571028147abf9c49ab57dda01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347944872d82a158911060589c4aaef6a791609131273a08473f354b2a89f8fc1729dccd6bea7b60428124afc56c04727c9307e31affb06a0b97539c5810193f02c908cb01683be1b65df3c1f0ae69ac1edda8ca5ef03f164a045bef3c8f45af0174311babea84ed659c7380a7249e2fe1ac5d7749f2918a383b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000842b32d64f822ea68347b784825208845b43120896d583010a088650617269747986312e32372e30826c69a050a48a17d371146ed450929c307fb76aa97a30a1404a1280cbba7677a931b77f88a97eddca84227c55f86bf8693b808252089400e731b291da0096c1e8df75d385b9bd88dda8e7893635c9adc5dea000008081baa0a3232bba8a5233109c6120bc917d22b6cfb50fdd122b872e58343fcb0b32845ca002d3997c632830f7c8fb6a632fae47e93bd7e7230daada05c69472cb6abb2648c0
Header: Header { parent_hash: 0x24a29d4ef3eb552d8724e0f6e0237c767eaafa1341571028147abf9c49ab57dd, timestamp: 1531122184, number: 11942, author: 0x4872d82a158911060589c4aaef6a791609131273, transactions_root: 0xb97539c5810193f02c908cb01683be1b65df3c1f0ae69ac1edda8ca5ef03f164, uncles_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, extra_data: [213, 131, 1, 10, 8, 134, 80, 97, 114, 105, 116, 121, 134, 49, 46, 50, 55, 46, 48, 130, 108, 105], state_root: 0x8473f354b2a89f8fc1729dccd6bea7b60428124afc56c04727c9307e31affb06, receipts_root: 0x45bef3c8f45af0174311babea84ed659c7380a7249e2fe1ac5d7749f2918a383, log_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, gas_used: 21000, gas_limit: 4700036, difficulty: 724751951, seal: [[160, 80, 164, 138, 23, 211, 113, 20, 110, 212, 80, 146, 156, 48, 127, 183, 106, 169, 122, 48, 161, 64, 74, 18, 128, 203, 186, 118, 119, 169, 49, 183, 127], [136, 169, 126, 221, 202, 132, 34, 124, 85]], hash: Some(0xf7aa0db94051727d681f76cb0c011f58bb4ea4d42ff8eb64ddbb7b83c391553f) }
Uncles: 
Transactions:[Tx 0] UnverifiedTransaction { unsigned: Transaction { nonce: 59, gas_price: 0, gas: 21000, action: Call(0x00e731b291da0096c1e8df75d385b9bd88dda8e7), value: 1000000000000000000000, data: [] }, v: 186, r: 73789135770549567992443388361393272777854418241860117516794680946040639751260, s: 1278489748501668162386229323790218221985152871276007390673188146118752151112, hash: 0x52589b64d6c65645868f8e0eec4a824a0435ebc2ed49b30701357afff946ab64 }

Two important notes:

1. Block #11942 is long before the upgrade

2. This node was synced to a much higher number but was stuck at that number when I downgraded. So I deleted the chains and cache directory. Now it gets stuck at #11942.

Can someone help me understand the meaning of this error? I do not expect an easy answer but I will start de-bugging. From #2794: chain spec built-in accounts have funding.

Similar but not the same: #10910

Edits:

Here is an important question: what is this error based on? I deleted chains/ and cache/ so what record is on the node that is conflicting with the information it gets from peers?

Does this matter?

2019-09-29 23:45:53 UTC State DB configuration: archive +Fat +Trace
2019-09-29 23:45:53 UTC Operating mode: active
2019-09-29 23:45:53 UTC Warning: Warp Sync is disabled because Fat DB is turned on.

The node is peering with other nodes. When I remove it, peer number drops on others by one, and it comes back when I re-connect.

[stone212]

Update. When I add --fat-db=off --tracing=off --pruning=fast to parity command this node syncs again.

[stone212]

@joshua-mir Do you think this error could be fixed by this: #5850? Basically "eip98Transition": "0x7fffffffffffffff" should be in the root params section

I did not have it because we use EIP-658 but now I think maybe the release notes were saying that eipTransition98 is only not-necessary if you are using EIP-658 and you have PoA chain? It is not clear:

#9955

[stone212]

Update: every node that has this problem has these parameters in the config.toml file:

[footprint]
 tracing = "on"
 fat_db = "on"
 pruning = "archive"

[stone212]

Update: every block that is "stuck" is a block that comes one before a block with a transaction.

[stone212]

@joshua-mir So I think we can say it can not be that I must add eip98Transition to the chain spec based on #9955 (comment)

So this is looking like a bug again, not a mis-configuration. Please see updates above with more information than original Issue comment.

[stone212]

@joshua-mir I have an update on this. When I delete chains/ and re-sync with Parity 2.0.9, I do not see this error. I do see this one: #11130

Does any of this information help you to know how we can start to de-bug this Issue?

[stone212]

Solved by #11133