Chain Stability - Solve Canonicalization Problem

[GuruKathryn]

We are getting the below canonicalization issue and we are wondering:

1. Is it a problem or is it normal and can be ignored?
2. Has anyone found a method to resolve it?

Here's what we know:

• It consistently begins at block 2400, and occurs in every block thereafter
• It occurs on --dev, --pruning archive, --unsafe-pruning (for any number) but only on validator nodes (not on sync nodes - that part makes sense)
• It occurs on all node accounts, including Bob and other development node accounts
• Sometimes it is followed by the comment Offchain: could not prune: no entry in pruning map for block, but not always.
• It does not seem to affect latency, block production, etc
• We are running the Substrate kitchen sink master 09.26

Log Sample:

2022-10-03 10:21:57 ✨ Imported paritytech#2400 (0xbcf6…a1d1)
2022-10-03 10:21:57 Could not canonicalize elem at pos 0 using key [12, 109, 109, 114, 179, 231, 150, 242, 116, 82, 112, 40, 61, 105, 19, 211, 47, 138, 252, 83, 183, 123, 184, 153, 192, 116, 137, 163, 141, 85, 35, 94, 41, 140, 112, 6, 0, 0, 0, 0, 0, 0, 0, 0]
2022-10-03 10:21:57 Offchain: could not prune: no entry in pruning map for block 0
2022-10-03 10:21:57 ♻️ Reorg on paritytech#2400,0xbcf6…a1d1 to paritytech#2400,0xf59d…0abd, common ancestor paritytech#2399,0xe757…9679
2022-10-03 10:21:57 ✨ Imported paritytech#2400 (0xf59d…0abd)
2022-10-03 10:21:57 Could not canonicalize elem at pos 0 using key [12, 109, 109, 114, 179, 231, 150, 242, 116, 82, 112, 40, 61, 105, 19, 211, 47, 138, 252, 83, 183, 123, 184, 153, 192, 116, 137, 163, 141, 85, 35, 94, 41, 140, 112, 6, 0, 0, 0, 0, 0, 0, 0, 0]
2022-10-03 10:21:57 Offchain: could not prune: no entry in pruning map for block 0
2022-10-03 10:22:00 👶 New epoch 12 launching at block 0x716f…5741 (block slot 554935640 >= start slot 554935640).
2022-10-03 10:22:00 👶 Next epoch starts at slot 554935840
2022-10-03 10:22:00 ✨ Imported paritytech#2401 (0x716f…5741)
2022-10-03 10:22:00 Could not canonicalize elem at pos 1 using key [12, 109, 109, 114, 231, 162, 175, 129, 3, 105, 243, 83, 25, 239, 92, 131, 52, 160, 52, 125, 39, 24, 187, 185, 18, 0, 111, 70, 240, 16, 197, 70, 27, 241, 63, 19, 1, 0, 0, 0, 0, 0, 0, 0]
2022-10-03 10:22:00 Could not canonicalize elem at pos 2 using key [12, 109, 109, 114, 231, 162, 175, 129, 3, 105, 243, 83, 25, 239, 92, 131, 52, 160, 52, 125, 39, 24, 187, 185, 18, 0, 111, 70, 240, 16, 197, 70, 27, 241, 63, 19, 2, 0, 0, 0, 0, 0, 0, 0]
substrate-nodeconsensus

[GuruKathryn]

Answer courtesy of Adrian...

This is coming from pallet-mmr: opened paritytech#12531 to fix it.

Is it a problem or is it normal and can be ignored?
You can definitely ignore those error logs (my guess is you are not actually using pallet-mmr).

Has anyone found a method to resolve it?
Possible workarounds:

1. remove pallet-mmr from your runtime (if you can, and don't need it),
2. run node with extra --enable-offchain-indexing=true param,
3. filter out logs with target=runtime::mmr::offchain ("problem" is still there, but you won't see these logs anymore).

[GuruKathryn]

Problem solved successfully by running nodes with extra --enable-offchain-indexing=true parameter.