pallet-mmr: handle forks without collisions in offchain storage

[acatangiu]

Description

pallet-mmr uses offchain db to store un-pruned MMR with full leafs contents.

Because MMR nodes are keyed/indexed in offchain db with their positional index, chain forks would end up overwriting same entry with fork-specific values.

This PR uses parent_hash of block that added MMR node(s), alongside the node(s) positional index as key in the offchain db, thus eliminating collisions.

Leaves are stored offchain keyed by (parent_hash, node_index) to make them fork-resistant.
frame_system::BlockHashCount blocks later, offchain worker task will "canonicalize" one leaf per block. At this point we are not worried about forks anymore (highly unlikely to have a fork in the chain that deep - 2400 blocks on polkadot, kusama, rococo, etc).
"Canonicalization" in this case means moving this leaf under a new key based only on the leaf's node_index.

All non-canon nodes are also pruned in offchain db.

The pallet API is unchanged, with the pallet transparently doing all the magic of finding the right block hashes to use as keys.

Related issues

Fixes paritytech/grandpa-bridge-gadget#336

[svyatonik]

Just a few comments, since it is WIP :)

[Lederstrumpf]

some simplifications to arithmetic - more review as it evolves.

[acatangiu]

@cheme I implemented pruning as well, ptal

[cheme]

LGTM. Maybe release note should specify the crate got specialized to only storing one value per blocks (or we can keep old storage implementation, but I am not sure if it is used or useful).

[Lederstrumpf]

lgtm.
as discussed out of band, this works for current leaf fields (

substrate/frame/beefy-mmr/src/lib.rs

Lines 157 to 162 in 6a946fc

	MmrLeaf {
	version: T::LeafVersion::get(),
	parent_number_and_hash: ParentNumberAndHash::<T>::leaf_data(),
	leaf_extra: T::BeefyDataProvider::extra_data(),
	beefy_next_authority_set: Pallet::<T>::beefy_next_authorities(),
	}

) since this is the same for forks as long as they share the same parent, i.e. are depth-1, which is handled by the ephemeral (parent_hash, block_number) keys introduced here.

if/when the leaf structure changes, we may want to move to something more robust than (parent_hash, block_number) for fork-aware indexing that takes into account the entire leaf's content, i.e. at least any fields that aren't invariant between forks of depth-1.

[acatangiu]

if/when the leaf structure changes, we may want to move to something more robust than (parent_hash, block_number) for fork-aware indexing that takes into account the entire leaf's content, i.e. at least any fields that aren't invariant between forks of depth-1.

Added documentation around LeafData for current requirements.

Opened #11799 for improving the pallet to relax said requirements 👍

[andresilva]

lgtm

[andresilva]

bot merge