Statesync broken on v10.0.0

[mturkia]

System information

Osmosis version: v10.0.0
OS & Version: Linux (most likely all OSes)
Commit hash: 7ca72c5

Expected behaviour

Chain successfully synced to last statesync snapshot.

Actual behaviour

Statesync fails with following error:

12:34PM INF Applied snapshot chunk to ABCI app chunk=67 format=1 height=4737000 module=statesync total=68
12:34PM ERR State sync failed err="snapshot restoration failed: app version mismatch. Expected: 9, got: 0" module=statesync

After setting expected app version and starting the chain again, I get following error:

Error: fromProto: validatorSet proposer error: nil validator

Steps to reproduce the behaviour

• Configure statesync
• Run osmosisd start to sync
• Once all statesync snapshot chunk are applied and you get the first error message about app version mismatch, terminate osmosisd
• Set the expected app-version with tendermint set-app-version 9 --home wherever See statesync: app version is not correctly set cosmos/cosmos-sdk#10791 for reference

[PolkachuIntern]

We have the exactly same issue while state-syncing.

[p0mvn]

Thanks for the issue.

Before we try anything to resolve this, I think we need to merge this first: #1698 and test again. That PR adds some logic for wiring wasm snapshotter correctly. We haven't had the time to include or test it for v10.

This is likely to be unrelated to the main issue but I would like to note that the app version did not get increased to v10:

12:34PM ERR State sync failed err="snapshot restoration failed: app version mismatch. Expected: 9, got: 0" module=statesync

The expected should be 10. I suspect that it did not get increased because we did a fork instead of a regular upgrade

CC: @czarcas7ic once we add logic for e2e testing forks, we should test the app version to be increased by 1

[p0mvn]

Got state sync to work with #1698 and osmosis-labs/cosmos-sdk#265 (WIP)

I discovered that state sync snapshots do not share the param store. The app version was stored there. As a result, the app version would not be propagated to the state-synching node, resulting in the reported error.

I made 2 changes. First, moved the app version to be stored in the multistore instead of the param store. Second, I had to make a breaking change where state-sync snapshots now include the app version and share it. As a result, the snapshot format is now different.

Due to these 2 changes, the fix will have to go to v11 so that the state-machine and snapshot format are compatible across nodes. By then, I should finish setting up state sync in e2e to further confirm the correctness of the new change. So far I did a manual test between 2 full mainnet nodes where one had the other as persisted peer and pex off.

Another important note is that node operators should not use cosmos/cosmos-sdk#10791 hack anymore. Once we release the fix, it should be working out of the box. Using the hack will break state sync.

[PolkachuIntern]

Thanks for the update. Look forward to v11.

[ValarDragon]

Closing for now, should be fixed in upcoming version!