State Sync is broken on mainnet

[faddat]

So basically the Block.Protocol.Version is set wrong in the block headers in osmosis-- I think. Here's the result:

(pending script completing a run)

[faddat]

230","port":2000}
8:11AM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
8:11AM INF Stopping queryMaj23Routine for peer module=consensus peer={"Data":{},"Logger":{}}
panic: Failed to process committed block (2284501:147C5444A238F247A79F08807FD33BE86D155E3385DB807165DACAFF06DD3C60): wrong Block.Header.Version. Expected {11 0}, got {11 1}

goroutine 6946 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0x140005d2000, 0x1)
	/Users/faddat/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:401 +0x10e4
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).SwitchToFastSync
	/Users/faddat/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:125 +0xe4

[dualsystems]

on emoney chain sometimes when sync via statesync, I get error:

9:54AM ERR WARNING: Attempt to allocate proposer rewards to unknown proposer . This should happen only if the proposer unbonded completely within a single block, which generally should not happen except in exceptional circumstances (or fuzz testing). We recommend you investigate immediately. module=x/distribution
9:54AM INF Endblock: Block 287401 was proposed by 51C818CB49A0B0C8FE8CDD7F217ABFF15AAE9988 module=emz
9:54AM INF executed block height=287401 module=state num_invalid_txs=0 num_valid_txs=0
9:54AM INF commit synced commit=436F6D6D697449447B5B31333020333220353320313635203134372031333420393320353820323620363620353520313420323037203131312032303320323438203230322031333520323620373420323020342032303220313337203135352031393620323320313833203136302031323620313038203134325D3A34363241397D
9:54AM INF committed state app_hash=822035A593865D3A1A42370ECF6FCBF8CA871A4A1404CA899BC417B7A07E6C8E height=287401 module=state num_txs=0
9:54AM INF indexed block height=287401 module=txindex
panic: Failed to process committed block (287402:1FA2B4F5E50AAE13B01760135A6D69957DB83BF6D7B06EF71E25712C2C2F6910): wrong Block.Header.AppHash.  Expected 822035A593865D3A1A42370ECF6FCBF8CA871A4A1404CA899BC417B7A07E6C8E, got 3CFB2E821E8F9FECAB9A793935BCCE45DAAE88365928BC90BFE4ED7AF71B424C

when change snapshot point height, sync is ok

[dualsystems]

tendermint/tendermint#6793

[dualsystems]

statesync restore snapshot without any errors, but when switch to fastsync process is break.

6:20PM INF Verified ABCI app appHash="���0*V�Ahfj\x06��?@'�_�\f���\x13��\x17'{��" height=2397500 module=statesync
6:20PM INF Snapshot restored format=1 hash="��bV_�����\rӒd�\x00�]�6�X��!\x0f\u009fZ\x03<�" height=2397500 module=statesync

6:20PM INF Starting BlockPool service impl=BlockPool module=blockchain

panic: Failed to process committed block (2397501:C85C412DEFBE083B3BAC5D3E6B7967366B300293E94C5A30FC7CD46B8B5C61E8): wrong Block.Header.Version. Expected {11 0}, got {11 1}

goroutine 199 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc0001f4c40, 0xc163591f01)
        github.com/tendermint/[email protected]/blockchain/v0/reactor.go:401 +0x15a5
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).SwitchToFastSync
        github.com/tendermint/[email protected]/blockchain/v0/reactor.go:125 +0xd5

[faddat]

@dualsystems

Yes:

3:02AM INF Dialing peer address={"id":"63aba59a7da5197c0fbcdc13e760d7561791dca8","ip":"162.55.132.230","port":2000} module=p2p
3:02AM INF Stopping gossipVotesRoutine for peer module=consensus peer={"Data":{},"Logger":{}}
3:02AM INF Stopping gossipDataRoutine for peer module=consensus peer={"Data":{},"Logger":{}}
panic: Failed to process committed block (2398801:D60C0B13FA626B3F8E81098403C6AD29C6678557677967C0AB4B73E1C24666B6): wrong Block.Header.Version. Expected {11 0}, got {11 1}

goroutine 16187 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0x140001a8380, 0x1)
	/Users/faddat/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:401 +0x10e4
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).SwitchToFastSync
	/Users/faddat/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:125 +0xe4

TM has a versions struct and one of them is the app version and the other is the block header version and -- sadly-- I haven't been able to figure out how to set the block header version manually and/or if that would resolve the problem or just explode further.

[dualsystems]

9:23PM INF Starting IndexerService service impl=IndexerService module=txindex
9:23PM INF Version info block=11 p2p=8 tendermint_version=0.34.14

If block version is 11, then in Expected {11 0}, got {11 1} 0 and 1 is app version.

Posible make exption for consensus module?

That is hard code constant of appVersion

osmosis/app/app.go

Line 387 in 4641e15

// Set all modules "old versions" to 1.

osmosis/x/pool-incentives/module.go

Line 178 in dedcfb8

func (AppModule) ConsensusVersion() uint64 { return 1 }

osmosis/x/claim/module.go

Line 163 in dedcfb8

func (AppModule) ConsensusVersion() uint64 { return 1 }

[dualsystems]

@faddat
Set app version via ABCI:
tendermint/tendermint#7461 (comment)
tendermint/tendermint#7463
tendermint/tendermint@b1e0d91

[daniel-farina]

Ref: #707

[ValarDragon]

Closing this for now, what needs to happen for fix is documented in #1805