Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gaia-9002 Crash log #3019

Closed
1 of 4 tasks
hendrikhofstadt opened this issue Dec 6, 2018 · 6 comments
Closed
1 of 4 tasks

Gaia-9002 Crash log #3019

hendrikhofstadt opened this issue Dec 6, 2018 · 6 comments

Comments

@hendrikhofstadt
Copy link
Contributor

hendrikhofstadt commented Dec 6, 2018 •
edited
Loading 

invariant broken: loose token invariance:
	pool.LooseTokens: 4409145988819987660.4767998960
	sum of account tokens: 4409145988819987660.4768000000" stack="goroutine 1201 [running]:
runtime/debug.Stack(0xc00a433ac0, 0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000497c00, 0x1217900)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:576 +0x57
panic(0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariants(0xc000af1000)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:30 +0x453
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000af1000, 0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:218 +0x241
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:165 +0xc8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00018a700, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:794 +0x105
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0001b4cc0, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000978d40, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1316900, 0xc00086a500, 0x131ba80, 0xc000978d40, 0xc002e3cd20, 0xc00f0ee570, 0x1320540, 0xc0001c46b0, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:265 +0x701
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000112700, 0x7, 0x0, 0xc000b2ca20, 0x6, 0xc000b2ca30, 0x9, 0x17b08, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addProposalBlockPart(0xc000497c00, 0xc00016dd40, 0xc00417e0f0, 0x28, 0xc00c3b3000, 0x2d9d, 0x2d9d)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1460 +0x4d1
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000497c00, 0x13084a0, 0xc00016dd40, 0xc00417e0f0, 0x28)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:638 +0x7a0
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000497c00, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x670
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:300 +0x132
invariant broken: loose token invariance:
	pool.LooseTokens: 4409145988819987660.4767998960
	sum of account tokens: 4409145988819987660.4768000000" stack="goroutine 1201 [running]:
runtime/debug.Stack(0xc00a433ac0, 0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000497c00, 0x1217900)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:576 +0x57
panic(0xf38d60, 0xc003f91540)
	/usr/lib/golang/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariants(0xc000af1000)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:30 +0x453
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000af1000, 0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:218 +0x241
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x1315a80, 0xc00f0eeb10, 0xc00c567ac0, 0xd, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:165 +0xc8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00018a700, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:794 +0x105
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0001b4cc0, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000978d40, 0x17b09, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1316900, 0xc00086a500, 0x131ba80, 0xc000978d40, 0xc002e3cd20, 0xc00f0ee570, 0x1320540, 0xc0001c46b0, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:265 +0x701
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000112700, 0x7, 0x0, 0xc000b2ca20, 0x6, 0xc000b2ca30, 0x9, 0x17b08, 0x24ca7, 0xc009c7e560, ...)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000497c00, 0x17b09)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addProposalBlockPart(0xc000497c00, 0xc00016dd40, 0xc00417e0f0, 0x28, 0xc00c3b3000, 0x2d9d, 0x2d9d)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1460 +0x4d1
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000497c00, 0x13084a0, 0xc00016dd40, 0xc00417e0f0, 0x28)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:638 +0x7a0
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000497c00, 0x0)
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x670
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
	src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:300 +0x132

For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@cwgoes
Copy link
Contributor

cwgoes commented Dec 6, 2018

Runtime invariants are asserted at the end of each block, so this discrepancy was caused by a transaction (or begin block / end block operation) in the block we were attempting to execute.

@hendrikhofstadt
Copy link
Contributor Author 

{
        "Height": 97032,
        "Hash": "3FF40CAD512D1379EE70E2FCD274C7D14F531B5CD62BADDBF391AC8F1EB863C0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                }
            ],
            "gas_used": 8082,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper1shuqhpl273t96yg6nnqvyfeewj3ew3md6uy7uj"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "85f80b87eaf4565d111a9cc0c2273974a397476d",
                "sequence": 3107
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "3F04C45EAD604CFF00BD4746ED015A64F617E6F2AEB3D11E03553D1F2E2CE7D0",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_delegation_rewards_all"
                },
                {
                    "key": "delegator",
                    "value": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                }
            ],
            "gas_used": 7498,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "delegator_addr": "cosmos1vkggdmhuacs2306uasqw3au8a8lmcmqhqngtj4"
                },
                "type": "withdraw_delegation_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "659086eefcee20a8bf5cec00e8f787e9ffbc6c17",
                "sequence": 3678
            }
        ]
    },
    {
        "Height": 97032,
        "Hash": "4610B78843D9AD031F3A72745E372C8AA868CB2A408095D30A7C2D3676448EB5",
        "Fee": {
            "gas": 200000,
            "amount": [
                {
                    "denom": "",
                    "amount": "0"
                }
            ]
        },
        "Memo": "",
        "IsOK": true,
        "ValidFormat": true,
        "Result": {
            "log": "Msg 0: ",
            "tags": [
                {
                    "key": "action",
                    "value": "withdraw_validator_rewards_all"
                },
                {
                    "key": "source-validator",
                    "value": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                }
            ],
            "gas_used": 12682,
            "gas_wanted": 200000
        },
        "Messages": [
            {
                "data": {
                    "validator_addr": "cosmosvaloper10505nl7yftsme9jk2glhjhta7w0475uvl4k8ju"
                },
                "type": "withdraw_validator_rewards_all"
            }
        ],
        "Signatures": [
            {
                "address": "7d1f49ffc44ae1bc9656523f795d7df39f5f538c",
                "sequence": 12164
            }
        ]
    },

These are the transactions of the block

@rigelrozanski
Copy link
Contributor

rigelrozanski commented Dec 6, 2018 •
edited
Loading

I bet that #2958 resolved this issue... wish there was a way to simulate the blockchain easily to test if this is true of not

@rigelrozanski rigelrozanski mentioned this issue Dec 6, 2018
Closed
@rigelrozanski
Copy link
Contributor

relevant now merged in (but wasn't for this testnet) #2982 (comment)

@cwgoes
Copy link
Contributor

cwgoes commented Dec 7, 2018

relevant now merged in (but wasn't for this testnet) #2982 (comment)

Hmm, maybe - #2982 only changed the logic in state export. Maybe we're leaving rewards in a dist info before deleting it here, though, and that's causing the problem (which we would need to fix separately).

This was referenced Dec 7, 2018
Merged
Merged
jaekwon added a commit that referenced this issue Dec 8, 2018
@jaekwon
Fix updateValidatorDistInfoFromPool (#3046)
bc51fa9 
Fixes regression introduced by #2984.
Continuiation of #3033 , which didn't fix the simulation issues.
(candidate) Complete solution for #3019, 9002 halt bug.

From #2984, it isn't sufficient to take the fee pool rewards of a validator. Since we don't track delegator accums (as we do with validator accums), and because onValidatorModified >updateValidatorDistInfoFromPool is also being called upon delegation updates (or at least I believe this is the reason), it is necessary to also withdraw self delegation.

TODO: I don't think self-delegation should be required to be modified here... consider using a delegation hook to do the self-delegation withdraw part instead, e.g. splitting the updateValidatorDistInfoFromPool function into two. It might not result in cleaner code, however. Think hard.
@zmanian
Copy link
Member

zmanian commented Jan 5, 2019

Pretty sure this can be closed.

@zmanian zmanian closed this as completed Jan 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@zmanian @hendrikhofstadt @rigelrozanski @cwgoes