e2e tests fail with --verbose

[p-offtermatt]

Summary of Bug

When running the happy-path, tests run fine for me when running

go run ./tests/e2e/... --happy-path-only

but fail when running

go run ./tests/e2e/... --happy-path-only --verbose

The end of the log looks like this:

running default: step 2 == submitConsumerAdditionProposalAction 
running default: step 3 == assignConsumerPubKeyAction 
assignConsumerPubKey cmd: /opt/homebrew/bin/docker exec interchain-security-instance /bin/bash -c interchain-security-pd tx provider assign-consensus-key consu '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"Ui5Gf1+mtWUdH8u3xlmzdKID+F3PK0sfXZ73GZ6q6is="}' --from validatorcarol --chain-id provi --home /provi/validatorcarol --node tcp://7.7.7.6:26658 --gas 90000 --keyring-backend test -y -o json
running default: step 4 == assignConsumerPubKeyAction 
assignConsumerPubKey cmd: /opt/homebrew/bin/docker exec interchain-security-instance /bin/bash -c interchain-security-pd tx provider assign-consensus-key consu '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"Ui5Gf1+mtWUdH8u3xlmzdKID+F3PK0sfXZ73GZ6q6is="}' --from validatorcarol --chain-id provi --home /provi/validatorcarol --node tcp://7.7.7.6:26658 --gas 90000 --keyring-backend test -y -o json
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x105141f48]

goroutine 1 [running]:
main.TestRun.assignConsumerPubKey({{{0x105188ca0, 0x1d}, {0x1051871bd, 0x1c}, {0x1053c3c70, 0x1}, {0xc1290b3b0602cd98, 0x108a437, 0x105f69c60}}, 0x1400094bb60, ...}, ...)
        /Users/offtermatt/projects/interchain-security/tests/e2e/actions.go:1851 +0x448
main.(*TestRun).runStep(0x1400055ac60, {{0x10559afc0?, 0x1400023ef40?}, 0x140008864e0?}, 0x68?)
        /Users/offtermatt/projects/interchain-security/tests/e2e/main.go:171 +0x14c
main.(*TestRun).executeSteps(0x1400055ac60, {0x1400049c700, 0x32, 0x0?})
        /Users/offtermatt/projects/interchain-security/tests/e2e/main.go:204 +0x17c
main.(*TestRun).Run(0x1400055ac60, {0x1400049c700, 0x32, 0x4a}, {0x0?, 0x1?}, 0x98?, {0x0?, 0x105f69c60?})
        /Users/offtermatt/projects/interchain-security/tests/e2e/main.go:103 +0x94
main.main()
        /Users/offtermatt/projects/interchain-security/tests/e2e/main.go:55 +0x2d0
exit status 2

Version

7484dbc

Steps to Reproduce

go run ./tests/e2e/... --happy-path-only --verbose

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned
• Is a spike necessary to map out how the issue should be approached?

[MSalopek]

I checked the behaviour, it seems that there's an actual bug in behaviour of either the tests or the chain.

If you look at this test output, you can see that none of the traces actually produce an error, but there definitely should be an error produced. Maybe it has to do with the --broadcast-mode block being removed

=============== started default tests ===============
running default: step 1 == StartChainAction
Started chain provi
running default: step 2 == submitConsumerAdditionProposalAction

#### THIS SHOULD WORK FINE ####
running default: step 3 == assignConsumerPubKeyAction
assignConsumerPubKey cmd: /usr/local/bin/docker exec interchain-security-instance /bin/bash -c interchain-security-pd tx provider assign-consensus-key consu '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"Ui5Gf1+mtWUdH8u3xlmzdKID+F3PK0sfXZ73GZ6q6is="}' --from validatorcarol --chain-id provi --home /provi/validatorcarol --node tcp://7.7.7.6:26658 --gas 90000 --keyring-backend test -y -o json
assignConsumerPubKey OUT: {"height":"0","txhash":"88A4A6C9D21ABC46BBA1962CB6E8AC1ACA8208E9479147C317381DE313768F67","codespace":"","code":0,"data":"","raw_log":"[]","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
 <nil>

#### THIS SHOULD ERROR ####
running default: step 4 == assignConsumerPubKeyAction
assignConsumerPubKey cmd: /usr/local/bin/docker exec interchain-security-instance /bin/bash -c interchain-security-pd tx provider assign-consensus-key consu '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"Ui5Gf1+mtWUdH8u3xlmzdKID+F3PK0sfXZ73GZ6q6is="}' --from validatorcarol --chain-id provi --home /provi/validatorcarol --node tcp://7.7.7.6:26658 --gas 90000 --keyring-backend test -y -o json
assignConsumerPubKey OUT: {"height":"0","txhash":"C6FF059002432153756EDAA23FDD56726FE96D5486BA674ED20A021253DFF494","codespace":"","code":0,"data":"","raw_log":"[]","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
 <nil>

#### THIS SHOULD ERROR ####
running default: step 5 == assignConsumerPubKeyAction
assignConsumerPubKey cmd: /usr/local/bin/docker exec interchain-security-instance /bin/bash -c interchain-security-pd tx provider assign-consensus-key consu '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"Ui5Gf1+mtWUdH8u3xlmzdKID+F3PK0sfXZ73GZ6q6is="}' --from validatorbob --chain-id provi --home /provi/validatorbob --node tcp://7.7.7.5:26658 --gas 90000 --keyring-backend test -y -o json
assignConsumerPubKey OUT: {"height":"0","txhash":"7958D8D615AACECC5877CDA8448B62B57030F0EDD108E7A6192697F0FD39E8A7","codespace":"","code":0,"data":"","raw_log":"[]","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
 <nil>

[MSalopek]

@bermuell
I checked this in e2e and found this:

[root@5fecb02c0e12 /]# interchain-security-pd tx provider assign-consensus-key consu '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"Ui5Gf1+mtWUdH8u3xlmzdKID+F3PK0sfXZ73GZ6q6is="}' --from validatorcarol --chain-id provi --home /provi/validatorcarol --node tcp://7.7.7.6:26658 --keyring-backend test -y --gas auto
Error: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: a validator has assigned the consumer key already: consumer key is already in use by a validator [/interchain-security/x/ccv/provider/keeper/key_assignment.go:414] With gas wanted: '18446744073709551615' and gas used: '32315' : unknown request
...

In order to produce an error, you have to provide --gas auto otherwise the tx wil fail silently.

If you are refactoring, please consider also changing --gas 90000 to --gas auto in the command invocation and changing how the error checks happen.

Thanks!