Genesis validator block 1 AppHash differs from other nodes

[michaelfig]

Describe the bug

To Reproduce

Steps to reproduce the behavior:

See https://github.com/Agoric/agoric-sdk/actions/workflows/deployment-test.yml
Its output ends with https://github.com/Agoric/agoric-sdk/runs/4361816825#step:7:2271

TASK [cosmos-validators : Wait for transfer of ubld to arrive at validator1] ***
FAILED - RETRYING: Wait for transfer of ubld to arrive at validator1 (60 retries left).
[...]
FAILED - RETRYING: Wait for transfer of ubld to arrive at validator1 (3 retries left).
FAILED - RETRYING: Wait for transfer of ubld to arrive at validator1 (2 retries left).
FAILED - RETRYING: Wait for transfer of ubld to arrive at validator1 (1 retries left).
fatal: [validator1]: FAILED! => {"attempts": 60, "changed": true, "cmd": "agd query bank balances agoric1hjhj05g3lvcaak487nvzx6jqluqn3tdqn25mz9", "delta": "0:00:00.075124", "end": "2021-11-30 02:22:12.829979", "msg": "", "rc": 0, "start": "2021-11-30 02:22:12.754855", "stderr": "", "stderr_lines": [], "stdout": "balances: []\npagination:\n  next_key: null\n  total: \"0\"", "stdout_lines": ["balances: []", "pagination:", "  next_key: null", "  total: \"0\""]}

PLAY RECAP *********************************************************************
validator0                 : ok=6    changed=3    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   
validator1                 : ok=7    changed=4    unreachable=0    failed=1    skipped=0    rescued=0    ignored=1   

error running ag-setup-cosmos: (Error#1)
Error#1: Unexpected exit: 2

Expected behavior

Chain should bootstrap successfully.

Platform Environment

• what OS are you using? Ubuntu Trusty what version of Node.js? Node.js 14.x
• is there anything special/unusual about your platform? GitHub Actions CI
• what version of the Agoric-SDK are you using? master

Additional context

@warner and @mhofman helped dig through the slog file outputs, which showed that SwingSet did not diverge, just that the second node to join the network never got to block 2 (AppHash failure on block 1 commit).

Next steps:

• trace the Cosmos kvstore output for both nodes and seeing what the divergence is (see agd start --trace-store=file.kvtrace)
• publish the kvstore and slog files as GitHub artifacts for this test (see https://github.com/actions/upload-artifact)

[michaelfig]

Output from a failed test can be found at: https://github.com/Agoric/agoric-sdk/actions/runs/1522707618