Treat Hermes 'empty wallet' error as non-recoverable

[adizere]

Context

This was a debugging session with a relayer operator.
We logged here our findings from running the Hermes dev feature from PR #1705. The problems documented below are, however, not related to this PR. The symptom the operator was reporting is that Hermes is not relaying anything. This was a red herring, however.

Takeaway

The important problem signaled in this issue is that Hermes misinterpret SDK error 13 (empty operator wallet).

Log remarks

Estimated gas higher than max gas

This is a misconfiguration.

• 1080 Jan 20 17:11:51 Imperator-Relayer hermes[1854166]: 2022-01-20T17:11:51.399192Z DEBUG ThreadId(49) send_tx_check{id=EYIMu5yhL8}:send_tx{id=panacea-3}: send_tx: estimated gas is higher than max gas id=panacea-3 estimated=1074310 max=700000

• this error appears more than once, eg

• 3562 Jan 20 17:49:29 Imperator-Relayer hermes[1854166]: 2022-01-20T17:49:29.580983Z DEBUG ThreadId(49) send_tx_check{id=JzxT5jQzws}:send_tx{id=panacea-3}: send_tx: estimated gas is higher than max gas id=panacea-3 estimated=1045902 max=700000
  3563 Jan 20 17:49:29 Imperator-Relayer hermes[1854166]: 2022-01-20T17:49:29.581497Z ERROR ThreadId(79) packet_cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-82 dst_chain=panacea-3}: will retry: schedule execution encountered error: failed with underlying error: pana cea-3 gas estimate 1045902 from simulated Tx exceeds the maximum configured 700000 retry_index=1

Potential fix: The relayer operator should set max_msg_num = 25 for Panacea chain to a smaller value, eg 10. Alternatively: increase the max_gas for that chain to above 1 Million.

Empty wallet on hub-4

This is a Hermes bug, but also a misconfiguration (i.e., the wallet was empty)

• 1131 Jan 20 17:11:56 Imperator-Relayer hermes[1854166]: 2022-01-20T17:11:56.794857Z ERROR ThreadId(42) send_tx_check{id=LbwPkAfBIu}:send_tx{id=cosmoshub-4}: broadcast_tx_sync: Response { code: Err(13), data: Data([]), log: Log("insufficient fees; got: 461uatom required: 11512uatom : insufficient fee"), hash: transaction::Hash(C74CC256F26AA388B54742360A534E10CB8E553D766D226F31449DE553DAA5EA) }: diagnostic: unknown SDK error: 13
  1132 Jan 20 17:11:56 Imperator-Relayer hermes[1854166]: 2022-01-20T17:11:56.795017Z INFO ThreadId(59) packet_cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:relay{odata=LbwPkAfBIu ->Destination @1-2876533; len=1}: [Async~>cosmoshub-4] respon se(s): 1; Err(13):C74CC256F26AA388B54742360A534E10CB8E553D766D226F31449DE553DAA5EA
  1133 Jan 20 17:11:56 Imperator-Relayer hermes[1854166]: 2022-01-20T17:11:56.795099Z INFO ThreadId(59) packet_cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-0 dst_chain=cosmoshub-4}:relay{odata=LbwPkAfBIu ->Destination @1-2876533; len=1}: success

• The problem here is that Hermes does not catch the "SDK error: 13" and instead interprets this as a "success".

Fix: At the very least, we should improve logs and treat "SDK error: 13" as an non-recoverable error.
Fix: Add more uatom to the wallet of chain cosmoshub-4

Version

#1705

Acceptance criteria

• improve logs and treat "SDK error: 13" as an non-recoverable error.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned