Packet takes 24sec to get relayed

[gamarin2]

Hermes caught a Sendevent, but took 24 sec to try and relay it. By the time it tried, the packet had already been relayed. Here are the relevant logs:

Aug 06 19:32:33.729 DEBUG [irishub-1:transfer/channel-12 -> cosmoshub-4] irishub-1 => SendPacketEv(SendPacket - h:1-11259223, seq:21, path:channel-12/transfer->channel-182/transfer, toh:0-0, tos:Timestamp(2021-08-06T19:37:03.175+00:00)))
Aug 06 19:32:37.504 DEBUG [cosmoshub-4 -> osmosis-1:07-tendermint-1] client was updated
Aug 06 19:32:57.407 DEBUG [irishub-1:transfer/channel-12 -> cosmoshub-4] SendPacket - h:1-11259223, seq:21, path:channel-12/transfer->channel-182/transfer, toh:0-0, tos:Timestamp(2021-08-06T19:37:03.175+00:00)) already handled

This is not an isolated event, happens very frequently (most packets are delayed actually)

Version: 0.6.2

[adizere]

Thank you for documenting this issue Gautier!

This is related to #1124, and the underlying cause is that Hermes has a blocking call waiting for each transaction to be confirmed before it can process a subsequent transaction for the same target chain. If rpc_timeout is configured to at least 24s in this case for the chain cosmoshub-4, then Hermes may block up to that time. We're actively looking into it and trying to redo the Hermes <> SDK interaction.