Investigate ethers-rs middleware for "stuck" txs

[tkporter]

Context

The relayer submits a transaction with a gas price that is no longer high enough to execute, but instead of the relayer submitting on certain networks like polygon the tx is stuck in the mempool. To "unstuck" the relayer needs to submit a higher bid with x% more gas, but our relayer currently lacks that logic.

https://docs.rs/ethers/latest/ethers/middleware/struct.GasEscalatorMiddleware.html is probably what we want

Some things to note:

• See https://cloudlogging.app.goo.gl/jXpgMgTHsWZpUk1s9 for examples of this. Interestingly, this only seems to happen with polygon txs and still occurs pretty frequently.
• This was originally written with Polygon in mind that we fixed in Use Polygon gas oracle #2965. Addressing Relayer tx issues  #2924 is now the main focus of this. Seemingly the Polygon fix wasn't totally sufficient
• Be careful of our usage around fallback provider here (e.g. if the gas escalation only works at the "inner provider" level and not at a higher level, this may not actually be useful)
• Be careful that we construct new providers pretty frequently, e.g. when constructing metadata. I believe we use the same provider for all tx submission, but maybe we don't. I believe we'd need to use the same long-living provider if we wanted to use the gas escalator middleware

[yorhodes]

https://docs.rs/ethers/latest/ethers/middleware/struct.GasEscalatorMiddleware.html

[tkporter]

nice, we'll have to make sure the footgun / issues aren't a problem for us & also that this covers the case of immediately getting back a "transaction underpriced" error and not just if the tx doesn't get included in a block in time

[daniel-savu]

Be careful of our usage around fallback provider here (e.g. if the gas escalation only works at the "inner provider" level and not at a higher level, this may not actually be useful)

Since our fallback provider logic is a client that middleware wraps, the gas escalator can actually only be used for the entire fallback rather than inner ones. See here how the build step that wraps with providers happens after the fallback client is built - and same for the retrying provider, in the match arm below. We're not susceptible to the "memory leak" bug as long as we have one RPC still live - because as soon as one of them comes back online, resubmits start happening.

Be careful that we construct new providers pretty frequently, e.g. when constructing metadata. I believe we use the same provider for all tx submission, but maybe we don't. I believe we'd need to use the same long-living provider if we wanted to use the gas escalator middleware

Had a look, indeed we only transact via long-lived instances (mailbox and validator_announce) - these are always wrapped in signing providers, and we can also add the gas escalator middleware where this happens.

the case of immediately getting back a "transaction underpriced" error and not just if the tx doesn't get included in a block in time

If we get back an error, it just gets propagated. I believe this is less of an issue since the message will be retried if submission fails, but we could include this specific error message in the retrying provider logic here.

[tkporter]

We're not susceptible to the "memory leak" bug as long as we have one RPC still live - because as soon as one of them comes back online, resubmits start happening.

I think it's still possible to get an error from a provider even if they're technically "live", (e.g. if we're low on funds or the gas price is too low or something and can't successfully submit a transaction) and the task will error here https://github.com/hyperlane-xyz/ethers-rs/blob/hyperlane/ethers-middleware/src/gas_escalator/mod.rs#L223

But in any case I'm comfortable treating this as an unlikely edge case that we're aware of and we can revisit if we have issues