Transactions are delayed in the network when a fork occurs

[lsaether]

While injecting transactions in recent versions of a Substrate-based chain I've noticed transactions get delayed in the tx-pool when a fork occurs. The strange thing about this delay is that it is actually pretty long -- 238 blocks on the last time occurred. By saying "delay" I mean that the transactions are no longer on the node I initially injected them to, but form what I can tell are floating around the network and not being included for a while.

Unfortunately I don't have any logs from when this happened and it may be hard to reproduce. If someone has any ideas for trying how I can best provide more information I would be happy to know.

[bkchr]

@tomusdrw any ideas?

[tomusdrw]

What version was it? Adding transactions from retracted blocks back to the pool was added some time already #3562.
I suspect that the issue is caused by the same thing as slow transaction pool culling. Receiving notification of imported block is significantly delayed - either we are doing something very heavy and need a lock (potentially transaction pool clean up might take long) or the task get's starved.

Would be good to collect logs with -l=txpool=debug with the latest version of substrate (we already do that for our nodes) - this will allow to figure out what's going on.