Layered Transaction Pool

[fab-10]

PR description

This implements a new transaction pool (txpool for brevity), with the main goal to better manage nonce gaps, i.e. the possibility that the list of transactions that we see for a sender could not be in order neither contiguous, that could happen just of the way there are broadcasted on the p2p network or intentionally to try to spam the txpool with non executable transactions (transactions that could not be included in a future block), so the goal is to try to keep in the pool transactions that could be selected for a future block proposal, and a the same time, without penalizing legitimate unordered transactions, that are only temporary non executable.

It is disabled by default, to enable use the option Xlayered-tx-pool=true

The main idea is to organize the txpool in an arbitrary number of layers, where each layer has specific rules and constraints that determine if a transaction belong or not to that layer and also the way transactions move across layers.

Some design choices that apply to all layers are that a transaction can only be in one layer at any time, and that layers are chained by priority, so the first layer has the transactions that are candidate for a block proposal, and the last layer basically is where transactions are dropped. Layers are meant to be added and removed in case of specific future needs.
When adding a new transaction, it is first tried on the first layer, if it is not accepted then the next one is tried and so on.
Layers could be limited by transaction number of by space, and when a layer if full, it overflow to the next one and so on, instead when some space is freed, usually when transactions are removed since confirmed in a block, transactions from the next layer are promoted until there is space.

The current implementation is based on 3 layers, plus the last one that just drop every transaction when the previous layers are full. The 3 layers are, in order:

1. Prioritized
2. Ready
3. Sparse

Prioritized:

This is where candidate transactions are selected for creating a new block.
Transaction are ordered by the effective priority fee, and it is limited by size, 2000 by default, to reduce the overhead of the sorting and because that number is enough to fill any block, at the current gas limit.
Does not allow nonce gaps, and the first transaction for each sender must be the next one for that sender.
Eviction is done removing the transaction with the higher nonce for the sender of the less valuable transaction, to avoid creating nonce gaps, evicted transactions go into the next layer Ready.

Ready:

Similar to the Prioritized, it does not allow nonce gaps, and the first transaction for each sender must be the next one for that sender, but it is limited by space instead of count, thus allowing many more transactions, think about this layer like a buffer for the Prioritized.
Since it is meant to keep ten to hundred of thousand of transactions, it does not have a full ordering, like the previous, but only the first transaction for each sender is ordered using a stable value that is the max fee per gas.
Eviction is the same as the Prioritized, and evicted transaction go into the next layer Sparse.

Sparse:

This is the first layer where nonce gaps are allowed and where the first transaction for a sender could not be the next expected one for that sender.
The main purpose of this layer is to act as a purgatory for temporary unordered and/or non contiguous transactions, so that they could become ready asap the missing transactions arrive, or they are eventually evicted.
It also keep the less valuable ready transactions, that are evicted from the previous layer.
It is limited by space, and eviction select the oldest transaction first, that is sent to the End Layer that just drop it.
When promoting to the prev layer Ready, only transactions that will not create nonce gaps are selected, for that we need to keep track of the nonce distance for each sender. So we can say that is ordered by nonce distance for promotion.

Changes in Besu options

When layered txpool is enabled some existing options are ignored and new options have been added, basically due to the fact that there is no more a limit in the number of transactions that the pool can contain, but the limit is the estimated amount of memory used.

Ignored options

• --tx-pool-max-size
• --tx-pool-limit-by-account-percentage

New options

Note: these options and their default values are subject to change while in the experimental phase.

• --Xlayered-tx-pool-layer-max-capacity the max amount of memory (in bytes) that a limited by space layer can use. Applies to Ready and Sparse layers. Default 50_000_000 bytes (50MB).
• --Xlayered-tx-pool-max-prioritized the maximum number of transaction that the Prioritized layer can contain. This Layer is limited by size to reduce the work needed for sorting. Default is 2000, and has been chosen since it is more than enough to fill any block at the current gas limit.
• --Xlayered-tx-pool-max-future-by-sender define the max difference between the incoming transaction nonce and the current sender nonce, for a transaction to be accepted in the txpool. This means that too far in the future transactions are ignored. Default is 200.

I have run it on mainnet and compared it against the current implementation, using mitmproxy plus this custom addon to simulate a block production every other FcU, and collected the result in this document Layered Transaction Pool metrics.ods, the results are good, and on average the Layered txpool produces with 12% more transactions in less time.

A bunch of new metrics have been added, that could be used to graph totals and details about each layer, in the official Besu Full dashboard

[fab-10]

I am going to merge this today since there are no more comments in the last weeks, the running instances are doing fine and the results are good. There are some improvements that can be done, but to avoid making this PR even bigger, I will create small PRs for them.

[hanniabu]

Are there any benchmarks for difference in fee revenue?

[fab-10]

@hanniabu you can check this spreadsheet for a comparison https://github.com/hyperledger/besu/files/11137473/Layered.Transaction.Pool.metrics.ods

[RogerHKW]

1. What is the effect of --tx-pool-limit-by-account-percentage when the layered pool is running? Does it count the transactions from one account over the whole pool or does it apply the percentage to each layer independently ?

2. What is the size of the second and third layers by default ? Your text talks about a hundred of thousands transactions in the second layer but your screenshot shows only 14k transactions in each layer with 2 millions transactions removed.

3. I would like to reach one hundred of thousands as advertised. Is there an option allowing me to customise their size ?

4. When an executable transaction with fees higher than the last transaction of the first layer reaches the client, is it inserted in the first layer immediately (thus shifting the rest of the layer and pushing the last transaction to the second layer) ?

5. If so, it has not been confronted to the sort by fee/gas values of the second layer, making it possibly suboptimal. As I expect it to happen quite often, I'm afraid that the first layer is in general very far from optimality. Do I misunderstand something?

Thanks 🙂

[fab-10]

Hi @RogerHKW, and thanks for checking out the layered txpool, feedback is important, and I found that the description was lacking documentation of changed options, so I added the Changes in Besu options section.
Moreover this is the first release of the new txpool, and there are more ideas for optimization that could be worth implementing if confirmed by the data collected.

What is the effect of --tx-pool-limit-by-account-percentage when the layered pool is running? Does it count the transactions from one account over the whole pool or does it apply the percentage to each layer independently ?

This option has no effect (there is warning in the log if you use it with the layered txpool) since there is no more a fixed max number of transactions, but this number constantly changes, so instead of using a percentage a fixed number is used and it can be specified by --Xlayered-tx-pool-max-future-by-sender options, see the description for more.

What is the size of the second and third layers by default ? Your text talks about a hundred of thousands transactions in the second layer but your screenshot shows only 14k transactions in each layer with 2 millions transactions removed.
I would like to reach one hundred of thousands as advertised. Is there an option allowing me to customise their size ?

By default the max memory these 2 layers can use is 50MB (now I have documented how to change this) and I did also tests with 100GB, and updated the screenshot with the metrics from that test. I have not found issues increasing that value to 100GB, but haven't tried with more, so if you want to push the limit, please report any issue, and also check the memory related metrics to see if they to be tuned.
In the next few days I will publish an updated Grafana dashboard with these new metrics included.

When an executable transaction with fees higher than the last transaction of the first layer reaches the client, is it inserted in the first layer immediately (thus shifting the rest of the layer and pushing the last transaction to the second layer) ?

correct

If so, it has not been confronted to the sort by fee/gas values of the second layer, making it possibly suboptimal. As I expect it to happen quite often, I'm afraid that the first layer is in general very far from optimality. Do I misunderstand something?

The way demoting and promotions between the Prioritized and the Ready layers work, should prevent this, because always the least priority tx(1) is moved to Ready, so ideally Ready only has txs that are lower in value that Prioritized(2), and as soon as there is space in Prioritized (usually when a new block arrives and confirmed txs are removed), most valuable Ready txs are promoted(2).
I hope this answer your doubt, otherwise if you still see there is flaw in my reasoning, please let me know, so I can review if I missed something in the approach.

(1) Specifically the last tx for the sender of the least priority tx, to avoid nonce gaps
(2) This is approximated to avoid a full ordering of all the txs in the large ready layers, and only the first tx for each sender is sorted, and the sort is static, so there could be cases where a following tx is better than some in the Prioritized layer, but it also means that the sender is doing strange things with his transactions pricing

[RogerHKW]

This is a great answer and big help, thanks!
(sorry for the previous version of this message. I'm discovering the functioning of Ethereum)

If I understand it right, the transactions in the first layer are sorted by tip per gas unit and will be taken in descending order when filling a block ? The transactions in the second layer are sorted by max fee per gas unit?

[fab-10]

This is a great answer and big help, thanks! (sorry for the previous version of this message. I'm discovering the functioning of Ethereum)

No problem, glad you are studying this, the feedback is also useful to make the code and the doc easier to understand.

If I understand it right, the transactions in the first layer are sorted by tip per gas unit and will be taken in descending order when filling a block ? The transactions in the second layer are sorted by max fee per gas unit?

Correct, the goal is to produce a block with the highest reward possible for the staker, in general this is a complex optimization problem, but in our case we take some shortcuts, since the blocks are seldomly full and so in the normal case we just need to put all the pending txs that are executable in that moment in the block, so the focus is more on which txs we keep in the pool for the future that could make the difference.