Besu Chain is stuck if too many smartcontract deployment requests in TX pool

[cdfly813]

Description

As an Besu Client, I set Besu blockperiod as 2 seconds,
I built a smartcontract, and use one single account to deploy this smartcontract 800 times. the deployment requests are come into 800 transactions.
then I submitted 800 transactions in transaction pool with nonce gap, for example, one account latest nonce is 5, and I set the 800 transaction's nonce get start with 10, then all the 800 transactions with nonce number is 10-810, and they will stay in transaction pool and wait the nonce gap is fixed.
s
so that after I fixed nonce from 5-9, then the whole TX pool seems be stuck, sometimes it will gave a "max selection timeout issue", and it can't generate the blocker anymore, when all the transactions was deleted after it's expired(13 hours), even the chain can generate the new block, when I submit several TX without nonce gap with same account address, it seems the TX was missed, I can't find it on chain or TX pool.

Acceptance Criteria

• The Besu can runs well after I fixed all the nonce gap issue in TX pool.

Steps to Reproduce (Bug)

1. To build a smart contract into binary code, and set Besu blockperiod time as 2 seconds;
2. To create a new account on chain;
3. To deploy the smart contract with the same account 800 times with nonce gap, and keep all the 800 TXs in TX pool;
4. Try to fix the nonce gap;
5. If you see there is no new block generate, this issue is reproducing successfully;

Expected behavior: [What you expect to happen]
When the nonce gap was fixed, all the 800 transactions will be submitted on chain, and block can be generated in a right way.

Actual behavior: [What actually happens]
The chain is stucked, and no new block is generated.

Frequency: [What percentage of the time does it occur?]
100%

Logs (if a bug)

Please post relevant logs from Besu (and the consensus client, if running proof of stake) from before and after the issue.

Versions (Add all that apply)

• Software version: [besu --version] Besu 23 ~ 24.7.0
• Java version: [java -version] JDK17 for Besu23, JDK21 for Besu 24
• OS Name & Version: [cat /etc/*release] Redhat
• Kernel Version: [uname -a]
• Virtual Machine software & version: [vmware -v]
• Docker Version: [docker version]
• Cloud VM, type, size: [Amazon Web Services I3-large]
• Consensus Client & Version if using Proof of Stake: [e.g. Teku, Lighthouse, Prysm, Nimbus, Lodestar]

Smart contract information (If you're reporting an issue arising from deploying or calling a smart contract, please supply related information)

• Solidity version [solc --version]
• Repo with minimal set of deployable/reproducible contract code - please provide a link
• Please include specifics on how you are deploying/calling the contract
• Have you reproduced the issue on other eth clients

Additional Information (Add any of the following or anything else that may be relevant)

• Besu setup info - genesis file, config options
• System info - memory, CPU

[fab-10]

Not sure if you have tuned some options, so in case I will suggest to try the following:

• Select the txpool implementation to use: layered or sequenced (AKA legacy). Since 23.10.0 layered is the default, use tx-pool option to switch between the 2
• Then since you are sending a lot of txs from the same sender, then depending on the implementation you choose, you need to tune it:
  • layered: tx-pool-max-future-by-sender to a value > of the max number of txs sent that a sender could send together, in your case >800
  • sequenced: tx-pool-limit-by-account-percentage=1.0 assuming tx-pool-max-size is big enough to keep all the pending txs for your network

after this tuning you should not experience nonce gap issues anymore (actual result could depend on the fact that other sender and pending txs are present in the pool) and if you still see that txs are still not included due to "max selection timeout issue" or other issues, let me know, so we can investigate further.

[abdul745]

I cannot exactly answer the question but I have recently raised the similar issue. I am unable to deploy smart contracts on my private network using Metamask.
Here is detail of the issue: #7612

Can you please help?

[cdfly813]

@fab-10 hi Fabio, sorry for late due to I was on my holiday this week.
I already used the sequenced TX pool and set tx-pool-limit-by-account-percentage=1.0 on Besu 24.7.0, but the issue is still there. When I change the blockperiodseconds from 2 to 5(not change this field, just change it in TX part as document told me), and the TX pool not stuck in 800 transactions, it will stuck around 2000 TXs in TX pool. it looks like the change state machine states is not right anymore.

[fab-10]

I suppose you are running a *BFT network, so in that case, if the issue is that a tx takes too long to execute, then you can try to adjust this option https://besu.hyperledger.org/development/private-networks/reference/cli/options#poa-block-txs-selection-max-time to something more than 100, so block creation could take more than 2 seconds.

If also that does not solve your problem, then it will be great if you can share how to reproduce the issue.