v1.29.0
Major Highlights
Breaking changes
Configuration flag Discovery.Discv5Enabled
is replaced with a more generic flag: Discovery.DiscoveryVersion
with V4
as a default value. To enable v5 discovery, use the following: --Discovery.DiscoveryVersion=V5
.
Heuristic transaction censorship detection
We have introduced heuristics-based censorship detection in the Nethermind client. Censorship is detected using two kinds of heuristics, namely, high-paying transaction censorship and address censorship.
In the former, a block is marked potentially censored if the highest non-blob transaction included in the block is worse than the best non-blob transaction available in the pool. The latter, which could also be opted out of, requires specifying a list of addresses that we want to track censorship for. We count the number of tracked addresses to which transactions better than the worst non-blob transaction in the block is sent. If the number of such addresses in the block is less than half the number of these addresses in the pool, we mark the block as potentially censored. If two consecutive blocks are flagged as potentially censored, regardless of the kind, we detect censorship.
Detection of censorship sets the ShouldOverrideBuilder
flag to true which acts as a circuit-breaker and communicates to the beacon node that it should fall back to local execution instead of relying on the builder's block.
Performance improvements
On Windows block validation performance used to degrade over several hours due to the default native allocator used by RocksDB. In #7418 we changed this to use the newer SegmentHeap
that was introduced in Windows 10 and you can see there is now no throughput degradation over-time and Nethermind maintains a very high tip of chain processing speed.
Base and Ethereum target MGas/s chain speed are added for reference above.
With this change, as well as 25 other performance improvements, Nethermind maintains a throughput an order of magnitude higher than even Base’s current gaslimit on a gaming or high end workstation PC (with locally PCIe attached Samsung 990 Pro NVMes).
This is also reflected in the end to end new payload processing; receiving and parsing the Json request from the consensus client to producing the validated Json response per block on average in under 20ms.
Slot time for mainnet is 12000ms, and ideally without ePBS the EL should complete in under 4000ms; with ePBS it has the full slot to complete. So Nethermind; on this setup, even at max time is completing the block with 98% of headroom remaining.
- Reduced memory during Archive, Full Sync and head catchup
- Introduced idle memory cleanup if waiting for new blocks from consensus layer; which can happen during head catchup, depending on consensus peering
- Introducing a new Macro Op:
EXTCODESIZE+POP
which was a common pattern during the Shanghai DoS attacks; however the opcode pair combination can also be evaluated to a NOP not requiring to access the account or code for the address referred to. - Non-allocating key generation for In64 keys; this becomes impactful as these sync modes process more than 3000 blocks per second.
- Calculate all the receipts’ blooms together in parallel
- Fast wait-free and lockless Keccak cache; applied in a targeted manner (as most Keccaks are random), where the same address or UInt256 is keccak’d multiple times.
- Further improvements to data prewarming for the transaction pipeline processing.
Changelog
New features
- Heuristic tx censorship detection by @Arindam2407 in #7259
- Add
engine_getBlobsV1
by @marcindsobczak in #7322 - Option to run Discovery V4 and V5 on the same port by @alexb5dh in #7213
- Implement eth_getRawTransactionByHash by @LukaszRozmej in #7425
- Feature/geth like system tx by @LukaszRozmej in #7252
Bug fixes and stability
- Incorrect batch unwrapping in JSON-RPC replay tool by @emlautarom1 in #7279
- Eth_simulate blob related bugfixes by @OlegJakushkin in #7308
- Bugfix/snap and syncerver by @MarekM25 in #7305
- fix eth_gasEstimate issue by @rjnrohit in #7312
- JSON RPC: fix missing params when all are optional by @LukaszRozmej in #7318
- Fix resources cleanup in eth/* protocols by @alexb5dh in #7311
- Removes custom args validation and uses library one by @LukaszRozmej in #7359
- Fix frequent warning on invalid Discovery packet size by @alexb5dh in #7395
Optimizations:
- Changes Windows native Allocator so it doesn't degrade over time by @benaadams in #7418
- Calculate receipt blooms in parallel by @benaadams in #7473
- Reduce backlog threshold for GC by @benaadams in #7415
- Introduces prewarming of all addresses; which can continue ahead of tx prewarming by @benaadams in #7423
- Macro Op: EXTCODESIZE+POP, turns a (historic) DoS vector into a NOP by @benaadams in #7437
- [Fix]
MultiSyncModeSelector
.Changed
event propagation being blocked by SyncFeed by @smartprogrammer93 in #6529 - Optimize out DaoTransition call by @benaadams in #7468
- Introduces memory cleanup while idle to reduce memory size by @benaadams in #7432
- Allows precaches to clear in background between blocks and reduces overclearing by @benaadams in #7442
- Use non-allocating Int64 key generation by @benaadams in #7413
- Fast SIMD address equals by @benaadams in #7417
- Removes two flow control exceptions for a faster path by @benaadams in #7436
- Reduce memory use during Archive Sync by @benaadams in #7407
- Add dual array concat overload by @benaadams in #7377
- Reduce collection resizing costs by @benaadams in #7360
- Only need single stack for stacks by @benaadams in #7375
- Prewarm txns in order (of processing) by @benaadams in #7327
- Improve parallel signature recover by @benaadams in #7353
- Reduce contention in ClockCache by @benaadams in #7338
- Clock cache additional improvements by @benaadams in #7339
- Move prewarm wait to end of all block activity by @benaadams in #7348
- Reduces memory use for peer Discovery message deserialization by @benaadams in #7427
- Use ClockCache rather than LurCache for blocks and headers by @benaadams in #7439
- Fast Keccak cache (wait-free and lockless) by @Scooletz in #7336
Other changes
- LoadFromTree improve tracing closure name by @benaadams in #7342
- Revise the configuration options docs by @rubo in #7465
- Fix compile errors by @rubo in #7357
- Fix argument null checks by @rubo in #7466
- Fix last seen asserts by @benaadams in #7384
- Refactor and improve DB size generator by @rubo in #7450
- Moved BlockProductionTimeout to config, increased BlockProductionTimeout by @MarekM25 in #7378
- OP funding by @MarekM25 in #7387
- Funding OP RPGF, project nr 2 by @MarekM25 in #7388
- Funding OP RPGF by @MarekM25 in #7389
- OP RPGF funding.json by @MarekM25 in #7391
- Upgrade to RocksDB v9.4.0 by @rubo in #7300
- Disable warning in engine_getBlobsV1 by @marcindsobczak in #7380
- removed log in EthereumEcdsa as dependency by @ak88 in #7217
- Configure Optimism and Base fast sync settings update by @rubo in #7448
- Refactor
TxValidator
by @emlautarom1 in #7386 - change PruningBoundry setting log to Info by @tanishqjasoria in #7396
- Refactor
TxDecoder
by @emlautarom1 in #7334 - Remove obsolete flag
FastBlocks
from configs by @marcindsobczak in #7363 - Remove coupling from merge plugin and aura by @smartprogrammer93 in #7044
- Detailed port-in-use error by @alexb5dh in #7316
- add prewarmer to existing tests by @smartprogrammer93 in #7272
- Some basic refactors of Nethermind.ExternalSigner.Plugin by @LukaszRozmej in #7349
- Unified invalid blocks handling in logs by @MarekM25 in #6749
- Display parameters of full pruning by @flcl42 in #7372
- Validate and log when an unrecognized parameter/flag is used by @wugalde19 in #7331
- 7465Add GenesisTimeoutMs to BlocksConfig by @LukaszRozmej in #7457
New Contributors
- @wugalde19 made their first contribution in #7331
- @Arindam2407 made their first contribution in #7259
Full Changelog: 1.28.0...1.29.0-rc