Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: adjust n-historical-states param #7104

Open
wants to merge 4 commits into
base: unstable
Choose a base branch
from
Open

fix: adjust n-historical-states param #7104

wants to merge 4 commits into from
+26 −18

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Sep 24, 2024 •
edited
Loading

Motivation

In holesky, lodestar had to reload and regen state for a weird orphaned block

Sep-22 00:17:11.240[chain]           ^[[34mdebug^[[39m: Replayed blocks to get state stateRoot=0x6f30844c4055a9e5567eab7fc205a7de97d7d40ca1e53b56a6381dcce4f58fd2, replaySlots=2588428,2588427,2588425,2588424,2588423,2588422,2588421,2588420,2588419,2588418,2588417
Sep-22 00:17:12.740[chain]         ^[[36mverbose^[[39m: Added checkpoint state to memory epoch=80889, rootHex=0xf41b326bcb2c7dfb29cf52dd0c70fd14dd58b280cf3069ff14dcc38b3ea9d2d0
Sep-22 00:17:12.740[chain]         ^[[36mverbose^[[39m: Clock slot slot=2588486  ====== epoch 80890
Sep-22 00:17:12.765[network]         ^[[34mdebug^[[39m: Received gossip block slot=2588454, root=0x5c40…cdc1, curentSlot=2588486, peerId=16Uiu2HAm5mSqbUSWaUwUHpDjy6VSoQAtdic7xnfshhZjsTiK35yh, delaySec=376.0390000343323, pending=blob, haveBlobs=0, expectedBlobs=3, recvToValLatency=0.034999847412109375, recvToValidation=8.7260000705719, validationTime=8.69100022315979

this happens on a n-historical states node

Description

  • Change DEFAULT_MAX_BLOCK_STATES to 64, this guarantees we don't have to reload state due to gossip block
  • Due to that, change DEFAULT_MAX_CP_STATE_EPOCHS_IN_MEMORY to 3.
# current config n-historical-states default config
block state cache size 64-96 64
checkpoint state cache epochs 3-4 3
  • Modify some logs to make it easier to debug later

@codecov Codecov
Copy link

codecov bot commented Sep 24, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 7.69231% with 12 lines in your changes missing coverage. Please review.

Project coverage is 48.95%. Comparing base (77006ea) to head (9411d35).
Report is 5 commits behind head on unstable.

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #7104      +/-   ##
============================================
+ Coverage     48.94%   48.95%   +0.01%     
============================================
  Files           597      597              
  Lines         39803    39833      +30     
  Branches       2065     2065              
============================================
+ Hits          19481    19502      +21     
- Misses        20280    20290      +10     
+ Partials         42       41       -1

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 24, 2024 •
edited
Loading

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite Current: 7dc8b5b Previous: fe7e21b Ratio
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 45.264 us/op 175.07 us/op 0.26
BLS verify - blst 894.54 us/op 2.7948 ms/op 0.32
BLS verifyMultipleSignatures 8 - blst 2.0473 ms/op 7.0275 ms/op 0.29
BLS verifyMultipleSignatures 32 - blst 4.4113 ms/op 20.324 ms/op 0.22
BLS verifyMultipleSignatures 64 - blst 8.0474 ms/op 37.674 ms/op 0.21
BLS verifyMultipleSignatures 128 - blst 15.945 ms/op 55.985 ms/op 0.28
BLS deserializing 10000 signatures 635.73 ms/op 2.0264 s/op 0.31
BLS deserializing 100000 signatures 6.1247 s/op 20.064 s/op 0.31
BLS verifyMultipleSignatures - same message - 3 - blst 968.50 us/op 3.4407 ms/op 0.28
BLS verifyMultipleSignatures - same message - 8 - blst 1.0786 ms/op 3.3593 ms/op 0.32
BLS verifyMultipleSignatures - same message - 32 - blst 1.7040 ms/op 6.0416 ms/op 0.28
BLS verifyMultipleSignatures - same message - 64 - blst 2.4530 ms/op 10.390 ms/op 0.24
BLS verifyMultipleSignatures - same message - 128 - blst 3.9538 ms/op 15.403 ms/op 0.26
BLS aggregatePubkeys 32 - blst 17.906 us/op 90.141 us/op 0.20
BLS aggregatePubkeys 128 - blst 63.067 us/op 270.70 us/op 0.23
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 43.640 ms/op 170.72 ms/op 0.26
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 30.142 ms/op 96.809 ms/op 0.31
proposeBlockBody type=full, size=empty 4.8977 ms/op 14.902 ms/op 0.33
batch validate gossip attestation - vc 640000 - chunk 128 92.086 us/op 277.69 us/op 0.33
pickEth1Vote - no votes 808.32 us/op 2.6460 ms/op 0.31
Object get x1000 4.8830 ns/op 17.779 ns/op 0.27
Map get x1000 5.8660 ns/op 20.442 ns/op 0.29
Object set x1000 22.886 ns/op 93.599 ns/op 0.24
Map set x1000 19.836 ns/op 60.696 ns/op 0.33
toHex 102.29 ns/op 386.16 ns/op 0.26
Buffer.from 100.08 ns/op 337.67 ns/op 0.30
shared Buffer 64.604 ns/op 224.86 ns/op 0.29
fastMsgIdFn sha256 / 10000 bytes 49.214 us/op 171.89 us/op 0.29
send data - 1000 65536B messages 243.45 ms/op 1.1354 s/op 0.21
Uint8Array.set 32 items 1.2710 us/op 4.2090 us/op 0.30
Buffer.copy 1.4760 us/op 4.9950 us/op 0.30
Uint8Array.set - with subarray 2.0130 us/op 7.9070 us/op 0.25
Set add up to 64 items then delete first 1.6969 us/op 6.3741 us/op 0.27
OrderedSet add up to 64 items then delete first 2.6415 us/op 10.273 us/op 0.26
Set add up to 64 items then delete last 1.9498 us/op 7.4399 us/op 0.26
Set add up to 64 items then delete middle 1.9418 us/op 6.1435 us/op 0.32
Set add up to 128 items then delete first 3.8532 us/op 13.941 us/op 0.28
OrderedSet add up to 128 items then delete first 5.6677 us/op 24.782 us/op 0.23
Set add up to 128 items then delete last 3.7099 us/op 15.192 us/op 0.24
OrderedSet add up to 128 items then delete last 5.8413 us/op 21.894 us/op 0.27
Set add up to 128 items then delete middle 3.7139 us/op 13.781 us/op 0.27
OrderedSet add up to 128 items then delete middle 11.397 us/op 37.267 us/op 0.31
Set add up to 256 items then delete first 7.4962 us/op 30.880 us/op 0.24
OrderedSet add up to 256 items then delete first 11.807 us/op 43.956 us/op 0.27
Set add up to 256 items then delete last 7.5620 us/op 25.945 us/op 0.29
pass gossip attestations to forkchoice per slot 2.4290 ms/op 8.5510 ms/op 0.28
forkChoice updateHead vc 600000 bc 64 eq 0 2.4995 ms/op 11.614 ms/op 0.22
forkChoice updateHead vc 1000000 bc 64 eq 0 4.2027 ms/op 13.358 ms/op 0.31
forkChoice updateHead vc 600000 bc 320 eq 0 2.5119 ms/op 8.8071 ms/op 0.29
forkChoice updateHead vc 600000 bc 1200 eq 0 2.4718 ms/op 7.8778 ms/op 0.31
forkChoice updateHead vc 600000 bc 64 eq 300000 12.064 ms/op 51.115 ms/op 0.24
computeDeltas 750000 validators 1200 proto nodes 4.3652 ms/op 13.694 ms/op 0.32
computeDeltas 750000 validators 7200 proto nodes 4.2995 ms/op 13.090 ms/op 0.33
computeDeltas 2100000 validators 300 proto nodes 12.163 ms/op 38.997 ms/op 0.31
computeDeltas 2100000 validators 1200 proto nodes 12.122 ms/op 42.353 ms/op 0.29
computeDeltas 2100000 validators 7200 proto nodes 12.021 ms/op 41.342 ms/op 0.29
altair processAttestation - setStatus - 4/5 committees join 489.62 us/op 1.5583 ms/op 0.31
Array.fill - length 1000000 2.2263 ms/op 8.9082 ms/op 0.25
Array push - length 1000000 13.477 ms/op 43.679 ms/op 0.31
Buffer.compare 32 - diff last byte 14.284 ns/op 43.380 ns/op 0.33
arrayWithProxy get 100000 times 11.125 ms/op 35.531 ms/op 0.31
findModifiedValidators - 10 modified validators 134.54 ms/op 421.23 ms/op 0.32
findModifiedValidators - 1 modified validators 128.08 ms/op 393.25 ms/op 0.33
Full benchmark results
Benchmark suite Current: 7dc8b5b Previous: fe7e21b Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.6276 ms/op 4.7169 ms/op 0.35
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 45.264 us/op 175.07 us/op 0.26
BLS verify - blst 894.54 us/op 2.7948 ms/op 0.32
BLS verifyMultipleSignatures 3 - blst 1.4275 ms/op 3.7195 ms/op 0.38
BLS verifyMultipleSignatures 8 - blst 2.0473 ms/op 7.0275 ms/op 0.29
BLS verifyMultipleSignatures 32 - blst 4.4113 ms/op 20.324 ms/op 0.22
BLS verifyMultipleSignatures 64 - blst 8.0474 ms/op 37.674 ms/op 0.21
BLS verifyMultipleSignatures 128 - blst 15.945 ms/op 55.985 ms/op 0.28
BLS deserializing 10000 signatures 635.73 ms/op 2.0264 s/op 0.31
BLS deserializing 100000 signatures 6.1247 s/op 20.064 s/op 0.31
BLS verifyMultipleSignatures - same message - 3 - blst 968.50 us/op 3.4407 ms/op 0.28
BLS verifyMultipleSignatures - same message - 8 - blst 1.0786 ms/op 3.3593 ms/op 0.32
BLS verifyMultipleSignatures - same message - 32 - blst 1.7040 ms/op 6.0416 ms/op 0.28
BLS verifyMultipleSignatures - same message - 64 - blst 2.4530 ms/op 10.390 ms/op 0.24
BLS verifyMultipleSignatures - same message - 128 - blst 3.9538 ms/op 15.403 ms/op 0.26
BLS aggregatePubkeys 32 - blst 17.906 us/op 90.141 us/op 0.20
BLS aggregatePubkeys 128 - blst 63.067 us/op 270.70 us/op 0.23
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 55.094 ms/op 124.47 ms/op 0.44
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 43.640 ms/op 170.72 ms/op 0.26
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 30.142 ms/op 96.809 ms/op 0.31
getSlashingsAndExits - default max 92.171 us/op 252.18 us/op 0.37
getSlashingsAndExits - 2k 284.06 us/op 475.83 us/op 0.60
proposeBlockBody type=full, size=empty 4.8977 ms/op 14.902 ms/op 0.33
isKnown best case - 1 super set check 479.00 ns/op 1.1150 us/op 0.43
isKnown normal case - 2 super set checks 460.00 ns/op 1.0330 us/op 0.45
isKnown worse case - 16 super set checks 503.00 ns/op 1.0610 us/op 0.47
InMemoryCheckpointStateCache - add get delete 3.2590 us/op 7.1060 us/op 0.46
updateUnfinalizedPubkeys - updating 10 pubkeys 874.02 us/op 1.9958 ms/op 0.44
updateUnfinalizedPubkeys - updating 100 pubkeys 2.5671 ms/op 11.103 ms/op 0.23
updateUnfinalizedPubkeys - updating 1000 pubkeys 36.751 ms/op 124.50 ms/op 0.30
validate api signedAggregateAndProof - struct 1.9075 ms/op 3.6943 ms/op 0.52
validate gossip signedAggregateAndProof - struct 1.8037 ms/op 3.2083 ms/op 0.56
validate gossip attestation - vc 640000 970.29 us/op 1.8684 ms/op 0.52
batch validate gossip attestation - vc 640000 - chunk 32 115.28 us/op 305.70 us/op 0.38
batch validate gossip attestation - vc 640000 - chunk 64 101.24 us/op 283.73 us/op 0.36
batch validate gossip attestation - vc 640000 - chunk 128 92.086 us/op 277.69 us/op 0.33
batch validate gossip attestation - vc 640000 - chunk 256 88.244 us/op 225.92 us/op 0.39
pickEth1Vote - no votes 808.32 us/op 2.6460 ms/op 0.31
pickEth1Vote - max votes 4.1915 ms/op 12.424 ms/op 0.34
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.6431 ms/op 27.255 ms/op 0.35
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.352 ms/op 38.255 ms/op 0.38
pickEth1Vote - Eth1Data fastSerialize value x2048 333.13 us/op 1.0652 ms/op 0.31
pickEth1Vote - Eth1Data fastSerialize tree x2048 1.9586 ms/op 6.1004 ms/op 0.32
bytes32 toHexString 572.00 ns/op 1.1570 us/op 0.49
bytes32 Buffer.toString(hex) 439.00 ns/op 619.00 ns/op 0.71
bytes32 Buffer.toString(hex) from Uint8Array 554.00 ns/op 918.00 ns/op 0.60
bytes32 Buffer.toString(hex) + 0x 452.00 ns/op 571.00 ns/op 0.79
Object access 1 prop 0.31200 ns/op 0.35300 ns/op 0.88
Map access 1 prop 0.31300 ns/op 0.32000 ns/op 0.98
Object get x1000 4.8830 ns/op 17.779 ns/op 0.27
Map get x1000 5.8660 ns/op 20.442 ns/op 0.29
Object set x1000 22.886 ns/op 93.599 ns/op 0.24
Map set x1000 19.836 ns/op 60.696 ns/op 0.33
Return object 10000 times 0.28680 ns/op 0.63430 ns/op 0.45
Throw Error 10000 times 2.5863 us/op 8.1671 us/op 0.32
toHex 102.29 ns/op 386.16 ns/op 0.26
Buffer.from 100.08 ns/op 337.67 ns/op 0.30
shared Buffer 64.604 ns/op 224.86 ns/op 0.29
fastMsgIdFn sha256 / 200 bytes 1.9570 us/op 4.8410 us/op 0.40
fastMsgIdFn h32 xxhash / 200 bytes 426.00 ns/op 491.00 ns/op 0.87
fastMsgIdFn h64 xxhash / 200 bytes 450.00 ns/op 548.00 ns/op 0.82
fastMsgIdFn sha256 / 1000 bytes 5.8510 us/op 16.999 us/op 0.34
fastMsgIdFn h32 xxhash / 1000 bytes 534.00 ns/op 847.00 ns/op 0.63
fastMsgIdFn h64 xxhash / 1000 bytes 512.00 ns/op 822.00 ns/op 0.62
fastMsgIdFn sha256 / 10000 bytes 49.214 us/op 171.89 us/op 0.29
fastMsgIdFn h32 xxhash / 10000 bytes 1.8580 us/op 4.3550 us/op 0.43
fastMsgIdFn h64 xxhash / 10000 bytes 1.3000 us/op 2.6800 us/op 0.49
send data - 1000 256B messages 9.9497 ms/op 24.282 ms/op 0.41
send data - 1000 512B messages 13.152 ms/op 34.865 ms/op 0.38
send data - 1000 1024B messages 22.704 ms/op 51.093 ms/op 0.44
send data - 1000 1200B messages 22.585 ms/op 46.894 ms/op 0.48
send data - 1000 2048B messages 27.812 ms/op 40.346 ms/op 0.69
send data - 1000 4096B messages 25.639 ms/op 60.136 ms/op 0.43
send data - 1000 16384B messages 65.899 ms/op 170.07 ms/op 0.39
send data - 1000 65536B messages 243.45 ms/op 1.1354 s/op 0.21
enrSubnets - fastDeserialize 64 bits 1.1460 us/op 2.5970 us/op 0.44
enrSubnets - ssz BitVector 64 bits 493.00 ns/op 883.00 ns/op 0.56
enrSubnets - fastDeserialize 4 bits 323.00 ns/op 383.00 ns/op 0.84
enrSubnets - ssz BitVector 4 bits 497.00 ns/op 964.00 ns/op 0.52
prioritizePeers score -10:0 att 32-0.1 sync 2-0 137.42 us/op 370.01 us/op 0.37
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 151.05 us/op 412.69 us/op 0.37
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 331.40 us/op 560.77 us/op 0.59
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 449.71 us/op 911.92 us/op 0.49
prioritizePeers score 0:0 att 64-1 sync 4-1 642.04 us/op 1.2805 ms/op 0.50
array of 16000 items push then shift 1.2333 us/op 3.3952 us/op 0.36
LinkedList of 16000 items push then shift 7.0870 ns/op 16.667 ns/op 0.43
array of 16000 items push then pop 82.000 ns/op 206.04 ns/op 0.40
LinkedList of 16000 items push then pop 6.1050 ns/op 15.268 ns/op 0.40
array of 24000 items push then shift 1.7747 us/op 4.0991 us/op 0.43
LinkedList of 24000 items push then shift 6.7060 ns/op 14.376 ns/op 0.47
array of 24000 items push then pop 105.04 ns/op 370.56 ns/op 0.28
LinkedList of 24000 items push then pop 6.1060 ns/op 18.928 ns/op 0.32
intersect bitArray bitLen 8 5.2590 ns/op 21.200 ns/op 0.25
intersect array and set length 8 36.853 ns/op 110.89 ns/op 0.33
intersect bitArray bitLen 128 25.980 ns/op 78.282 ns/op 0.33
intersect array and set length 128 552.52 ns/op 2.2886 us/op 0.24
bitArray.getTrueBitIndexes() bitLen 128 2.0120 us/op 5.0550 us/op 0.40
bitArray.getTrueBitIndexes() bitLen 248 3.2100 us/op 8.2770 us/op 0.39
bitArray.getTrueBitIndexes() bitLen 512 7.3130 us/op 21.514 us/op 0.34
Buffer.concat 32 items 940.00 ns/op 2.7940 us/op 0.34
Uint8Array.set 32 items 1.2710 us/op 4.2090 us/op 0.30
Buffer.copy 1.4760 us/op 4.9950 us/op 0.30
Uint8Array.set - with subarray 2.0130 us/op 7.9070 us/op 0.25
Uint8Array.set - without subarray 1.4740 us/op 3.2860 us/op 0.45
getUint32 - dataview 383.00 ns/op 651.00 ns/op 0.59
getUint32 - manual 314.00 ns/op 515.00 ns/op 0.61
Set add up to 64 items then delete first 1.6969 us/op 6.3741 us/op 0.27
OrderedSet add up to 64 items then delete first 2.6415 us/op 10.273 us/op 0.26
Set add up to 64 items then delete last 1.9498 us/op 7.4399 us/op 0.26
OrderedSet add up to 64 items then delete last 3.1039 us/op 8.6487 us/op 0.36
Set add up to 64 items then delete middle 1.9418 us/op 6.1435 us/op 0.32
OrderedSet add up to 64 items then delete middle 4.3402 us/op 11.537 us/op 0.38
Set add up to 128 items then delete first 3.8532 us/op 13.941 us/op 0.28
OrderedSet add up to 128 items then delete first 5.6677 us/op 24.782 us/op 0.23
Set add up to 128 items then delete last 3.7099 us/op 15.192 us/op 0.24
OrderedSet add up to 128 items then delete last 5.8413 us/op 21.894 us/op 0.27
Set add up to 128 items then delete middle 3.7139 us/op 13.781 us/op 0.27
OrderedSet add up to 128 items then delete middle 11.397 us/op 37.267 us/op 0.31
Set add up to 256 items then delete first 7.4962 us/op 30.880 us/op 0.24
OrderedSet add up to 256 items then delete first 11.807 us/op 43.956 us/op 0.27
Set add up to 256 items then delete last 7.5620 us/op 25.945 us/op 0.29
OrderedSet add up to 256 items then delete last 11.891 us/op 29.882 us/op 0.40
Set add up to 256 items then delete middle 7.3689 us/op 19.261 us/op 0.38
OrderedSet add up to 256 items then delete middle 33.362 us/op 76.914 us/op 0.43
transfer serialized Status (84 B) 1.4650 us/op 2.2100 us/op 0.66
copy serialized Status (84 B) 1.2290 us/op 1.9550 us/op 0.63
transfer serialized SignedVoluntaryExit (112 B) 1.4820 us/op 2.3760 us/op 0.62
copy serialized SignedVoluntaryExit (112 B) 1.3200 us/op 2.2190 us/op 0.59
transfer serialized ProposerSlashing (416 B) 1.6970 us/op 4.0410 us/op 0.42
copy serialized ProposerSlashing (416 B) 1.6920 us/op 4.2540 us/op 0.40
transfer serialized Attestation (485 B) 2.4840 us/op 4.7790 us/op 0.52
copy serialized Attestation (485 B) 1.7240 us/op 4.3660 us/op 0.39
transfer serialized AttesterSlashing (33232 B) 2.2580 us/op 5.2030 us/op 0.43
copy serialized AttesterSlashing (33232 B) 4.8850 us/op 23.559 us/op 0.21
transfer serialized Small SignedBeaconBlock (128000 B) 2.7600 us/op 5.3950 us/op 0.51
copy serialized Small SignedBeaconBlock (128000 B) 15.632 us/op 36.692 us/op 0.43
transfer serialized Avg SignedBeaconBlock (200000 B) 3.2110 us/op 5.4930 us/op 0.58
copy serialized Avg SignedBeaconBlock (200000 B) 24.055 us/op 133.25 us/op 0.18
transfer serialized BlobsSidecar (524380 B) 3.2400 us/op 6.2350 us/op 0.52
copy serialized BlobsSidecar (524380 B) 76.560 us/op 142.07 us/op 0.54
transfer serialized Big SignedBeaconBlock (1000000 B) 3.5440 us/op 5.4780 us/op 0.65
copy serialized Big SignedBeaconBlock (1000000 B) 137.53 us/op 246.93 us/op 0.56
pass gossip attestations to forkchoice per slot 2.4290 ms/op 8.5510 ms/op 0.28
forkChoice updateHead vc 100000 bc 64 eq 0 559.03 us/op 1.4313 ms/op 0.39
forkChoice updateHead vc 600000 bc 64 eq 0 2.4995 ms/op 11.614 ms/op 0.22
forkChoice updateHead vc 1000000 bc 64 eq 0 4.2027 ms/op 13.358 ms/op 0.31
forkChoice updateHead vc 600000 bc 320 eq 0 2.5119 ms/op 8.8071 ms/op 0.29
forkChoice updateHead vc 600000 bc 1200 eq 0 2.4718 ms/op 7.8778 ms/op 0.31
forkChoice updateHead vc 600000 bc 7200 eq 0 3.1805 ms/op 7.3968 ms/op 0.43
forkChoice updateHead vc 600000 bc 64 eq 1000 9.4395 ms/op 25.218 ms/op 0.37
forkChoice updateHead vc 600000 bc 64 eq 10000 9.3867 ms/op 23.441 ms/op 0.40
forkChoice updateHead vc 600000 bc 64 eq 300000 12.064 ms/op 51.115 ms/op 0.24
computeDeltas 500000 validators 300 proto nodes 2.9045 ms/op 6.8431 ms/op 0.42
computeDeltas 500000 validators 1200 proto nodes 2.9692 ms/op 7.6924 ms/op 0.39
computeDeltas 500000 validators 7200 proto nodes 2.9350 ms/op 8.1579 ms/op 0.36
computeDeltas 750000 validators 300 proto nodes 4.5052 ms/op 11.312 ms/op 0.40
computeDeltas 750000 validators 1200 proto nodes 4.3652 ms/op 13.694 ms/op 0.32
computeDeltas 750000 validators 7200 proto nodes 4.2995 ms/op 13.090 ms/op 0.33
computeDeltas 1400000 validators 300 proto nodes 8.0770 ms/op 21.890 ms/op 0.37
computeDeltas 1400000 validators 1200 proto nodes 8.0148 ms/op 18.953 ms/op 0.42
computeDeltas 1400000 validators 7200 proto nodes 7.9709 ms/op 17.632 ms/op 0.45
computeDeltas 2100000 validators 300 proto nodes 12.163 ms/op 38.997 ms/op 0.31
computeDeltas 2100000 validators 1200 proto nodes 12.122 ms/op 42.353 ms/op 0.29
computeDeltas 2100000 validators 7200 proto nodes 12.021 ms/op 41.342 ms/op 0.29
altair processAttestation - 250000 vs - 7PWei normalcase 1.5032 ms/op 4.3774 ms/op 0.34
altair processAttestation - 250000 vs - 7PWei worstcase 2.2538 ms/op 5.8156 ms/op 0.39
altair processAttestation - setStatus - 1/6 committees join 93.164 us/op 213.56 us/op 0.44
altair processAttestation - setStatus - 1/3 committees join 178.29 us/op 461.57 us/op 0.39
altair processAttestation - setStatus - 1/2 committees join 269.69 us/op 680.00 us/op 0.40
altair processAttestation - setStatus - 2/3 committees join 354.85 us/op 967.23 us/op 0.37
altair processAttestation - setStatus - 4/5 committees join 489.62 us/op 1.5583 ms/op 0.31
altair processAttestation - setStatus - 100% committees join 585.56 us/op 1.3392 ms/op 0.44
altair processBlock - 250000 vs - 7PWei normalcase 3.3414 ms/op 9.8911 ms/op 0.34
altair processBlock - 250000 vs - 7PWei normalcase hashState 24.836 ms/op 52.769 ms/op 0.47
altair processBlock - 250000 vs - 7PWei worstcase 38.662 ms/op 73.997 ms/op 0.52
altair processBlock - 250000 vs - 7PWei worstcase hashState 75.018 ms/op 131.03 ms/op 0.57
phase0 processBlock - 250000 vs - 7PWei normalcase 1.7789 ms/op 3.8590 ms/op 0.46
phase0 processBlock - 250000 vs - 7PWei worstcase 23.067 ms/op 38.886 ms/op 0.59
altair processEth1Data - 250000 vs - 7PWei normalcase 238.92 us/op 677.22 us/op 0.35
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 2.7470 us/op 8.1650 us/op 0.34
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 30.924 us/op 53.469 us/op 0.58
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 8.4380 us/op 14.651 us/op 0.58
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 5.3480 us/op 11.870 us/op 0.45
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 117.32 us/op 211.22 us/op 0.56
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 765.97 us/op 2.0662 ms/op 0.37
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.0471 ms/op 2.8623 ms/op 0.37
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.2003 ms/op 3.0193 ms/op 0.40
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 2.6542 ms/op 7.1082 ms/op 0.37
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.1313 ms/op 3.1089 ms/op 0.36
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 2.9456 ms/op 7.3160 ms/op 0.40
Tree 40 250000 create 182.40 ms/op 395.03 ms/op 0.46
Tree 40 250000 get(125000) 114.08 ns/op 275.09 ns/op 0.41
Tree 40 250000 set(125000) 549.07 ns/op 1.0792 us/op 0.51
Tree 40 250000 toArray() 9.6426 ms/op 22.791 ms/op 0.42
Tree 40 250000 iterate all - toArray() + loop 9.6567 ms/op 23.471 ms/op 0.41
Tree 40 250000 iterate all - get(i) 40.442 ms/op 88.460 ms/op 0.46
Array 250000 create 2.3682 ms/op 5.2210 ms/op 0.45
Array 250000 clone - spread 1.1267 ms/op 2.4276 ms/op 0.46
Array 250000 get(125000) 0.56500 ns/op 0.74200 ns/op 0.76
Array 250000 set(125000) 0.57300 ns/op 0.83700 ns/op 0.68
Array 250000 iterate all - loop 72.624 us/op 141.19 us/op 0.51
phase0 afterProcessEpoch - 250000 vs - 7PWei 68.374 ms/op 184.92 ms/op 0.37
Array.fill - length 1000000 2.2263 ms/op 8.9082 ms/op 0.25
Array push - length 1000000 13.477 ms/op 43.679 ms/op 0.31
Array.get 0.25821 ns/op 0.44581 ns/op 0.58
Uint8Array.get 0.34145 ns/op 0.70910 ns/op 0.48
phase0 beforeProcessEpoch - 250000 vs - 7PWei 12.115 ms/op 31.815 ms/op 0.38
altair processEpoch - mainnet_e81889 288.09 ms/op 473.43 ms/op 0.61
mainnet_e81889 - altair beforeProcessEpoch 15.660 ms/op 30.342 ms/op 0.52
mainnet_e81889 - altair processJustificationAndFinalization 5.7530 us/op 23.129 us/op 0.25
mainnet_e81889 - altair processInactivityUpdates 3.9497 ms/op 7.0594 ms/op 0.56
mainnet_e81889 - altair processRewardsAndPenalties 53.199 ms/op 68.917 ms/op 0.77
mainnet_e81889 - altair processRegistryUpdates 1.8340 us/op 2.8930 us/op 0.63
mainnet_e81889 - altair processSlashings 792.00 ns/op 743.00 ns/op 1.07
mainnet_e81889 - altair processEth1DataReset 745.00 ns/op 516.00 ns/op 1.44
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.6313 ms/op 2.0618 ms/op 0.79
mainnet_e81889 - altair processSlashingsReset 2.1470 us/op 4.7750 us/op 0.45
mainnet_e81889 - altair processRandaoMixesReset 2.6540 us/op 6.5870 us/op 0.40
mainnet_e81889 - altair processHistoricalRootsUpdate 659.00 ns/op 839.00 ns/op 0.79
mainnet_e81889 - altair processParticipationFlagUpdates 1.4580 us/op 2.4300 us/op 0.60
mainnet_e81889 - altair processSyncCommitteeUpdates 592.00 ns/op 577.00 ns/op 1.03
mainnet_e81889 - altair afterProcessEpoch 77.406 ms/op 139.25 ms/op 0.56
capella processEpoch - mainnet_e217614 1.0845 s/op 1.5813 s/op 0.69
mainnet_e217614 - capella beforeProcessEpoch 61.463 ms/op 85.124 ms/op 0.72
mainnet_e217614 - capella processJustificationAndFinalization 6.9830 us/op 20.023 us/op 0.35
mainnet_e217614 - capella processInactivityUpdates 12.221 ms/op 20.933 ms/op 0.58
mainnet_e217614 - capella processRewardsAndPenalties 241.87 ms/op 295.58 ms/op 0.82
mainnet_e217614 - capella processRegistryUpdates 11.993 us/op 17.168 us/op 0.70
mainnet_e217614 - capella processSlashings 796.00 ns/op 559.00 ns/op 1.42
mainnet_e217614 - capella processEth1DataReset 698.00 ns/op 508.00 ns/op 1.37
mainnet_e217614 - capella processEffectiveBalanceUpdates 12.142 ms/op 16.758 ms/op 0.72
mainnet_e217614 - capella processSlashingsReset 2.3390 us/op 11.492 us/op 0.20
mainnet_e217614 - capella processRandaoMixesReset 3.0030 us/op 5.5460 us/op 0.54
mainnet_e217614 - capella processHistoricalRootsUpdate 681.00 ns/op 760.00 ns/op 0.90
mainnet_e217614 - capella processParticipationFlagUpdates 1.9010 us/op 2.3280 us/op 0.82
mainnet_e217614 - capella afterProcessEpoch 183.53 ms/op 327.00 ms/op 0.56
phase0 processEpoch - mainnet_e58758 343.00 ms/op 421.81 ms/op 0.81
mainnet_e58758 - phase0 beforeProcessEpoch 74.137 ms/op 85.921 ms/op 0.86
mainnet_e58758 - phase0 processJustificationAndFinalization 8.2160 us/op 16.611 us/op 0.49
mainnet_e58758 - phase0 processRewardsAndPenalties 36.652 ms/op 38.127 ms/op 0.96
mainnet_e58758 - phase0 processRegistryUpdates 6.7640 us/op 10.598 us/op 0.64
mainnet_e58758 - phase0 processSlashings 804.00 ns/op 521.00 ns/op 1.54
mainnet_e58758 - phase0 processEth1DataReset 713.00 ns/op 543.00 ns/op 1.31
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.9761 ms/op 1.9828 ms/op 1.00
mainnet_e58758 - phase0 processSlashingsReset 2.1980 us/op 3.8650 us/op 0.57
mainnet_e58758 - phase0 processRandaoMixesReset 2.8640 us/op 4.7540 us/op 0.60
mainnet_e58758 - phase0 processHistoricalRootsUpdate 667.00 ns/op 631.00 ns/op 1.06
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.1120 us/op 4.1730 us/op 0.75
mainnet_e58758 - phase0 afterProcessEpoch 61.199 ms/op 119.12 ms/op 0.51
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.0406 ms/op 2.1000 ms/op 0.50
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.6563 ms/op 3.6733 ms/op 0.45
altair processInactivityUpdates - 250000 normalcase 16.627 ms/op 22.773 ms/op 0.73
altair processInactivityUpdates - 250000 worstcase 16.444 ms/op 31.045 ms/op 0.53
phase0 processRegistryUpdates - 250000 normalcase 6.0200 us/op 13.907 us/op 0.43
phase0 processRegistryUpdates - 250000 badcase_full_deposits 289.13 us/op 520.16 us/op 0.56
phase0 processRegistryUpdates - 250000 worstcase 0.5 113.76 ms/op 201.15 ms/op 0.57
altair processRewardsAndPenalties - 250000 normalcase 53.268 ms/op 55.927 ms/op 0.95
altair processRewardsAndPenalties - 250000 worstcase 44.581 ms/op 66.958 ms/op 0.67
phase0 getAttestationDeltas - 250000 normalcase 6.3488 ms/op 11.125 ms/op 0.57
phase0 getAttestationDeltas - 250000 worstcase 5.9154 ms/op 12.691 ms/op 0.47
phase0 processSlashings - 250000 worstcase 50.477 us/op 114.68 us/op 0.44
altair processSyncCommitteeUpdates - 250000 97.729 ms/op 205.88 ms/op 0.47
BeaconState.hashTreeRoot - No change 414.00 ns/op 334.00 ns/op 1.24
BeaconState.hashTreeRoot - 1 full validator 102.47 us/op 149.22 us/op 0.69
BeaconState.hashTreeRoot - 32 full validator 1.0524 ms/op 1.4662 ms/op 0.72
BeaconState.hashTreeRoot - 512 full validator 9.9306 ms/op 16.344 ms/op 0.61
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 118.50 us/op 190.19 us/op 0.62
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.7646 ms/op 2.8136 ms/op 0.63
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 18.727 ms/op 34.316 ms/op 0.55
BeaconState.hashTreeRoot - 1 balances 101.82 us/op 142.21 us/op 0.72
BeaconState.hashTreeRoot - 32 balances 778.63 us/op 1.3937 ms/op 0.56
BeaconState.hashTreeRoot - 512 balances 7.8643 ms/op 15.254 ms/op 0.52
BeaconState.hashTreeRoot - 250000 balances 157.69 ms/op 239.76 ms/op 0.66
aggregationBits - 2048 els - zipIndexesInBitList 19.811 us/op 39.369 us/op 0.50
byteArrayEquals 32 48.267 ns/op 87.034 ns/op 0.55
Buffer.compare 32 15.502 ns/op 30.328 ns/op 0.51
byteArrayEquals 1024 1.2684 us/op 2.3244 us/op 0.55
Buffer.compare 1024 22.467 ns/op 41.134 ns/op 0.55
byteArrayEquals 16384 19.967 us/op 38.448 us/op 0.52
Buffer.compare 16384 196.44 ns/op 330.04 ns/op 0.60
byteArrayEquals 123687377 147.45 ms/op 325.11 ms/op 0.45
Buffer.compare 123687377 5.3978 ms/op 9.5798 ms/op 0.56
byteArrayEquals 32 - diff last byte 44.670 ns/op 127.55 ns/op 0.35
Buffer.compare 32 - diff last byte 14.284 ns/op 43.380 ns/op 0.33
byteArrayEquals 1024 - diff last byte 1.1805 us/op 3.5043 us/op 0.34
Buffer.compare 1024 - diff last byte 21.146 ns/op 54.528 ns/op 0.39
byteArrayEquals 16384 - diff last byte 18.593 us/op 55.163 us/op 0.34
Buffer.compare 16384 - diff last byte 165.33 ns/op 390.36 ns/op 0.42
byteArrayEquals 123687377 - diff last byte 143.39 ms/op 382.41 ms/op 0.37
Buffer.compare 123687377 - diff last byte 5.3641 ms/op 8.9067 ms/op 0.60
byteArrayEquals 32 - random bytes 4.7530 ns/op 9.7980 ns/op 0.49
Buffer.compare 32 - random bytes 14.987 ns/op 33.231 ns/op 0.45
byteArrayEquals 1024 - random bytes 4.7590 ns/op 10.096 ns/op 0.47
Buffer.compare 1024 - random bytes 14.589 ns/op 37.940 ns/op 0.38
byteArrayEquals 16384 - random bytes 4.8880 ns/op 10.579 ns/op 0.46
Buffer.compare 16384 - random bytes 15.577 ns/op 36.006 ns/op 0.43
byteArrayEquals 123687377 - random bytes 7.7900 ns/op 13.810 ns/op 0.56
Buffer.compare 123687377 - random bytes 18.690 ns/op 46.570 ns/op 0.40
regular array get 100000 times 31.152 us/op 80.143 us/op 0.39
wrappedArray get 100000 times 31.086 us/op 59.339 us/op 0.52
arrayWithProxy get 100000 times 11.125 ms/op 35.531 ms/op 0.31
ssz.Root.equals 44.183 ns/op 85.888 ns/op 0.51
byteArrayEquals 44.207 ns/op 81.313 ns/op 0.54
Buffer.compare 8.8830 ns/op 23.257 ns/op 0.38
shuffle list - 16384 els 5.3628 ms/op 11.350 ms/op 0.47
shuffle list - 250000 els 78.426 ms/op 169.03 ms/op 0.46
processSlot - 1 slots 10.030 us/op 27.146 us/op 0.37
processSlot - 32 slots 1.8282 ms/op 4.7410 ms/op 0.39
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 39.378 ms/op 69.701 ms/op 0.56
getCommitteeAssignments - req 1 vs - 250000 vc 1.7556 ms/op 4.2035 ms/op 0.42
getCommitteeAssignments - req 100 vs - 250000 vc 3.4560 ms/op 7.3561 ms/op 0.47
getCommitteeAssignments - req 1000 vs - 250000 vc 3.6689 ms/op 8.4647 ms/op 0.43
findModifiedValidators - 10000 modified validators 230.67 ms/op 584.15 ms/op 0.39
findModifiedValidators - 1000 modified validators 157.77 ms/op 455.67 ms/op 0.35
findModifiedValidators - 100 modified validators 143.54 ms/op 402.69 ms/op 0.36
findModifiedValidators - 10 modified validators 134.54 ms/op 421.23 ms/op 0.32
findModifiedValidators - 1 modified validators 128.08 ms/op 393.25 ms/op 0.33
findModifiedValidators - no difference 146.87 ms/op 384.05 ms/op 0.38
compare ViewDUs 3.3966 s/op 5.4231 s/op 0.63
compare each validator Uint8Array 1.6756 s/op 2.5755 s/op 0.65
compare ViewDU to Uint8Array 721.05 ms/op 2.0990 s/op 0.34
migrate state 1000000 validators, 24 modified, 0 new 834.25 ms/op 1.2529 s/op 0.67
migrate state 1000000 validators, 1700 modified, 1000 new 1.0843 s/op 1.3325 s/op 0.81
migrate state 1000000 validators, 3400 modified, 2000 new 1.2824 s/op 1.5773 s/op 0.81
migrate state 1500000 validators, 24 modified, 0 new 858.79 ms/op 1.0498 s/op 0.82
migrate state 1500000 validators, 1700 modified, 1000 new 1.0559 s/op 1.2457 s/op 0.85
migrate state 1500000 validators, 3400 modified, 2000 new 1.2426 s/op 1.4443 s/op 0.86
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.5500 ns/op 4.5100 ns/op 1.23
state getBlockRootAtSlot - 250000 vs - 7PWei 657.35 ns/op 726.73 ns/op 0.90
computeProposers - vc 250000 5.8225 ms/op 7.0528 ms/op 0.83
computeEpochShuffling - vc 250000 79.190 ms/op 97.492 ms/op 0.81
getNextSyncCommittee - vc 250000 99.921 ms/op 127.24 ms/op 0.79
computeSigningRoot for AttestationData 25.464 us/op 25.947 us/op 0.98
hash AttestationData serialized data then Buffer.toString(base64) 1.1627 us/op 1.6844 us/op 0.69
toHexString serialized data 793.76 ns/op 983.71 ns/op 0.81
Buffer.toString(base64) 141.43 ns/op 194.73 ns/op 0.73
nodejs block root to RootHex using toHex 128.30 ns/op 161.78 ns/op 0.79
nodejs block root to RootHex using toRootHex 76.145 ns/op 99.962 ns/op 0.76
browser block root to RootHex using the deprecated toHexString 206.56 ns/op 246.85 ns/op 0.84
browser block root to RootHex using toHex 164.63 ns/op 190.53 ns/op 0.86
browser block root to RootHex using toRootHex 145.10 ns/op 172.49 ns/op 0.84

by benchmarkbot/action

@twoeths
Copy link
Contributor Author

twoeths commented Sep 24, 2024

beacon-node heap is about 350MB-400MB less than unstable

Screenshot 2024-09-24 at 20 38 20

@philknows philknows added this to the v1.23.0 milestone Sep 24, 2024
@twoeths twoeths mentioned this pull request Sep 26, 2024
Merged
@twoeths
Copy link
Contributor Author

twoeths commented Sep 26, 2024

blocked by #7109

@twoeths twoeths force-pushed the te/adjust_n_historical_states_param branch from a3d0117 to 296ba0e Compare September 27, 2024 02:40
@twoeths
Copy link
Contributor Author

twoeths commented Sep 27, 2024

as discussed in team's meeting, I enabled nHistoricalStates flag. Some facts:

  • we've tested this for more than 3 months with the old default config (maxBlockStates=32, maxCPStateEpochsInMemory=2) on production nodes
    • mainnet: no reload happened, the 2 new state caches work well
    • holesky: 1 reload happened and resolved by this issue
  • we've also tested the minimal config on feat4 for more than 3 months (maxBlockStates=1, maxCPStateEpochsInMemory=0) without any issues
    • holesky: states are reloaded 5-6 times per day
    • mainnet: states are reloaded 2-3 times per day

with the modified default params in this PR:

  • it's a guarantee that we don't need to reload any states at all although we've implemented the state reload mechanism, this is for the future if we want to change the default params (to reduce memory for example)
  • with current network condition, we even don't need to serialize checkpoint state and persist to disk per epoch. Just in case we don't finalize for some epochs, we start persisting it to disk to make sure the beacon node functions well, and that's the main thing for nHistoricalStates. It should takes ~300ms to serialize state, see fix: n-historical-states state serialization panel #7111

@twoeths twoeths marked this pull request as ready for review September 27, 2024 08:22
@twoeths twoeths requested a review from a team as a code owner September 27, 2024 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.23.0
Development

Successfully merging this pull request may close these issues.
3 participants
@twoeths @nflaig @philknows