Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setup mergemock based builder tests #4703

Merged
merged 6 commits into from
Nov 1, 2022
Merged

Setup mergemock based builder tests #4703

merged 6 commits into from
Nov 1, 2022

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Oct 29, 2022 •
edited
Loading

Builder sim tests are currently missing in the repo,
this PR adds mergemock based builder simulation, the first 10-15 blocks are engines because of the builder circuti that disables the builder on a startup, then the builder activates and the blocks start building .

Currently there is no way to verify builder blocks, and may be in a followup validator based event can be added and counted if the test produces x number of builder blocks.

Closes #4663

@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 7b11b29 Previous: 14fe299 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.4565 ms/op 1.5674 ms/op 1.57
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 92.052 us/op 64.122 us/op 1.44
BLS verify - blst-native 2.6549 ms/op 2.1630 ms/op 1.23
BLS verifyMultipleSignatures 3 - blst-native 5.5416 ms/op 4.4828 ms/op 1.24
BLS verifyMultipleSignatures 8 - blst-native 12.054 ms/op 9.6855 ms/op 1.24
BLS verifyMultipleSignatures 32 - blst-native 43.554 ms/op 35.169 ms/op 1.24
BLS aggregatePubkeys 32 - blst-native 59.996 us/op 46.808 us/op 1.28
BLS aggregatePubkeys 128 - blst-native 234.59 us/op 182.72 us/op 1.28
getAttestationsForBlock 108.23 ms/op 73.288 ms/op 1.48
isKnown best case - 1 super set check 528.00 ns/op 467.00 ns/op 1.13
isKnown normal case - 2 super set checks 524.00 ns/op 461.00 ns/op 1.14
isKnown worse case - 16 super set checks 511.00 ns/op 465.00 ns/op 1.10
CheckpointStateCache - add get delete 11.521 us/op 9.0610 us/op 1.27
validate gossip signedAggregateAndProof - struct 6.4414 ms/op 5.0304 ms/op 1.28
validate gossip attestation - struct 3.0380 ms/op 2.3638 ms/op 1.29
pickEth1Vote - no votes 2.5692 ms/op 2.2468 ms/op 1.14
pickEth1Vote - max votes 24.226 ms/op 21.163 ms/op 1.14
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.948 ms/op 12.148 ms/op 1.15
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 23.812 ms/op 18.861 ms/op 1.26
pickEth1Vote - Eth1Data fastSerialize value x2048 1.7734 ms/op 1.3935 ms/op 1.27
pickEth1Vote - Eth1Data fastSerialize tree x2048 15.154 ms/op 12.030 ms/op 1.26
bytes32 toHexString 1.2670 us/op 913.00 ns/op 1.39
bytes32 Buffer.toString(hex) 817.00 ns/op 743.00 ns/op 1.10
bytes32 Buffer.toString(hex) from Uint8Array 1.1690 us/op 973.00 ns/op 1.20
bytes32 Buffer.toString(hex) + 0x 807.00 ns/op 742.00 ns/op 1.09
Object access 1 prop 0.44800 ns/op 0.33600 ns/op 1.33
Map access 1 prop 0.35900 ns/op 0.29700 ns/op 1.21
Object get x1000 16.237 ns/op 10.833 ns/op 1.50
Map get x1000 1.0160 ns/op 0.92300 ns/op 1.10
Object set x1000 123.91 ns/op 72.108 ns/op 1.72
Map set x1000 89.762 ns/op 46.351 ns/op 1.94
Return object 10000 times 0.43640 ns/op 0.43220 ns/op 1.01
Throw Error 10000 times 8.4313 us/op 5.8250 us/op 1.45
fastMsgIdFn sha256 / 200 bytes 5.5030 us/op 4.8570 us/op 1.13
fastMsgIdFn h32 xxhash / 200 bytes 620.00 ns/op 582.00 ns/op 1.07
fastMsgIdFn h64 xxhash / 200 bytes 872.00 ns/op 755.00 ns/op 1.15
fastMsgIdFn sha256 / 1000 bytes 17.163 us/op 15.450 us/op 1.11
fastMsgIdFn h32 xxhash / 1000 bytes 842.00 ns/op 747.00 ns/op 1.13
fastMsgIdFn h64 xxhash / 1000 bytes 1.0330 us/op 850.00 ns/op 1.22
fastMsgIdFn sha256 / 10000 bytes 161.49 us/op 134.02 us/op 1.20
fastMsgIdFn h32 xxhash / 10000 bytes 2.8450 us/op 2.6040 us/op 1.09
fastMsgIdFn h64 xxhash / 10000 bytes 2.3010 us/op 1.8720 us/op 1.23
enrSubnets - fastDeserialize 64 bits 3.7090 us/op 2.8170 us/op 1.32
enrSubnets - ssz BitVector 64 bits 938.00 ns/op 824.00 ns/op 1.14
enrSubnets - fastDeserialize 4 bits 459.00 ns/op 419.00 ns/op 1.10
enrSubnets - ssz BitVector 4 bits 877.00 ns/op 828.00 ns/op 1.06
prioritizePeers score -10:0 att 32-0.1 sync 2-0 114.99 us/op 81.129 us/op 1.42
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 157.47 us/op 123.31 us/op 1.28
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 280.43 us/op 224.44 us/op 1.25
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 569.17 us/op 331.79 us/op 1.72
prioritizePeers score 0:0 att 64-1 sync 4-1 621.68 us/op 404.24 us/op 1.54
RateTracker 1000000 limit, 1 obj count per request 228.40 ns/op 173.20 ns/op 1.32
RateTracker 1000000 limit, 2 obj count per request 161.04 ns/op 127.29 ns/op 1.27
RateTracker 1000000 limit, 4 obj count per request 137.08 ns/op 103.78 ns/op 1.32
RateTracker 1000000 limit, 8 obj count per request 117.31 ns/op 93.085 ns/op 1.26
RateTracker with prune 6.2720 us/op 3.8140 us/op 1.64
array of 16000 items push then shift 5.3535 us/op 51.561 us/op 0.10
LinkedList of 16000 items push then shift 19.760 ns/op 12.116 ns/op 1.63
array of 16000 items push then pop 270.28 ns/op 178.91 ns/op 1.51
LinkedList of 16000 items push then pop 19.446 ns/op 11.761 ns/op 1.65
array of 24000 items push then shift 8.3535 us/op 77.344 us/op 0.11
LinkedList of 24000 items push then shift 20.313 ns/op 12.289 ns/op 1.65
array of 24000 items push then pop 239.54 ns/op 189.63 ns/op 1.26
LinkedList of 24000 items push then pop 20.574 ns/op 11.888 ns/op 1.73
intersect bitArray bitLen 8 12.765 ns/op 11.085 ns/op 1.15
intersect array and set length 8 201.23 ns/op 134.26 ns/op 1.50
intersect bitArray bitLen 128 69.631 ns/op 57.745 ns/op 1.21
intersect array and set length 128 2.4947 us/op 1.7251 us/op 1.45
Buffer.concat 32 items 2.8100 ns/op 1.7290 ns/op 1.63
pass gossip attestations to forkchoice per slot 4.7958 ms/op 4.8364 ms/op 0.99
computeDeltas 6.3424 ms/op 5.2666 ms/op 1.20
computeProposerBoostScoreFromBalances 905.34 us/op 803.82 us/op 1.13
altair processAttestation - 250000 vs - 7PWei normalcase 5.5425 ms/op 3.2591 ms/op 1.70
altair processAttestation - 250000 vs - 7PWei worstcase 7.9300 ms/op 5.8495 ms/op 1.36
altair processAttestation - setStatus - 1/6 committees join 247.61 us/op 185.86 us/op 1.33
altair processAttestation - setStatus - 1/3 committees join 482.60 us/op 361.51 us/op 1.33
altair processAttestation - setStatus - 1/2 committees join 666.23 us/op 516.13 us/op 1.29
altair processAttestation - setStatus - 2/3 committees join 982.91 us/op 673.70 us/op 1.46
altair processAttestation - setStatus - 4/5 committees join 1.2568 ms/op 938.56 us/op 1.34
altair processAttestation - setStatus - 100% committees join 1.4956 ms/op 1.1335 ms/op 1.32
altair processBlock - 250000 vs - 7PWei normalcase 32.078 ms/op 26.649 ms/op 1.20
altair processBlock - 250000 vs - 7PWei normalcase hashState 45.592 ms/op 37.832 ms/op 1.21
altair processBlock - 250000 vs - 7PWei worstcase 110.08 ms/op 80.516 ms/op 1.37
altair processBlock - 250000 vs - 7PWei worstcase hashState 127.91 ms/op 110.59 ms/op 1.16
phase0 processBlock - 250000 vs - 7PWei normalcase 4.4993 ms/op 3.0657 ms/op 1.47
phase0 processBlock - 250000 vs - 7PWei worstcase 64.014 ms/op 49.560 ms/op 1.29
altair processEth1Data - 250000 vs - 7PWei normalcase 1.0477 ms/op 684.07 us/op 1.53
Tree 40 250000 create 1.0420 s/op 674.42 ms/op 1.54
Tree 40 250000 get(125000) 319.96 ns/op 250.87 ns/op 1.28
Tree 40 250000 set(125000) 3.0825 us/op 2.0833 us/op 1.48
Tree 40 250000 toArray() 36.093 ms/op 24.799 ms/op 1.46
Tree 40 250000 iterate all - toArray() + loop 35.693 ms/op 27.138 ms/op 1.32
Tree 40 250000 iterate all - get(i) 132.86 ms/op 106.38 ms/op 1.25
MutableVector 250000 create 18.158 ms/op 12.413 ms/op 1.46
MutableVector 250000 get(125000) 14.935 ns/op 10.926 ns/op 1.37
MutableVector 250000 set(125000) 1.0252 us/op 635.34 ns/op 1.61
MutableVector 250000 toArray() 8.0702 ms/op 6.9032 ms/op 1.17
MutableVector 250000 iterate all - toArray() + loop 9.0897 ms/op 6.8582 ms/op 1.33
MutableVector 250000 iterate all - get(i) 3.8281 ms/op 2.5989 ms/op 1.47
Array 250000 create 7.1972 ms/op 5.3348 ms/op 1.35
Array 250000 clone - spread 3.2832 ms/op 2.4230 ms/op 1.36
Array 250000 get(125000) 1.4960 ns/op 1.1500 ns/op 1.30
Array 250000 set(125000) 1.5030 ns/op 1.1890 ns/op 1.26
Array 250000 iterate all - loop 149.48 us/op 151.86 us/op 0.98
effectiveBalanceIncrements clone Uint8Array 300000 100.98 us/op 34.458 us/op 2.93
effectiveBalanceIncrements clone MutableVector 300000 948.00 ns/op 717.00 ns/op 1.32
effectiveBalanceIncrements rw all Uint8Array 300000 307.23 us/op 247.52 us/op 1.24
effectiveBalanceIncrements rw all MutableVector 300000 263.83 ms/op 128.12 ms/op 2.06
phase0 afterProcessEpoch - 250000 vs - 7PWei 212.09 ms/op 188.91 ms/op 1.12
phase0 beforeProcessEpoch - 250000 vs - 7PWei 77.418 ms/op 58.772 ms/op 1.32
altair processEpoch - mainnet_e81889 662.58 ms/op 540.86 ms/op 1.23
mainnet_e81889 - altair beforeProcessEpoch 180.14 ms/op 128.62 ms/op 1.40
mainnet_e81889 - altair processJustificationAndFinalization 61.234 us/op 32.892 us/op 1.86
mainnet_e81889 - altair processInactivityUpdates 12.100 ms/op 8.5709 ms/op 1.41
mainnet_e81889 - altair processRewardsAndPenalties 108.62 ms/op 77.595 ms/op 1.40
mainnet_e81889 - altair processRegistryUpdates 13.359 us/op 3.9330 us/op 3.40
mainnet_e81889 - altair processSlashings 2.6690 us/op 1.2500 us/op 2.14
mainnet_e81889 - altair processEth1DataReset 3.2330 us/op 629.00 ns/op 5.14
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.6046 ms/op 2.0767 ms/op 1.25
mainnet_e81889 - altair processSlashingsReset 24.537 us/op 10.349 us/op 2.37
mainnet_e81889 - altair processRandaoMixesReset 23.395 us/op 10.158 us/op 2.30
mainnet_e81889 - altair processHistoricalRootsUpdate 3.6620 us/op 1.3000 us/op 2.82
mainnet_e81889 - altair processParticipationFlagUpdates 14.998 us/op 3.8500 us/op 3.90
mainnet_e81889 - altair processSyncCommitteeUpdates 3.4230 us/op 1.6680 us/op 2.05
mainnet_e81889 - altair afterProcessEpoch 219.43 ms/op 222.63 ms/op 0.99
phase0 processEpoch - mainnet_e58758 622.05 ms/op 529.85 ms/op 1.17
mainnet_e58758 - phase0 beforeProcessEpoch 297.31 ms/op 148.88 ms/op 2.00
mainnet_e58758 - phase0 processJustificationAndFinalization 61.397 us/op 29.177 us/op 2.10
mainnet_e58758 - phase0 processRewardsAndPenalties 154.17 ms/op 99.175 ms/op 1.55
mainnet_e58758 - phase0 processRegistryUpdates 31.917 us/op 8.3550 us/op 3.82
mainnet_e58758 - phase0 processSlashings 2.8170 us/op 639.00 ns/op 4.41
mainnet_e58758 - phase0 processEth1DataReset 3.1150 us/op 609.00 ns/op 5.11
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.4001 ms/op 1.5902 ms/op 1.51
mainnet_e58758 - phase0 processSlashingsReset 15.500 us/op 7.2920 us/op 2.13
mainnet_e58758 - phase0 processRandaoMixesReset 24.305 us/op 7.2890 us/op 3.33
mainnet_e58758 - phase0 processHistoricalRootsUpdate 3.9010 us/op 788.00 ns/op 4.95
mainnet_e58758 - phase0 processParticipationRecordUpdates 27.703 us/op 5.0370 us/op 5.50
mainnet_e58758 - phase0 afterProcessEpoch 173.60 ms/op 164.66 ms/op 1.05
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.3746 ms/op 2.0265 ms/op 1.17
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.7419 ms/op 2.4191 ms/op 1.13
altair processInactivityUpdates - 250000 normalcase 47.957 ms/op 39.049 ms/op 1.23
altair processInactivityUpdates - 250000 worstcase 59.199 ms/op 32.158 ms/op 1.84
phase0 processRegistryUpdates - 250000 normalcase 29.754 us/op 9.3630 us/op 3.18
phase0 processRegistryUpdates - 250000 badcase_full_deposits 521.73 us/op 373.19 us/op 1.40
phase0 processRegistryUpdates - 250000 worstcase 0.5 238.00 ms/op 194.84 ms/op 1.22
altair processRewardsAndPenalties - 250000 normalcase 149.60 ms/op 105.06 ms/op 1.42
altair processRewardsAndPenalties - 250000 worstcase 92.335 ms/op 71.916 ms/op 1.28
phase0 getAttestationDeltas - 250000 normalcase 15.806 ms/op 12.183 ms/op 1.30
phase0 getAttestationDeltas - 250000 worstcase 16.353 ms/op 12.318 ms/op 1.33
phase0 processSlashings - 250000 worstcase 6.7099 ms/op 5.0309 ms/op 1.33
altair processSyncCommitteeUpdates - 250000 366.51 ms/op 287.35 ms/op 1.28
BeaconState.hashTreeRoot - No change 591.00 ns/op 542.00 ns/op 1.09
BeaconState.hashTreeRoot - 1 full validator 74.269 us/op 73.036 us/op 1.02
BeaconState.hashTreeRoot - 32 full validator 753.44 us/op 820.60 us/op 0.92
BeaconState.hashTreeRoot - 512 full validator 8.2644 ms/op 6.8018 ms/op 1.22
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 107.64 us/op 86.762 us/op 1.24
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.5790 ms/op 1.2446 ms/op 1.27
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 20.112 ms/op 16.245 ms/op 1.24
BeaconState.hashTreeRoot - 1 balances 73.992 us/op 66.418 us/op 1.11
BeaconState.hashTreeRoot - 32 balances 723.53 us/op 618.47 us/op 1.17
BeaconState.hashTreeRoot - 512 balances 7.0372 ms/op 6.0450 ms/op 1.16
BeaconState.hashTreeRoot - 250000 balances 118.55 ms/op 100.90 ms/op 1.17
aggregationBits - 2048 els - zipIndexesInBitList 38.193 us/op 27.875 us/op 1.37
regular array get 100000 times 52.723 us/op 60.529 us/op 0.87
wrappedArray get 100000 times 52.281 us/op 60.604 us/op 0.86
arrayWithProxy get 100000 times 34.467 ms/op 28.302 ms/op 1.22
ssz.Root.equals 558.00 ns/op 557.00 ns/op 1.00
byteArrayEquals 561.00 ns/op 551.00 ns/op 1.02
shuffle list - 16384 els 11.655 ms/op 11.677 ms/op 1.00
shuffle list - 250000 els 166.33 ms/op 172.77 ms/op 0.96
processSlot - 1 slots 16.005 us/op 13.881 us/op 1.15
processSlot - 32 slots 2.1618 ms/op 1.9316 ms/op 1.12
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 445.50 us/op 355.75 us/op 1.25
getCommitteeAssignments - req 1 vs - 250000 vc 5.5412 ms/op 5.4674 ms/op 1.01
getCommitteeAssignments - req 100 vs - 250000 vc 7.5502 ms/op 7.9894 ms/op 0.95
getCommitteeAssignments - req 1000 vs - 250000 vc 7.9917 ms/op 8.5571 ms/op 0.93
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 10.350 ns/op 8.1500 ns/op 1.27
state getBlockRootAtSlot - 250000 vs - 7PWei 1.2313 us/op 1.0509 us/op 1.17
computeProposers - vc 250000 20.625 ms/op 16.644 ms/op 1.24
computeEpochShuffling - vc 250000 188.27 ms/op 173.57 ms/op 1.08
getNextSyncCommittee - vc 250000 365.85 ms/op 277.41 ms/op 1.32

by benchmarkbot/action

@g11tech g11tech mentioned this pull request Oct 31, 2022
Closed
6 tasks
@g11tech g11tech marked this pull request as ready for review October 31, 2022 16:58
@g11tech g11tech requested a review from a team as a code owner October 31, 2022 16:58
@wemeetagain wemeetagain merged commit 2ec8a0c into unstable Nov 1, 2022
@wemeetagain wemeetagain deleted the g11tech/mergemock-builder branch November 1, 2022 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Mergemock builder integration tests
2 participants
@g11tech @wemeetagain