Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mock validator client #4932

Closed
wants to merge 7 commits into from
Closed

Mock validator client #4932

wants to merge 7 commits into from

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Dec 22, 2022

Motivation

We want to track the I/O lag issue statically

Description

  • Write a mock vc that produce attestation right at 1/3 of slot
  • New sub-command, for example validator mock --beaconNode http://localhost:9596 --network goerli --metrics true

part of #4002, #4881

@github-actions
Copy link
Contributor

github-actions bot commented Dec 22, 2022 •
edited
Loading

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: 3edf84c Previous: 3166e0e Ratio
altair processAttestation - 250000 vs - 7PWei worstcase 18.839 ms/op 5.5605 ms/op 3.39
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 24.684 us/op 7.7200 us/op 3.20
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 23.627 us/op 7.4800 us/op 3.16
Full benchmark results
Benchmark suite Current: 3edf84c Previous: 3166e0e Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.4232 ms/op 1.1474 ms/op 1.24
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 88.968 us/op 65.694 us/op 1.35
BLS verify - blst-native 2.3819 ms/op 2.1760 ms/op 1.09
BLS verifyMultipleSignatures 3 - blst-native 5.0420 ms/op 4.4877 ms/op 1.12
BLS verifyMultipleSignatures 8 - blst-native 11.092 ms/op 9.7009 ms/op 1.14
BLS verifyMultipleSignatures 32 - blst-native 38.716 ms/op 35.257 ms/op 1.10
BLS aggregatePubkeys 32 - blst-native 52.822 us/op 46.401 us/op 1.14
BLS aggregatePubkeys 128 - blst-native 206.18 us/op 182.10 us/op 1.13
getAttestationsForBlock 121.10 ms/op 80.339 ms/op 1.51
isKnown best case - 1 super set check 544.00 ns/op 450.00 ns/op 1.21
isKnown normal case - 2 super set checks 526.00 ns/op 443.00 ns/op 1.19
isKnown worse case - 16 super set checks 512.00 ns/op 441.00 ns/op 1.16
CheckpointStateCache - add get delete 10.605 us/op 8.4450 us/op 1.26
validate gossip signedAggregateAndProof - struct 5.4916 ms/op 5.0149 ms/op 1.10
validate gossip attestation - struct 2.7004 ms/op 2.3595 ms/op 1.14
pickEth1Vote - no votes 2.7458 ms/op 2.1804 ms/op 1.26
pickEth1Vote - max votes 20.659 ms/op 17.142 ms/op 1.21
pickEth1Vote - Eth1Data hashTreeRoot value x2048 16.735 ms/op 13.217 ms/op 1.27
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 27.000 ms/op 21.441 ms/op 1.26
pickEth1Vote - Eth1Data fastSerialize value x2048 1.5811 ms/op 1.1270 ms/op 1.40
pickEth1Vote - Eth1Data fastSerialize tree x2048 12.083 ms/op 10.964 ms/op 1.10
bytes32 toHexString 1.2640 us/op 985.00 ns/op 1.28
bytes32 Buffer.toString(hex) 691.00 ns/op 553.00 ns/op 1.25
bytes32 Buffer.toString(hex) from Uint8Array 1.0980 us/op 881.00 ns/op 1.25
bytes32 Buffer.toString(hex) + 0x 698.00 ns/op 573.00 ns/op 1.22
Object access 1 prop 0.34400 ns/op 0.28500 ns/op 1.21
Map access 1 prop 0.34600 ns/op 0.26700 ns/op 1.30
Object get x1000 16.734 ns/op 10.560 ns/op 1.58
Map get x1000 1.0390 ns/op 0.95600 ns/op 1.09
Object set x1000 118.80 ns/op 81.041 ns/op 1.47
Map set x1000 81.659 ns/op 49.491 ns/op 1.65
Return object 10000 times 0.45440 ns/op 0.43610 ns/op 1.04
Throw Error 10000 times 8.2523 us/op 6.5713 us/op 1.26
fastMsgIdFn sha256 / 200 bytes 5.5430 us/op 4.8630 us/op 1.14
fastMsgIdFn h32 xxhash / 200 bytes 595.00 ns/op 462.00 ns/op 1.29
fastMsgIdFn h64 xxhash / 200 bytes 824.00 ns/op 655.00 ns/op 1.26
fastMsgIdFn sha256 / 1000 bytes 17.518 us/op 15.744 us/op 1.11
fastMsgIdFn h32 xxhash / 1000 bytes 760.00 ns/op 635.00 ns/op 1.20
fastMsgIdFn h64 xxhash / 1000 bytes 916.00 ns/op 757.00 ns/op 1.21
fastMsgIdFn sha256 / 10000 bytes 154.50 us/op 137.33 us/op 1.13
fastMsgIdFn h32 xxhash / 10000 bytes 2.9500 us/op 2.5560 us/op 1.15
fastMsgIdFn h64 xxhash / 10000 bytes 2.1570 us/op 1.7440 us/op 1.24
enrSubnets - fastDeserialize 64 bits 2.9380 us/op 2.3900 us/op 1.23
enrSubnets - ssz BitVector 64 bits 1.0070 us/op 847.00 ns/op 1.19
enrSubnets - fastDeserialize 4 bits 364.00 ns/op 294.00 ns/op 1.24
enrSubnets - ssz BitVector 4 bits 1.0200 us/op 846.00 ns/op 1.21
prioritizePeers score -10:0 att 32-0.1 sync 2-0 112.64 us/op 80.375 us/op 1.40
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 159.49 us/op 112.73 us/op 1.41
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 290.45 us/op 215.83 us/op 1.35
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 490.45 us/op 321.71 us/op 1.52
prioritizePeers score 0:0 att 64-1 sync 4-1 608.46 us/op 407.13 us/op 1.49
array of 16000 items push then shift 5.8199 us/op 51.658 us/op 0.11
LinkedList of 16000 items push then shift 21.606 ns/op 12.647 ns/op 1.71
array of 16000 items push then pop 257.68 ns/op 191.12 ns/op 1.35
LinkedList of 16000 items push then pop 19.566 ns/op 12.193 ns/op 1.60
array of 24000 items push then shift 8.4884 us/op 77.419 us/op 0.11
LinkedList of 24000 items push then shift 21.274 ns/op 13.515 ns/op 1.57
array of 24000 items push then pop 276.31 ns/op 201.28 ns/op 1.37
LinkedList of 24000 items push then pop 20.127 ns/op 12.287 ns/op 1.64
intersect bitArray bitLen 8 30.675 ns/op 21.701 ns/op 1.41
intersect array and set length 8 209.97 ns/op 138.83 ns/op 1.51
intersect bitArray bitLen 128 102.12 ns/op 74.920 ns/op 1.36
intersect array and set length 128 2.3282 us/op 1.7435 us/op 1.34
Buffer.concat 32 items 6.3040 us/op 4.4880 us/op 1.40
Uint8Array.set 32 items 5.2470 us/op 3.7880 us/op 1.39
pass gossip attestations to forkchoice per slot 7.5409 ms/op 3.7701 ms/op 2.00
computeDeltas 5.8499 ms/op 4.6423 ms/op 1.26
computeProposerBoostScoreFromBalances 2.9219 ms/op 2.4899 ms/op 1.17
altair processAttestation - 250000 vs - 7PWei normalcase 5.1519 ms/op 3.7448 ms/op 1.38
altair processAttestation - 250000 vs - 7PWei worstcase 18.839 ms/op 5.5605 ms/op 3.39
altair processAttestation - setStatus - 1/6 committees join 269.49 us/op 179.18 us/op 1.50
altair processAttestation - setStatus - 1/3 committees join 497.13 us/op 353.96 us/op 1.40
altair processAttestation - setStatus - 1/2 committees join 692.16 us/op 514.03 us/op 1.35
altair processAttestation - setStatus - 2/3 committees join 953.36 us/op 673.38 us/op 1.42
altair processAttestation - setStatus - 4/5 committees join 1.2811 ms/op 936.13 us/op 1.37
altair processAttestation - setStatus - 100% committees join 1.5736 ms/op 1.1290 ms/op 1.39
altair processBlock - 250000 vs - 7PWei normalcase 32.989 ms/op 23.561 ms/op 1.40
altair processBlock - 250000 vs - 7PWei normalcase hashState 51.809 ms/op 32.401 ms/op 1.60
altair processBlock - 250000 vs - 7PWei worstcase 96.633 ms/op 78.918 ms/op 1.22
altair processBlock - 250000 vs - 7PWei worstcase hashState 133.62 ms/op 112.59 ms/op 1.19
phase0 processBlock - 250000 vs - 7PWei normalcase 4.5001 ms/op 3.1431 ms/op 1.43
phase0 processBlock - 250000 vs - 7PWei worstcase 56.093 ms/op 50.990 ms/op 1.10
altair processEth1Data - 250000 vs - 7PWei normalcase 1.0050 ms/op 672.05 us/op 1.50
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 24.684 us/op 7.7200 us/op 3.20
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 47.317 us/op 22.224 us/op 2.13
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 28.812 us/op 10.275 us/op 2.80
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 23.627 us/op 7.4800 us/op 3.16
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 145.21 us/op 94.048 us/op 1.54
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.2514 ms/op 862.13 us/op 1.45
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.6319 ms/op 1.1719 ms/op 1.39
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.6061 ms/op 1.1169 ms/op 1.44
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 5.4607 ms/op 4.0308 ms/op 1.35
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 3.6822 ms/op 2.4195 ms/op 1.52
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 9.2439 ms/op 7.1178 ms/op 1.30
Tree 40 250000 create 858.39 ms/op 595.36 ms/op 1.44
Tree 40 250000 get(125000) 331.99 ns/op 228.03 ns/op 1.46
Tree 40 250000 set(125000) 2.8875 us/op 1.9180 us/op 1.51
Tree 40 250000 toArray() 36.861 ms/op 26.112 ms/op 1.41
Tree 40 250000 iterate all - toArray() + loop 36.515 ms/op 26.631 ms/op 1.37
Tree 40 250000 iterate all - get(i) 134.64 ms/op 109.08 ms/op 1.23
MutableVector 250000 create 17.378 ms/op 14.579 ms/op 1.19
MutableVector 250000 get(125000) 14.660 ns/op 10.547 ns/op 1.39
MutableVector 250000 set(125000) 716.38 ns/op 510.15 ns/op 1.40
MutableVector 250000 toArray() 7.3219 ms/op 5.6667 ms/op 1.29
MutableVector 250000 iterate all - toArray() + loop 7.7560 ms/op 5.7573 ms/op 1.35
MutableVector 250000 iterate all - get(i) 3.9523 ms/op 3.0103 ms/op 1.31
Array 250000 create 7.2598 ms/op 5.7006 ms/op 1.27
Array 250000 clone - spread 3.1972 ms/op 2.5363 ms/op 1.26
Array 250000 get(125000) 1.4040 ns/op 1.1520 ns/op 1.22
Array 250000 set(125000) 1.3540 ns/op 1.1840 ns/op 1.14
Array 250000 iterate all - loop 149.50 us/op 150.90 us/op 0.99
effectiveBalanceIncrements clone Uint8Array 300000 93.595 us/op 42.223 us/op 2.22
effectiveBalanceIncrements clone MutableVector 300000 983.00 ns/op 781.00 ns/op 1.26
effectiveBalanceIncrements rw all Uint8Array 300000 306.34 us/op 243.99 us/op 1.26
effectiveBalanceIncrements rw all MutableVector 300000 228.84 ms/op 153.82 ms/op 1.49
phase0 afterProcessEpoch - 250000 vs - 7PWei 209.92 ms/op 187.65 ms/op 1.12
phase0 beforeProcessEpoch - 250000 vs - 7PWei 79.825 ms/op 65.702 ms/op 1.21
altair processEpoch - mainnet_e81889 668.77 ms/op 543.53 ms/op 1.23
mainnet_e81889 - altair beforeProcessEpoch 171.68 ms/op 76.274 ms/op 2.25
mainnet_e81889 - altair processJustificationAndFinalization 73.020 us/op 15.744 us/op 4.64
mainnet_e81889 - altair processInactivityUpdates 11.464 ms/op 9.0205 ms/op 1.27
mainnet_e81889 - altair processRewardsAndPenalties 169.82 ms/op 139.37 ms/op 1.22
mainnet_e81889 - altair processRegistryUpdates 12.513 us/op 2.1400 us/op 5.85
mainnet_e81889 - altair processSlashings 2.9800 us/op 471.00 ns/op 6.33
mainnet_e81889 - altair processEth1DataReset 3.1810 us/op 497.00 ns/op 6.40
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.8039 ms/op 2.2366 ms/op 1.25
mainnet_e81889 - altair processSlashingsReset 20.515 us/op 4.4640 us/op 4.60
mainnet_e81889 - altair processRandaoMixesReset 25.304 us/op 3.8330 us/op 6.60
mainnet_e81889 - altair processHistoricalRootsUpdate 3.8950 us/op 580.00 ns/op 6.72
mainnet_e81889 - altair processParticipationFlagUpdates 11.643 us/op 1.9780 us/op 5.89
mainnet_e81889 - altair processSyncCommitteeUpdates 3.2500 us/op 582.00 ns/op 5.58
mainnet_e81889 - altair afterProcessEpoch 226.48 ms/op 195.95 ms/op 1.16
phase0 processEpoch - mainnet_e58758 699.97 ms/op 533.45 ms/op 1.31
mainnet_e58758 - phase0 beforeProcessEpoch 239.51 ms/op 153.01 ms/op 1.57
mainnet_e58758 - phase0 processJustificationAndFinalization 62.789 us/op 16.953 us/op 3.70
mainnet_e58758 - phase0 processRewardsAndPenalties 142.91 ms/op 106.53 ms/op 1.34
mainnet_e58758 - phase0 processRegistryUpdates 32.708 us/op 7.9260 us/op 4.13
mainnet_e58758 - phase0 processSlashings 2.6940 us/op 552.00 ns/op 4.88
mainnet_e58758 - phase0 processEth1DataReset 3.0700 us/op 538.00 ns/op 5.71
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.2287 ms/op 1.5859 ms/op 1.41
mainnet_e58758 - phase0 processSlashingsReset 17.640 us/op 3.2870 us/op 5.37
mainnet_e58758 - phase0 processRandaoMixesReset 24.729 us/op 4.1100 us/op 6.02
mainnet_e58758 - phase0 processHistoricalRootsUpdate 3.5520 us/op 573.00 ns/op 6.20
mainnet_e58758 - phase0 processParticipationRecordUpdates 22.232 us/op 3.2060 us/op 6.93
mainnet_e58758 - phase0 afterProcessEpoch 183.57 ms/op 162.37 ms/op 1.13
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.4784 ms/op 1.9309 ms/op 1.28
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.7032 ms/op 2.1987 ms/op 1.23
altair processInactivityUpdates - 250000 normalcase 53.534 ms/op 39.728 ms/op 1.35
altair processInactivityUpdates - 250000 worstcase 51.531 ms/op 32.353 ms/op 1.59
phase0 processRegistryUpdates - 250000 normalcase 28.981 us/op 6.1850 us/op 4.69
phase0 processRegistryUpdates - 250000 badcase_full_deposits 532.98 us/op 390.80 us/op 1.36
phase0 processRegistryUpdates - 250000 worstcase 0.5 258.86 ms/op 175.43 ms/op 1.48
altair processRewardsAndPenalties - 250000 normalcase 133.57 ms/op 75.867 ms/op 1.76
altair processRewardsAndPenalties - 250000 worstcase 160.97 ms/op 105.34 ms/op 1.53
phase0 getAttestationDeltas - 250000 normalcase 13.600 ms/op 11.756 ms/op 1.16
phase0 getAttestationDeltas - 250000 worstcase 13.489 ms/op 11.955 ms/op 1.13
phase0 processSlashings - 250000 worstcase 7.0750 ms/op 5.1644 ms/op 1.37
altair processSyncCommitteeUpdates - 250000 333.56 ms/op 295.59 ms/op 1.13
BeaconState.hashTreeRoot - No change 591.00 ns/op 486.00 ns/op 1.22
BeaconState.hashTreeRoot - 1 full validator 80.270 us/op 66.036 us/op 1.22
BeaconState.hashTreeRoot - 32 full validator 776.35 us/op 744.07 us/op 1.04
BeaconState.hashTreeRoot - 512 full validator 7.9854 ms/op 6.8929 ms/op 1.16
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 108.16 us/op 86.674 us/op 1.25
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.5627 ms/op 1.2550 ms/op 1.25
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 20.216 ms/op 16.564 ms/op 1.22
BeaconState.hashTreeRoot - 1 balances 80.598 us/op 69.918 us/op 1.15
BeaconState.hashTreeRoot - 32 balances 691.63 us/op 634.76 us/op 1.09
BeaconState.hashTreeRoot - 512 balances 7.3763 ms/op 6.2491 ms/op 1.18
BeaconState.hashTreeRoot - 250000 balances 123.64 ms/op 109.64 ms/op 1.13
aggregationBits - 2048 els - zipIndexesInBitList 28.678 us/op 23.113 us/op 1.24
regular array get 100000 times 59.791 us/op 60.503 us/op 0.99
wrappedArray get 100000 times 59.919 us/op 60.524 us/op 0.99
arrayWithProxy get 100000 times 35.724 ms/op 26.172 ms/op 1.36
ssz.Root.equals 1.1990 us/op 907.00 ns/op 1.32
byteArrayEquals 1.1920 us/op 892.00 ns/op 1.34
shuffle list - 16384 els 12.119 ms/op 11.455 ms/op 1.06
shuffle list - 250000 els 180.85 ms/op 168.07 ms/op 1.08
processSlot - 1 slots 17.339 us/op 13.679 us/op 1.27
processSlot - 32 slots 2.3963 ms/op 1.9464 ms/op 1.23
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 399.43 us/op 315.67 us/op 1.27
getCommitteeAssignments - req 1 vs - 250000 vc 5.9682 ms/op 5.4352 ms/op 1.10
getCommitteeAssignments - req 100 vs - 250000 vc 8.4028 ms/op 7.9563 ms/op 1.06
getCommitteeAssignments - req 1000 vs - 250000 vc 8.8858 ms/op 8.4955 ms/op 1.05
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 9.3300 ns/op 7.5800 ns/op 1.23
state getBlockRootAtSlot - 250000 vs - 7PWei 1.4506 us/op 1.1514 us/op 1.26
computeProposers - vc 250000 20.726 ms/op 17.733 ms/op 1.17
computeEpochShuffling - vc 250000 187.23 ms/op 172.19 ms/op 1.09
getNextSyncCommittee - vc 250000 327.88 ms/op 293.38 ms/op 1.12

by benchmarkbot/action

@dapplion
Copy link
Contributor

@tuyennhv Can you check if this tool already serves your needs? https://github.com/michaelsproul/blockdreamer

dapplion
dapplion suggested changes Dec 22, 2022
View reviewed changes
Copy link
Contributor

@dapplion dapplion left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to not merge such a niche tool an our production CLI, specially if it shows in --help. Please develop as a standalone repo or just use blockdreamer

@twoeths
Copy link
Contributor Author

twoeths commented Dec 23, 2022

I would prefer to not merge such a niche tool an our production CLI, specially if it shows in --help. Please develop as a standalone repo or just use blockdreamer

@dapplion it's a hidden subcommand so will not be shown in --help, we want to issue produceAttestationData right at 1/3 of slot (the busiest time of our node) so cannot use blockdreamer. Moreover we want to utilize our api client metric to measure the http call time and show it in our grafana dashboard.

I'd keep this branch to see how it work first, will create a repo when we think we really need to.

@twoeths twoeths closed this Dec 23, 2022
@twoeths twoeths deleted the tuyen/mock-vc branch December 18, 2023 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@twoeths @dapplion