Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge single and multi block processors #4317

Merged
merged 1 commit into from
Jul 17, 2022

Conversation

dapplion
Copy link
Contributor

Motivation

Description

Merge single and multi block processors

  • Equivalent logic
  • Moves options from wrapper block type to an opts object
  • Drops ChainSegmentError. Once blocks are verified in batched it would not be trivial to know if a previous block was actually valid. Since this is just an optimization for rare cases, not worth the engineering effort now.

@dapplion dapplion requested a review from a team as a code owner July 17, 2022 15:24
}
} catch (e) {
// above functions should only throw BlockError
const err = getBlockError(e, blocks[0]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the block error be for the block that errored?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After completing this work everything is processed in batch so you don't know which block caused the error. Each function should take care of adding the right block to BlockError which is already happening, but here, at this line you have no clue which block caused this. blocks[0] is a placeholder, it shouldn't really happen. In future PRs we can review what to do in this case

if (res.err instanceof ChainSegmentError && res.err.importedBlocks > 0) {
this.advanceChain(batch.startEpoch);
}
// TODO: Disabled for now
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make an issue for this? I guess it's a small thing, an optimization to avoid reprocessing up to batch.length - 1 blocks.

Copy link
Contributor Author

@dapplion dapplion Jul 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do an issue but I don't see this ever being implemented judging for our priorities in the last 2 years. i.e. to help process sync we have tens higher priority items to go after

@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: de2ec33 Previous: ab13fe2 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 3.1012 ms/op 2.2205 ms/op 1.40
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 90.053 us/op 78.474 us/op 1.15
BLS verify - blst-native 2.2475 ms/op 2.5544 ms/op 0.88
BLS verifyMultipleSignatures 3 - blst-native 4.6309 ms/op 5.2227 ms/op 0.89
BLS verifyMultipleSignatures 8 - blst-native 10.015 ms/op 11.260 ms/op 0.89
BLS verifyMultipleSignatures 32 - blst-native 36.341 ms/op 42.163 ms/op 0.86
BLS aggregatePubkeys 32 - blst-native 49.808 us/op 56.362 us/op 0.88
BLS aggregatePubkeys 128 - blst-native 191.60 us/op 226.71 us/op 0.85
getAttestationsForBlock 59.737 ms/op 50.749 ms/op 1.18
isKnown best case - 1 super set check 634.00 ns/op 475.00 ns/op 1.33
isKnown normal case - 2 super set checks 625.00 ns/op 486.00 ns/op 1.29
isKnown worse case - 16 super set checks 634.00 ns/op 459.00 ns/op 1.38
CheckpointStateCache - add get delete 15.162 us/op 13.697 us/op 1.11
validate gossip signedAggregateAndProof - struct 5.2350 ms/op 6.1618 ms/op 0.85
validate gossip attestation - struct 2.5131 ms/op 2.7679 ms/op 0.91
altair verifyImport mainnet_s3766816:31 13.484 s/op 15.135 s/op 0.89
pickEth1Vote - no votes 2.5795 ms/op 2.3981 ms/op 1.08
pickEth1Vote - max votes 34.183 ms/op 26.625 ms/op 1.28
pickEth1Vote - Eth1Data hashTreeRoot value x2048 16.455 ms/op 14.733 ms/op 1.12
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 30.562 ms/op 23.421 ms/op 1.30
pickEth1Vote - Eth1Data fastSerialize value x2048 2.0524 ms/op 1.7248 ms/op 1.19
pickEth1Vote - Eth1Data fastSerialize tree x2048 24.502 ms/op 18.158 ms/op 1.35
bytes32 toHexString 1.5050 us/op 1.2440 us/op 1.21
bytes32 Buffer.toString(hex) 882.00 ns/op 844.00 ns/op 1.05
bytes32 Buffer.toString(hex) from Uint8Array 1.2560 us/op 1.0830 us/op 1.16
bytes32 Buffer.toString(hex) + 0x 896.00 ns/op 755.00 ns/op 1.19
Object access 1 prop 0.50900 ns/op 0.39300 ns/op 1.30
Map access 1 prop 0.35600 ns/op 0.35800 ns/op 0.99
Object get x1000 17.151 ns/op 15.396 ns/op 1.11
Map get x1000 0.99400 ns/op 0.93400 ns/op 1.06
Object set x1000 148.35 ns/op 117.20 ns/op 1.27
Map set x1000 100.83 ns/op 77.327 ns/op 1.30
Return object 10000 times 0.44700 ns/op 0.40360 ns/op 1.11
Throw Error 10000 times 7.4379 us/op 6.6512 us/op 1.12
enrSubnets - fastDeserialize 64 bits 3.9710 us/op 3.0350 us/op 1.31
enrSubnets - ssz BitVector 64 bits 963.00 ns/op 854.00 ns/op 1.13
enrSubnets - fastDeserialize 4 bits 527.00 ns/op 455.00 ns/op 1.16
enrSubnets - ssz BitVector 4 bits 893.00 ns/op 970.00 ns/op 0.92
prioritizePeers score -10:0 att 32-0.1 sync 2-0 127.56 us/op 110.06 us/op 1.16
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 159.53 us/op 149.16 us/op 1.07
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 319.71 us/op 276.54 us/op 1.16
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 676.91 us/op 589.30 us/op 1.15
prioritizePeers score 0:0 att 64-1 sync 4-1 576.01 us/op 607.95 us/op 0.95
RateTracker 1000000 limit, 1 obj count per request 223.59 ns/op 203.90 ns/op 1.10
RateTracker 1000000 limit, 2 obj count per request 171.46 ns/op 153.64 ns/op 1.12
RateTracker 1000000 limit, 4 obj count per request 144.30 ns/op 125.71 ns/op 1.15
RateTracker 1000000 limit, 8 obj count per request 128.72 ns/op 112.22 ns/op 1.15
RateTracker with prune 6.8080 us/op 5.2430 us/op 1.30
array of 16000 items push then shift 5.3970 us/op 4.8624 us/op 1.11
LinkedList of 16000 items push then shift 31.512 ns/op 27.156 ns/op 1.16
array of 16000 items push then pop 307.01 ns/op 260.32 ns/op 1.18
LinkedList of 16000 items push then pop 26.536 ns/op 24.605 ns/op 1.08
array of 24000 items push then shift 7.9209 us/op 7.5690 us/op 1.05
LinkedList of 24000 items push then shift 30.916 ns/op 27.061 ns/op 1.14
array of 24000 items push then pop 244.31 ns/op 202.84 ns/op 1.20
LinkedList of 24000 items push then pop 26.310 ns/op 24.054 ns/op 1.09
intersect bitArray bitLen 8 16.807 ns/op 11.714 ns/op 1.43
intersect array and set length 8 345.56 ns/op 184.57 ns/op 1.87
intersect bitArray bitLen 128 73.213 ns/op 65.020 ns/op 1.13
intersect array and set length 128 4.2631 us/op 2.2439 us/op 1.90
pass gossip attestations to forkchoice per slot 3.6687 ms/op 3.2677 ms/op 1.12
computeDeltas 3.7599 ms/op 3.4918 ms/op 1.08
computeProposerBoostScoreFromBalances 907.99 us/op 821.45 us/op 1.11
altair processAttestation - 250000 vs - 7PWei normalcase 6.3079 ms/op 4.8179 ms/op 1.31
altair processAttestation - 250000 vs - 7PWei worstcase 8.5864 ms/op 6.7156 ms/op 1.28
altair processAttestation - setStatus - 1/6 committees join 261.97 us/op 235.36 us/op 1.11
altair processAttestation - setStatus - 1/3 committees join 533.01 us/op 445.07 us/op 1.20
altair processAttestation - setStatus - 1/2 committees join 738.13 us/op 589.87 us/op 1.25
altair processAttestation - setStatus - 2/3 committees join 974.25 us/op 832.72 us/op 1.17
altair processAttestation - setStatus - 4/5 committees join 1.3117 ms/op 1.1005 ms/op 1.19
altair processAttestation - setStatus - 100% committees join 1.5860 ms/op 1.3909 ms/op 1.14
altair processBlock - 250000 vs - 7PWei normalcase 32.170 ms/op 28.939 ms/op 1.11
altair processBlock - 250000 vs - 7PWei normalcase hashState 49.093 ms/op 38.852 ms/op 1.26
altair processBlock - 250000 vs - 7PWei worstcase 98.113 ms/op 108.05 ms/op 0.91
altair processBlock - 250000 vs - 7PWei worstcase hashState 122.72 ms/op 119.25 ms/op 1.03
phase0 processBlock - 250000 vs - 7PWei normalcase 5.9571 ms/op 4.9435 ms/op 1.21
phase0 processBlock - 250000 vs - 7PWei worstcase 68.197 ms/op 62.742 ms/op 1.09
altair processEth1Data - 250000 vs - 7PWei normalcase 1.0083 ms/op 1.0197 ms/op 0.99
Tree 40 250000 create 1.0469 s/op 931.41 ms/op 1.12
Tree 40 250000 get(125000) 333.34 ns/op 309.54 ns/op 1.08
Tree 40 250000 set(125000) 3.5825 us/op 2.7916 us/op 1.28
Tree 40 250000 toArray() 38.392 ms/op 33.683 ms/op 1.14
Tree 40 250000 iterate all - toArray() + loop 39.056 ms/op 34.972 ms/op 1.12
Tree 40 250000 iterate all - get(i) 136.40 ms/op 123.81 ms/op 1.10
MutableVector 250000 create 19.461 ms/op 19.003 ms/op 1.02
MutableVector 250000 get(125000) 14.713 ns/op 13.281 ns/op 1.11
MutableVector 250000 set(125000) 1.0521 us/op 793.62 ns/op 1.33
MutableVector 250000 toArray() 8.6394 ms/op 6.5368 ms/op 1.32
MutableVector 250000 iterate all - toArray() + loop 8.5964 ms/op 6.5089 ms/op 1.32
MutableVector 250000 iterate all - get(i) 3.5154 ms/op 3.6781 ms/op 0.96
Array 250000 create 7.6001 ms/op 6.2117 ms/op 1.22
Array 250000 clone - spread 4.6641 ms/op 4.0908 ms/op 1.14
Array 250000 get(125000) 1.9660 ns/op 1.7480 ns/op 1.12
Array 250000 set(125000) 2.1850 ns/op 1.7760 ns/op 1.23
Array 250000 iterate all - loop 147.99 us/op 137.18 us/op 1.08
effectiveBalanceIncrements clone Uint8Array 300000 202.29 us/op 114.81 us/op 1.76
effectiveBalanceIncrements clone MutableVector 300000 750.00 ns/op 1.4520 us/op 0.52
effectiveBalanceIncrements rw all Uint8Array 300000 296.84 us/op 273.51 us/op 1.09
effectiveBalanceIncrements rw all MutableVector 300000 241.52 ms/op 276.44 ms/op 0.87
phase0 afterProcessEpoch - 250000 vs - 7PWei 220.67 ms/op 199.77 ms/op 1.10
phase0 beforeProcessEpoch - 250000 vs - 7PWei 83.979 ms/op 109.68 ms/op 0.77
altair processEpoch - mainnet_e81889 691.41 ms/op 614.99 ms/op 1.12
mainnet_e81889 - altair beforeProcessEpoch 220.32 ms/op 157.42 ms/op 1.40
mainnet_e81889 - altair processJustificationAndFinalization 69.322 us/op 77.403 us/op 0.90
mainnet_e81889 - altair processInactivityUpdates 12.322 ms/op 11.075 ms/op 1.11
mainnet_e81889 - altair processRewardsAndPenalties 110.29 ms/op 100.12 ms/op 1.10
mainnet_e81889 - altair processRegistryUpdates 15.843 us/op 16.973 us/op 0.93
mainnet_e81889 - altair processSlashings 4.5830 us/op 5.2260 us/op 0.88
mainnet_e81889 - altair processEth1DataReset 4.3850 us/op 4.6090 us/op 0.95
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.4835 ms/op 2.7005 ms/op 0.92
mainnet_e81889 - altair processSlashingsReset 27.981 us/op 32.134 us/op 0.87
mainnet_e81889 - altair processRandaoMixesReset 25.678 us/op 26.451 us/op 0.97
mainnet_e81889 - altair processHistoricalRootsUpdate 4.1150 us/op 5.9120 us/op 0.70
mainnet_e81889 - altair processParticipationFlagUpdates 15.357 us/op 16.323 us/op 0.94
mainnet_e81889 - altair processSyncCommitteeUpdates 3.5160 us/op 3.7220 us/op 0.94
mainnet_e81889 - altair afterProcessEpoch 218.74 ms/op 198.67 ms/op 1.10
phase0 processEpoch - mainnet_e58758 644.82 ms/op 588.90 ms/op 1.09
mainnet_e58758 - phase0 beforeProcessEpoch 301.64 ms/op 281.85 ms/op 1.07
mainnet_e58758 - phase0 processJustificationAndFinalization 64.770 us/op 69.622 us/op 0.93
mainnet_e58758 - phase0 processRewardsAndPenalties 171.59 ms/op 131.41 ms/op 1.31
mainnet_e58758 - phase0 processRegistryUpdates 33.661 us/op 31.223 us/op 1.08
mainnet_e58758 - phase0 processSlashings 3.8420 us/op 3.7560 us/op 1.02
mainnet_e58758 - phase0 processEth1DataReset 3.6400 us/op 4.0870 us/op 0.89
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 3.0249 ms/op 2.4019 ms/op 1.26
mainnet_e58758 - phase0 processSlashingsReset 19.000 us/op 21.063 us/op 0.90
mainnet_e58758 - phase0 processRandaoMixesReset 24.756 us/op 26.925 us/op 0.92
mainnet_e58758 - phase0 processHistoricalRootsUpdate 4.1020 us/op 5.3820 us/op 0.76
mainnet_e58758 - phase0 processParticipationRecordUpdates 23.625 us/op 25.138 us/op 0.94
mainnet_e58758 - phase0 afterProcessEpoch 185.30 ms/op 158.22 ms/op 1.17
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.6332 ms/op 2.3317 ms/op 1.13
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.8575 ms/op 2.6541 ms/op 1.08
altair processInactivityUpdates - 250000 normalcase 56.591 ms/op 46.754 ms/op 1.21
altair processInactivityUpdates - 250000 worstcase 66.648 ms/op 56.438 ms/op 1.18
phase0 processRegistryUpdates - 250000 normalcase 27.815 us/op 28.684 us/op 0.97
phase0 processRegistryUpdates - 250000 badcase_full_deposits 516.29 us/op 502.76 us/op 1.03
phase0 processRegistryUpdates - 250000 worstcase 0.5 259.94 ms/op 231.51 ms/op 1.12
altair processRewardsAndPenalties - 250000 normalcase 158.05 ms/op 140.26 ms/op 1.13
altair processRewardsAndPenalties - 250000 worstcase 102.43 ms/op 115.16 ms/op 0.89
phase0 getAttestationDeltas - 250000 normalcase 16.606 ms/op 13.789 ms/op 1.20
phase0 getAttestationDeltas - 250000 worstcase 16.265 ms/op 13.896 ms/op 1.17
phase0 processSlashings - 250000 worstcase 6.6267 ms/op 5.9543 ms/op 1.11
altair processSyncCommitteeUpdates - 250000 356.70 ms/op 309.60 ms/op 1.15
BeaconState.hashTreeRoot - No change 858.00 ns/op 593.00 ns/op 1.45
BeaconState.hashTreeRoot - 1 full validator 76.153 us/op 72.180 us/op 1.06
BeaconState.hashTreeRoot - 32 full validator 739.22 us/op 703.76 us/op 1.05
BeaconState.hashTreeRoot - 512 full validator 7.7430 ms/op 7.1369 ms/op 1.08
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 96.851 us/op 93.676 us/op 1.03
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.4131 ms/op 1.3326 ms/op 1.06
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.263 ms/op 15.598 ms/op 1.11
BeaconState.hashTreeRoot - 1 balances 71.164 us/op 67.387 us/op 1.06
BeaconState.hashTreeRoot - 32 balances 619.91 us/op 579.59 us/op 1.07
BeaconState.hashTreeRoot - 512 balances 5.7582 ms/op 5.6758 ms/op 1.01
BeaconState.hashTreeRoot - 250000 balances 121.37 ms/op 111.75 ms/op 1.09
aggregationBits - 2048 els - zipIndexesInBitList 41.714 us/op 31.598 us/op 1.32
regular array get 100000 times 58.624 us/op 53.465 us/op 1.10
wrappedArray get 100000 times 58.047 us/op 52.694 us/op 1.10
arrayWithProxy get 100000 times 34.986 ms/op 34.261 ms/op 1.02
ssz.Root.equals 582.00 ns/op 556.00 ns/op 1.05
byteArrayEquals 599.00 ns/op 532.00 ns/op 1.13
shuffle list - 16384 els 12.513 ms/op 11.500 ms/op 1.09
shuffle list - 250000 els 178.80 ms/op 172.98 ms/op 1.03
processSlot - 1 slots 16.458 us/op 16.545 us/op 0.99
processSlot - 32 slots 2.3581 ms/op 2.3179 ms/op 1.02
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 621.43 us/op 682.70 us/op 0.91
getCommitteeAssignments - req 1 vs - 250000 vc 5.6744 ms/op 5.0814 ms/op 1.12
getCommitteeAssignments - req 100 vs - 250000 vc 8.0792 ms/op 7.4753 ms/op 1.08
getCommitteeAssignments - req 1000 vs - 250000 vc 8.5802 ms/op 8.0539 ms/op 1.07
computeProposers - vc 250000 22.916 ms/op 20.762 ms/op 1.10
computeEpochShuffling - vc 250000 189.50 ms/op 172.51 ms/op 1.10
getNextSyncCommittee - vc 250000 353.33 ms/op 305.26 ms/op 1.16

by benchmarkbot/action

@wemeetagain wemeetagain merged commit 9f3fa90 into unstable Jul 17, 2022
@wemeetagain wemeetagain deleted the dapplion/merge-block-processor-single-multi branch July 17, 2022 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants