Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add TOO_MANY_SKIPPED_SLOTS attestation error #5390

Merged
merged 1 commit into from
Apr 20, 2023
Merged

Add TOO_MANY_SKIPPED_SLOTS attestation error #5390

merged 1 commit into from
Apr 20, 2023

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Apr 20, 2023 •
edited
Loading

Motivation

  • Regen module is overloaded, see High regen queue job wait time #5384
  • We want to have DOS protection to avoid having to process attestations with very old head block that can potentially cause our regen to be overloaded

Description

  • Follow Lighthouse to add TOO_MANY_SKIPPED_SLOTS error
  • By default, configure maxSkipSlots of chain to be 32:
    • for gossip block validation, it's unlikely we see a reorg with 32 slots
    • for attestation validation, having this value ensures we don't have to regen states most of the time (because attestation has to be in the last 32 slots + maxSkipSlots of 32 above => headBlock of attestation should be in the last 64 slots and it's likely we can get its state from the cache). Also most of Attestations (>90%) are caught by the SeenAttestationDatas cache, it's no problem not to import these weird attestations.
  • Import block: add postState to state cache before we emit "block" event since some handlers need that
  • Add metrics to track this new error

part of #5384

@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 962b0a8 Previous: 35fe214 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 859.81 us/op 895.18 us/op 0.96
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 45.450 us/op 78.171 us/op 0.58
BLS verify - blst-native 1.2048 ms/op 1.8978 ms/op 0.63
BLS verifyMultipleSignatures 3 - blst-native 2.4555 ms/op 3.4992 ms/op 0.70
BLS verifyMultipleSignatures 8 - blst-native 5.2677 ms/op 6.3512 ms/op 0.83
BLS verifyMultipleSignatures 32 - blst-native 19.037 ms/op 23.447 ms/op 0.81
BLS aggregatePubkeys 32 - blst-native 25.458 us/op 31.114 us/op 0.82
BLS aggregatePubkeys 128 - blst-native 99.501 us/op 124.86 us/op 0.80
getAttestationsForBlock 51.581 ms/op 104.20 ms/op 0.49
isKnown best case - 1 super set check 251.00 ns/op 356.00 ns/op 0.71
isKnown normal case - 2 super set checks 250.00 ns/op 296.00 ns/op 0.84
isKnown worse case - 16 super set checks 244.00 ns/op 319.00 ns/op 0.76
CheckpointStateCache - add get delete 4.9030 us/op 7.3770 us/op 0.66
validate gossip signedAggregateAndProof - struct 2.7220 ms/op 3.1878 ms/op 0.85
validate gossip attestation - struct 1.2965 ms/op 1.7663 ms/op 0.73
pickEth1Vote - no votes 1.2085 ms/op 1.5865 ms/op 0.76
pickEth1Vote - max votes 9.2327 ms/op 16.371 ms/op 0.56
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.4782 ms/op 13.183 ms/op 0.64
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 12.955 ms/op 21.800 ms/op 0.59
pickEth1Vote - Eth1Data fastSerialize value x2048 634.86 us/op 931.01 us/op 0.68
pickEth1Vote - Eth1Data fastSerialize tree x2048 7.9941 ms/op 7.5940 ms/op 1.05
bytes32 toHexString 459.00 ns/op 1.0680 us/op 0.43
bytes32 Buffer.toString(hex) 327.00 ns/op 527.00 ns/op 0.62
bytes32 Buffer.toString(hex) from Uint8Array 520.00 ns/op 733.00 ns/op 0.71
bytes32 Buffer.toString(hex) + 0x 381.00 ns/op 495.00 ns/op 0.77
Object access 1 prop 0.15700 ns/op 0.24400 ns/op 0.64
Map access 1 prop 0.15300 ns/op 0.19300 ns/op 0.79
Object get x1000 6.1300 ns/op 7.5130 ns/op 0.82
Map get x1000 0.49800 ns/op 0.85600 ns/op 0.58
Object set x1000 49.293 ns/op 89.315 ns/op 0.55
Map set x1000 40.945 ns/op 66.314 ns/op 0.62
Return object 10000 times 0.22330 ns/op 0.30200 ns/op 0.74
Throw Error 10000 times 3.9679 us/op 6.4323 us/op 0.62
fastMsgIdFn sha256 / 200 bytes 3.2800 us/op 4.8430 us/op 0.68
fastMsgIdFn h32 xxhash / 200 bytes 263.00 ns/op 482.00 ns/op 0.55
fastMsgIdFn h64 xxhash / 200 bytes 374.00 ns/op 618.00 ns/op 0.61
fastMsgIdFn sha256 / 1000 bytes 10.991 us/op 14.537 us/op 0.76
fastMsgIdFn h32 xxhash / 1000 bytes 386.00 ns/op 612.00 ns/op 0.63
fastMsgIdFn h64 xxhash / 1000 bytes 442.00 ns/op 860.00 ns/op 0.51
fastMsgIdFn sha256 / 10000 bytes 98.659 us/op 133.59 us/op 0.74
fastMsgIdFn h32 xxhash / 10000 bytes 1.8060 us/op 2.2760 us/op 0.79
fastMsgIdFn h64 xxhash / 10000 bytes 1.2770 us/op 1.7070 us/op 0.75
enrSubnets - fastDeserialize 64 bits 1.2200 us/op 2.0920 us/op 0.58
enrSubnets - ssz BitVector 64 bits 485.00 ns/op 709.00 ns/op 0.68
enrSubnets - fastDeserialize 4 bits 159.00 ns/op 252.00 ns/op 0.63
enrSubnets - ssz BitVector 4 bits 480.00 ns/op 734.00 ns/op 0.65
prioritizePeers score -10:0 att 32-0.1 sync 2-0 100.41 us/op 142.83 us/op 0.70
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 129.06 us/op 196.96 us/op 0.66
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 169.09 us/op 230.01 us/op 0.74
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 315.06 us/op 507.91 us/op 0.62
prioritizePeers score 0:0 att 64-1 sync 4-1 349.41 us/op 525.52 us/op 0.66
array of 16000 items push then shift 1.5664 us/op 2.1819 us/op 0.72
LinkedList of 16000 items push then shift 8.4070 ns/op 11.267 ns/op 0.75
array of 16000 items push then pop 80.456 ns/op 143.29 ns/op 0.56
LinkedList of 16000 items push then pop 8.1400 ns/op 12.286 ns/op 0.66
array of 24000 items push then shift 2.3558 us/op 3.5229 us/op 0.67
LinkedList of 24000 items push then shift 8.9820 ns/op 15.293 ns/op 0.59
array of 24000 items push then pop 80.534 ns/op 135.91 ns/op 0.59
LinkedList of 24000 items push then pop 8.7110 ns/op 13.051 ns/op 0.67
intersect bitArray bitLen 8 13.326 ns/op 22.157 ns/op 0.60
intersect array and set length 8 86.544 ns/op 138.57 ns/op 0.62
intersect bitArray bitLen 128 44.322 ns/op 68.213 ns/op 0.65
intersect array and set length 128 1.1640 us/op 1.8323 us/op 0.64
Buffer.concat 32 items 2.9540 us/op 4.3000 us/op 0.69
Uint8Array.set 32 items 2.1980 us/op 4.1020 us/op 0.54
pass gossip attestations to forkchoice per slot 2.7200 ms/op 4.5814 ms/op 0.59
computeDeltas 2.8027 ms/op 3.9152 ms/op 0.72
computeProposerBoostScoreFromBalances 1.7330 ms/op 2.1735 ms/op 0.80
altair processAttestation - 250000 vs - 7PWei normalcase 2.1718 ms/op 4.3472 ms/op 0.50
altair processAttestation - 250000 vs - 7PWei worstcase 3.2265 ms/op 5.7078 ms/op 0.57
altair processAttestation - setStatus - 1/6 committees join 132.18 us/op 208.69 us/op 0.63
altair processAttestation - setStatus - 1/3 committees join 268.24 us/op 367.37 us/op 0.73
altair processAttestation - setStatus - 1/2 committees join 351.36 us/op 614.54 us/op 0.57
altair processAttestation - setStatus - 2/3 committees join 454.85 us/op 681.62 us/op 0.67
altair processAttestation - setStatus - 4/5 committees join 634.14 us/op 884.28 us/op 0.72
altair processAttestation - setStatus - 100% committees join 745.78 us/op 954.54 us/op 0.78
altair processBlock - 250000 vs - 7PWei normalcase 16.321 ms/op 26.958 ms/op 0.61
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.358 ms/op 39.614 ms/op 0.69
altair processBlock - 250000 vs - 7PWei worstcase 46.737 ms/op 74.809 ms/op 0.62
altair processBlock - 250000 vs - 7PWei worstcase hashState 70.310 ms/op 99.292 ms/op 0.71
phase0 processBlock - 250000 vs - 7PWei normalcase 1.9689 ms/op 3.4705 ms/op 0.57
phase0 processBlock - 250000 vs - 7PWei worstcase 26.972 ms/op 40.463 ms/op 0.67
altair processEth1Data - 250000 vs - 7PWei normalcase 459.02 us/op 885.37 us/op 0.52
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 6.5960 us/op 14.874 us/op 0.44
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 19.154 us/op 41.383 us/op 0.46
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 8.5620 us/op 21.332 us/op 0.40
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 6.2580 us/op 16.640 us/op 0.38
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 75.725 us/op 123.24 us/op 0.61
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 616.64 us/op 939.69 us/op 0.66
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 891.57 us/op 1.0015 ms/op 0.89
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 877.46 us/op 1.4407 ms/op 0.61
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.2505 ms/op 4.5426 ms/op 0.50
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.4252 ms/op 2.5841 ms/op 0.55
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.7731 ms/op 7.1813 ms/op 0.53
Tree 40 250000 create 308.00 ms/op 824.19 ms/op 0.37
Tree 40 250000 get(125000) 176.60 ns/op 233.07 ns/op 0.76
Tree 40 250000 set(125000) 921.74 ns/op 2.7860 us/op 0.33
Tree 40 250000 toArray() 17.761 ms/op 29.665 ms/op 0.60
Tree 40 250000 iterate all - toArray() + loop 16.184 ms/op 30.121 ms/op 0.54
Tree 40 250000 iterate all - get(i) 63.353 ms/op 93.518 ms/op 0.68
MutableVector 250000 create 9.8106 ms/op 14.440 ms/op 0.68
MutableVector 250000 get(125000) 6.1120 ns/op 7.4300 ns/op 0.82
MutableVector 250000 set(125000) 249.07 ns/op 730.10 ns/op 0.34
MutableVector 250000 toArray() 2.6506 ms/op 5.5196 ms/op 0.48
MutableVector 250000 iterate all - toArray() + loop 2.7688 ms/op 6.0607 ms/op 0.46
MutableVector 250000 iterate all - get(i) 1.4620 ms/op 2.1970 ms/op 0.67
Array 250000 create 2.5232 ms/op 5.9505 ms/op 0.42
Array 250000 clone - spread 1.1296 ms/op 2.9590 ms/op 0.38
Array 250000 get(125000) 0.52800 ns/op 1.4880 ns/op 0.35
Array 250000 set(125000) 0.60800 ns/op 1.3950 ns/op 0.44
Array 250000 iterate all - loop 105.33 us/op 125.06 us/op 0.84
effectiveBalanceIncrements clone Uint8Array 300000 24.603 us/op 78.129 us/op 0.31
effectiveBalanceIncrements clone MutableVector 300000 349.00 ns/op 721.00 ns/op 0.48
effectiveBalanceIncrements rw all Uint8Array 300000 159.84 us/op 201.41 us/op 0.79
effectiveBalanceIncrements rw all MutableVector 300000 80.139 ms/op 188.95 ms/op 0.42
phase0 afterProcessEpoch - 250000 vs - 7PWei 108.61 ms/op 178.68 ms/op 0.61
phase0 beforeProcessEpoch - 250000 vs - 7PWei 33.907 ms/op 61.339 ms/op 0.55
altair processEpoch - mainnet_e81889 316.64 ms/op 462.78 ms/op 0.68
mainnet_e81889 - altair beforeProcessEpoch 63.341 ms/op 100.77 ms/op 0.63
mainnet_e81889 - altair processJustificationAndFinalization 17.843 us/op 41.399 us/op 0.43
mainnet_e81889 - altair processInactivityUpdates 5.5458 ms/op 9.5439 ms/op 0.58
mainnet_e81889 - altair processRewardsAndPenalties 64.122 ms/op 75.740 ms/op 0.85
mainnet_e81889 - altair processRegistryUpdates 3.3630 us/op 8.6870 us/op 0.39
mainnet_e81889 - altair processSlashings 509.00 ns/op 1.6110 us/op 0.32
mainnet_e81889 - altair processEth1DataReset 574.00 ns/op 1.9780 us/op 0.29
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3395 ms/op 1.8819 ms/op 0.71
mainnet_e81889 - altair processSlashingsReset 5.1990 us/op 14.029 us/op 0.37
mainnet_e81889 - altair processRandaoMixesReset 5.5050 us/op 12.776 us/op 0.43
mainnet_e81889 - altair processHistoricalRootsUpdate 1.1460 us/op 2.4420 us/op 0.47
mainnet_e81889 - altair processParticipationFlagUpdates 3.0110 us/op 9.4340 us/op 0.32
mainnet_e81889 - altair processSyncCommitteeUpdates 805.00 ns/op 2.1490 us/op 0.37
mainnet_e81889 - altair afterProcessEpoch 137.33 ms/op 184.83 ms/op 0.74
phase0 processEpoch - mainnet_e58758 357.43 ms/op 569.50 ms/op 0.63
mainnet_e58758 - phase0 beforeProcessEpoch 151.12 ms/op 254.50 ms/op 0.59
mainnet_e58758 - phase0 processJustificationAndFinalization 19.595 us/op 40.239 us/op 0.49
mainnet_e58758 - phase0 processRewardsAndPenalties 65.097 ms/op 64.063 ms/op 1.02
mainnet_e58758 - phase0 processRegistryUpdates 7.8430 us/op 22.687 us/op 0.35
mainnet_e58758 - phase0 processSlashings 493.00 ns/op 1.8950 us/op 0.26
mainnet_e58758 - phase0 processEth1DataReset 494.00 ns/op 1.8020 us/op 0.27
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.3384 ms/op 1.9964 ms/op 0.67
mainnet_e58758 - phase0 processSlashingsReset 4.8980 us/op 9.8590 us/op 0.50
mainnet_e58758 - phase0 processRandaoMixesReset 4.5050 us/op 15.657 us/op 0.29
mainnet_e58758 - phase0 processHistoricalRootsUpdate 601.00 ns/op 2.3240 us/op 0.26
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.7580 us/op 12.583 us/op 0.30
mainnet_e58758 - phase0 afterProcessEpoch 99.648 ms/op 124.76 ms/op 0.80
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2430 ms/op 2.3862 ms/op 0.52
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4738 ms/op 2.6006 ms/op 0.57
altair processInactivityUpdates - 250000 normalcase 23.731 ms/op 37.153 ms/op 0.64
altair processInactivityUpdates - 250000 worstcase 27.441 ms/op 29.482 ms/op 0.93
phase0 processRegistryUpdates - 250000 normalcase 8.4370 us/op 17.953 us/op 0.47
phase0 processRegistryUpdates - 250000 badcase_full_deposits 284.35 us/op 307.45 us/op 0.92
phase0 processRegistryUpdates - 250000 worstcase 0.5 137.18 ms/op 129.77 ms/op 1.06
altair processRewardsAndPenalties - 250000 normalcase 71.191 ms/op 57.883 ms/op 1.23
altair processRewardsAndPenalties - 250000 worstcase 72.142 ms/op 55.191 ms/op 1.31
phase0 getAttestationDeltas - 250000 normalcase 6.7469 ms/op 7.0069 ms/op 0.96
phase0 getAttestationDeltas - 250000 worstcase 6.4363 ms/op 7.0916 ms/op 0.91
phase0 processSlashings - 250000 worstcase 3.5965 ms/op 3.5441 ms/op 1.01
altair processSyncCommitteeUpdates - 250000 171.48 ms/op 187.78 ms/op 0.91
BeaconState.hashTreeRoot - No change 268.00 ns/op 273.00 ns/op 0.98
BeaconState.hashTreeRoot - 1 full validator 61.042 us/op 54.180 us/op 1.13
BeaconState.hashTreeRoot - 32 full validator 503.10 us/op 533.26 us/op 0.94
BeaconState.hashTreeRoot - 512 full validator 4.9795 ms/op 6.0275 ms/op 0.83
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 62.196 us/op 67.514 us/op 0.92
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 873.23 us/op 925.59 us/op 0.94
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.093 ms/op 12.151 ms/op 0.91
BeaconState.hashTreeRoot - 1 balances 48.643 us/op 48.314 us/op 1.01
BeaconState.hashTreeRoot - 32 balances 441.11 us/op 467.12 us/op 0.94
BeaconState.hashTreeRoot - 512 balances 4.2786 ms/op 4.7142 ms/op 0.91
BeaconState.hashTreeRoot - 250000 balances 77.910 ms/op 69.519 ms/op 1.12
aggregationBits - 2048 els - zipIndexesInBitList 16.750 us/op 19.489 us/op 0.86
regular array get 100000 times 32.719 us/op 33.960 us/op 0.96
wrappedArray get 100000 times 32.660 us/op 34.400 us/op 0.95
arrayWithProxy get 100000 times 15.820 ms/op 16.826 ms/op 0.94
ssz.Root.equals 549.00 ns/op 578.00 ns/op 0.95
byteArrayEquals 536.00 ns/op 574.00 ns/op 0.93
shuffle list - 16384 els 6.8371 ms/op 7.2520 ms/op 0.94
shuffle list - 250000 els 99.559 ms/op 107.07 ms/op 0.93
processSlot - 1 slots 9.2250 us/op 9.1680 us/op 1.01
processSlot - 32 slots 1.3330 ms/op 1.4139 ms/op 0.94
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 37.588 ms/op 33.799 ms/op 1.11
getCommitteeAssignments - req 1 vs - 250000 vc 2.8798 ms/op 2.9347 ms/op 0.98
getCommitteeAssignments - req 100 vs - 250000 vc 4.1243 ms/op 4.1443 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4358 ms/op 4.5331 ms/op 0.98
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.7900 ns/op 4.7700 ns/op 1.00
state getBlockRootAtSlot - 250000 vs - 7PWei 754.62 ns/op 780.80 ns/op 0.97
computeProposers - vc 250000 10.922 ms/op 10.648 ms/op 1.03
computeEpochShuffling - vc 250000 103.37 ms/op 102.74 ms/op 1.01
getNextSyncCommittee - vc 250000 176.93 ms/op 175.92 ms/op 1.01
computeSigningRoot for AttestationData 13.005 us/op 13.991 us/op 0.93
hash AttestationData serialized data then Buffer.toString(base64) 2.4607 us/op 2.4579 us/op 1.00
toHexString serialized data 1.0799 us/op 1.1221 us/op 0.96
Buffer.toString(base64) 314.23 ns/op 361.70 ns/op 0.87

by benchmarkbot/action

@twoeths
Copy link
Contributor Author

twoeths commented Apr 20, 2023

feat1

  • Was able to cache some attestations with very old block head

Screenshot 2023-04-20 at 19 32 14

- While there is no cache miss due to `getState()` entry point

Screenshot 2023-04-20 at 19 34 30

unstable

  • It shows there are always some cache miss due to getState() consistently (from attestation validation) and it'll cause regen

Screenshot 2023-04-20 at 19 35 26

@twoeths twoeths marked this pull request as ready for review April 20, 2023 12:36
@twoeths twoeths requested a review from a team as a code owner April 20, 2023 12:37
g11tech
g11tech approved these changes Apr 20, 2023
View reviewed changes
Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

nflaig
nflaig approved these changes Apr 20, 2023
View reviewed changes
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, great comments, makes it even for me easy to understand 👍

@wemeetagain wemeetagain merged commit 7af22d7 into unstable Apr 20, 2023
@wemeetagain wemeetagain deleted the tuyen/too_many_skipped_slots_error branch April 20, 2023 14:06
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.8.0 🎉

@wemeetagain wemeetagain mentioned this pull request May 16, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

@nflaig nflaig nflaig approved these changes

@g11tech g11tech g11tech approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@twoeths @wemeetagain @nflaig @g11tech