Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Submit unaggregated attestations in one go #3923

Merged
merged 2 commits into from
Apr 15, 2022
Merged

Submit unaggregated attestations in one go #3923

merged 2 commits into from
Apr 15, 2022

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Apr 14, 2022

Motivation

There are a lot of same logs like below in hetzner nodes:

Apr-13 23:55:16.172[]                 ^[[32minfo^[[39m: Published attestations slot=2782176, index=8, count=1
Apr-13 23:55:16.182[]                 ^[[32minfo^[[39m: Published attestations slot=2782176, index=35, count=1
Apr-13 23:55:16.192[]                 ^[[32minfo^[[39m: Published attestations slot=2782176, index=27, count=1
Apr-13 23:55:16.202[]                 ^[[32minfo^[[39m: Published attestations slot=2782176, index=22, count=1
...
Apr-13 23:55:16.210[]                 ^[[32minfo^[[39m: Published attestations slot=2782176, index=44, count=1
Apr-13 23:55:16.219[]                 ^[[32minfo^[[39m: Published attestations slot=2782176, index=9, count=1
Apr-13 23:55:16.228[]                 ^[[32minfo^[[39m: Published attestations slot=2782176, index=29, count=1
  • We could submit all attestations of the same slot in one go, to save I/O interactions to the beacon node
  • Contabo nodes of 50 validators suffer from I/O lag, this should mitigate the issue

Description

Before

  • Wait for 1/3 of slot or chain head
  • For each committee index
    • Sign attestations
    • Submit attestations
    • Wait for 2/3 of slots
    • Produce and publish AggregateAndProofs

After

  • Wait for 1/3 of slot or chain head
  • For each committee index
    • Sign attestations
  • Collect all signed attestations, submit all in 1 go
  • Wait for 2/3 of slots
  • For each committee index
    • Produce and publish AggregateAndProofs

part of #3922

@codecov
Copy link

codecov bot commented Apr 14, 2022 •
edited
Loading

Codecov Report

Merging #3923 (9050015) into master (053f76a) will increase coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3923      +/-   ##
==========================================
+ Coverage   36.32%   36.48%   +0.15%     
==========================================
  Files         324      324              
  Lines        9059     9091      +32     
  Branches     1453     1470      +17     
==========================================
+ Hits         3291     3317      +26     
- Misses       5597     5600       +3     
- Partials      171      174       +3

@github-actions
Copy link
Contributor

github-actions bot commented Apr 14, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: d5acf7b Previous: 8898b2a Ratio
BeaconState.hashTreeRoot - No change 826.00 ns/op 528.00 ns/op 1.56
BeaconState.hashTreeRoot - 1 full validator 76.720 us/op 56.527 us/op 1.36
BeaconState.hashTreeRoot - 32 full validator 777.57 us/op 542.28 us/op 1.43
BeaconState.hashTreeRoot - 512 full validator 7.9372 ms/op 5.9371 ms/op 1.34
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 97.758 us/op 68.801 us/op 1.42
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3133 ms/op 947.52 us/op 1.39
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 16.863 ms/op 12.682 ms/op 1.33
BeaconState.hashTreeRoot - 1 balances 70.360 us/op 52.575 us/op 1.34
BeaconState.hashTreeRoot - 32 balances 631.91 us/op 473.01 us/op 1.34
BeaconState.hashTreeRoot - 512 balances 6.3318 ms/op 4.5662 ms/op 1.39
BeaconState.hashTreeRoot - 250000 balances 134.56 ms/op 99.328 ms/op 1.35
processSlot - 1 slots 14.464 us/op 9.3900 us/op 1.54
processSlot - 32 slots 2.2319 ms/op 1.5474 ms/op 1.44
getCommitteeAssignments - req 1 vs - 250000 vc 5.8486 ms/op 5.2663 ms/op 1.11
getCommitteeAssignments - req 100 vs - 250000 vc 8.3764 ms/op 7.3396 ms/op 1.14
getCommitteeAssignments - req 1000 vs - 250000 vc 8.6417 ms/op 7.8563 ms/op 1.10
computeProposers - vc 250000 21.164 ms/op 15.937 ms/op 1.33
computeEpochShuffling - vc 250000 169.02 ms/op 158.09 ms/op 1.07
getNextSyncCommittee - vc 250000 347.16 ms/op 264.79 ms/op 1.31
altair processAttestation - 250000 vs - 7PWei normalcase 5.1628 ms/op 3.4784 ms/op 1.48
altair processAttestation - 250000 vs - 7PWei worstcase 7.3275 ms/op 5.2259 ms/op 1.40
altair processAttestation - setStatus - 1/6 committees join 246.93 us/op 192.57 us/op 1.28
altair processAttestation - setStatus - 1/3 committees join 449.13 us/op 372.05 us/op 1.21
altair processAttestation - setStatus - 1/2 committees join 632.09 us/op 523.45 us/op 1.21
altair processAttestation - setStatus - 2/3 committees join 853.60 us/op 675.85 us/op 1.26
altair processAttestation - setStatus - 4/5 committees join 1.2522 ms/op 928.89 us/op 1.35
altair processAttestation - setStatus - 100% committees join 1.4020 ms/op 1.1130 ms/op 1.26
altair processBlock - 250000 vs - 7PWei normalcase 33.505 ms/op 26.675 ms/op 1.26
altair processBlock - 250000 vs - 7PWei normalcase hashState 41.302 ms/op 33.760 ms/op 1.22
altair processBlock - 250000 vs - 7PWei worstcase 90.915 ms/op 69.808 ms/op 1.30
altair processBlock - 250000 vs - 7PWei worstcase hashState 128.24 ms/op 108.31 ms/op 1.18
altair processEth1Data - 250000 vs - 7PWei normalcase 965.85 us/op 722.31 us/op 1.34
altair processEpoch - mainnet_e81889 642.02 ms/op 537.68 ms/op 1.19
mainnet_e81889 - altair beforeProcessEpoch 174.82 ms/op 133.27 ms/op 1.31
mainnet_e81889 - altair processJustificationAndFinalization 72.490 us/op 26.275 us/op 2.76
mainnet_e81889 - altair processInactivityUpdates 11.502 ms/op 10.876 ms/op 1.06
mainnet_e81889 - altair processRewardsAndPenalties 151.31 ms/op 79.284 ms/op 1.91
mainnet_e81889 - altair processRegistryUpdates 19.556 us/op 4.0020 us/op 4.89
mainnet_e81889 - altair processSlashings 6.1860 us/op 1.2750 us/op 4.85
mainnet_e81889 - altair processEth1DataReset 6.8610 us/op 1.4370 us/op 4.77
mainnet_e81889 - altair processEffectiveBalanceUpdates 9.0312 ms/op 6.4732 ms/op 1.40
mainnet_e81889 - altair processSlashingsReset 24.163 us/op 9.4070 us/op 2.57
mainnet_e81889 - altair processRandaoMixesReset 29.435 us/op 11.467 us/op 2.57
mainnet_e81889 - altair processHistoricalRootsUpdate 9.4340 us/op 2.5530 us/op 3.70
mainnet_e81889 - altair processParticipationFlagUpdates 15.683 us/op 4.8180 us/op 3.26
mainnet_e81889 - altair processSyncCommitteeUpdates 5.4920 us/op 1.8280 us/op 3.00
mainnet_e81889 - altair afterProcessEpoch 205.56 ms/op 183.38 ms/op 1.12
altair processInactivityUpdates - 250000 normalcase 41.154 ms/op 32.454 ms/op 1.27
altair processInactivityUpdates - 250000 worstcase 34.751 ms/op 28.454 ms/op 1.22
altair processRewardsAndPenalties - 250000 normalcase 137.69 ms/op 80.167 ms/op 1.72
altair processRewardsAndPenalties - 250000 worstcase 90.345 ms/op 77.529 ms/op 1.17
altair processSyncCommitteeUpdates - 250000 365.62 ms/op 273.41 ms/op 1.34
Tree 40 250000 create 901.23 ms/op 653.69 ms/op 1.38
Tree 40 250000 get(125000) 345.11 ns/op 288.33 ns/op 1.20
Tree 40 250000 set(125000) 3.4797 us/op 2.1520 us/op 1.62
Tree 40 250000 toArray() 37.636 ms/op 30.667 ms/op 1.23
Tree 40 250000 iterate all - toArray() + loop 37.547 ms/op 31.001 ms/op 1.21
Tree 40 250000 iterate all - get(i) 141.45 ms/op 109.74 ms/op 1.29
MutableVector 250000 create 19.947 ms/op 16.162 ms/op 1.23
MutableVector 250000 get(125000) 15.029 ns/op 13.168 ns/op 1.14
MutableVector 250000 set(125000) 900.54 ns/op 544.82 ns/op 1.65
MutableVector 250000 toArray() 7.0660 ms/op 6.1836 ms/op 1.14
MutableVector 250000 iterate all - toArray() + loop 7.4410 ms/op 6.5115 ms/op 1.14
MutableVector 250000 iterate all - get(i) 3.6844 ms/op 3.2745 ms/op 1.13
Array 250000 create 6.8581 ms/op 5.5039 ms/op 1.25
Array 250000 clone - spread 2.7791 ms/op 2.5295 ms/op 1.10
Array 250000 get(125000) 1.4010 ns/op 1.1690 ns/op 1.20
Array 250000 set(125000) 1.3940 ns/op 1.1780 ns/op 1.18
Array 250000 iterate all - loop 141.48 us/op 167.84 us/op 0.84
effectiveBalanceIncrements clone Uint8Array 300000 91.764 us/op 72.427 us/op 1.27
effectiveBalanceIncrements clone MutableVector 300000 981.00 ns/op 804.00 ns/op 1.22
effectiveBalanceIncrements rw all Uint8Array 300000 220.12 us/op 301.86 us/op 0.73
effectiveBalanceIncrements rw all MutableVector 300000 284.50 ms/op 171.85 ms/op 1.66
aggregationBits - 2048 els - zipIndexesInBitList 36.803 us/op 28.382 us/op 1.30
regular array get 100000 times 60.984 us/op 67.432 us/op 0.90
wrappedArray get 100000 times 60.974 us/op 67.475 us/op 0.90
arrayWithProxy get 100000 times 36.689 ms/op 29.475 ms/op 1.24
ssz.Root.equals 650.00 ns/op 526.00 ns/op 1.24
byteArrayEquals 513.00 ns/op 404.00 ns/op 1.27
phase0 processBlock - 250000 vs - 7PWei normalcase 4.2826 ms/op 3.2065 ms/op 1.34
phase0 processBlock - 250000 vs - 7PWei worstcase 58.654 ms/op 44.492 ms/op 1.32
phase0 afterProcessEpoch - 250000 vs - 7PWei 177.14 ms/op 173.47 ms/op 1.02
phase0 beforeProcessEpoch - 250000 vs - 7PWei 80.699 ms/op 70.758 ms/op 1.14
phase0 processEpoch - mainnet_e58758 597.07 ms/op 489.58 ms/op 1.22
mainnet_e58758 - phase0 beforeProcessEpoch 298.01 ms/op 208.33 ms/op 1.43
mainnet_e58758 - phase0 processJustificationAndFinalization 71.573 us/op 30.587 us/op 2.34
mainnet_e58758 - phase0 processRewardsAndPenalties 83.693 ms/op 68.092 ms/op 1.23
mainnet_e58758 - phase0 processRegistryUpdates 34.670 us/op 15.624 us/op 2.22
mainnet_e58758 - phase0 processSlashings 7.2900 us/op 2.5440 us/op 2.87
mainnet_e58758 - phase0 processEth1DataReset 6.0530 us/op 2.0070 us/op 3.02
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 6.4149 ms/op 5.4144 ms/op 1.18
mainnet_e58758 - phase0 processSlashingsReset 22.008 us/op 7.7700 us/op 2.83
mainnet_e58758 - phase0 processRandaoMixesReset 27.525 us/op 9.7710 us/op 2.82
mainnet_e58758 - phase0 processHistoricalRootsUpdate 8.0790 us/op 2.3900 us/op 3.38
mainnet_e58758 - phase0 processParticipationRecordUpdates 25.720 us/op 9.3690 us/op 2.75
mainnet_e58758 - phase0 afterProcessEpoch 167.78 ms/op 153.35 ms/op 1.09
phase0 processEffectiveBalanceUpdates - 250000 normalcase 7.4916 ms/op 6.4338 ms/op 1.16
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 7.7672 ms/op 6.8827 ms/op 1.13
phase0 processRegistryUpdates - 250000 normalcase 31.721 us/op 13.567 us/op 2.34
phase0 processRegistryUpdates - 250000 badcase_full_deposits 516.66 us/op 585.26 us/op 0.88
phase0 processRegistryUpdates - 250000 worstcase 0.5 249.07 ms/op 218.22 ms/op 1.14
phase0 getAttestationDeltas - 250000 normalcase 16.398 ms/op 14.303 ms/op 1.15
phase0 getAttestationDeltas - 250000 worstcase 16.359 ms/op 14.028 ms/op 1.17
phase0 processSlashings - 250000 worstcase 6.9058 ms/op 5.5905 ms/op 1.24
shuffle list - 16384 els 11.319 ms/op 10.629 ms/op 1.06
shuffle list - 250000 els 164.19 ms/op 154.28 ms/op 1.06
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 444.70 us/op 375.86 us/op 1.18
pass gossip attestations to forkchoice per slot 4.0629 ms/op 3.3513 ms/op 1.21
computeDeltas 3.7480 ms/op 3.3201 ms/op 1.13
computeProposerBoostScoreFromBalances 498.21 us/op 502.86 us/op 0.99
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.4308 ms/op 1.8787 ms/op 1.29
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 97.416 us/op 74.590 us/op 1.31
BLS verify - blst-native 2.4903 ms/op 1.8598 ms/op 1.34
BLS verifyMultipleSignatures 3 - blst-native 4.7880 ms/op 3.7988 ms/op 1.26
BLS verifyMultipleSignatures 8 - blst-native 10.274 ms/op 8.1801 ms/op 1.26
BLS verifyMultipleSignatures 32 - blst-native 36.982 ms/op 29.656 ms/op 1.25
BLS aggregatePubkeys 32 - blst-native 49.762 us/op 39.471 us/op 1.26
BLS aggregatePubkeys 128 - blst-native 175.58 us/op 153.44 us/op 1.14
getAttestationsForBlock 70.099 ms/op 54.907 ms/op 1.28
CheckpointStateCache - add get delete 13.671 us/op 10.090 us/op 1.35
validate gossip signedAggregateAndProof - struct 5.3370 ms/op 4.2601 ms/op 1.25
validate gossip attestation - struct 2.6545 ms/op 2.0262 ms/op 1.31
pickEth1Vote - no votes 2.7503 ms/op 2.2425 ms/op 1.23
pickEth1Vote - max votes 22.730 ms/op 18.566 ms/op 1.22
pickEth1Vote - Eth1Data hashTreeRoot value x2048 14.587 ms/op 10.495 ms/op 1.39
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 24.231 ms/op 19.179 ms/op 1.26
pickEth1Vote - Eth1Data fastSerialize value x2048 2.1717 ms/op 1.6527 ms/op 1.31
pickEth1Vote - Eth1Data fastSerialize tree x2048 15.012 ms/op 11.681 ms/op 1.29
bytes32 toHexString 1.2130 us/op 920.00 ns/op 1.32
bytes32 Buffer.toString(hex) 836.00 ns/op 677.00 ns/op 1.23
bytes32 Buffer.toString(hex) from Uint8Array 1.1020 us/op 856.00 ns/op 1.29
bytes32 Buffer.toString(hex) + 0x 845.00 ns/op 676.00 ns/op 1.25
Object access 1 prop 0.40600 ns/op 0.30900 ns/op 1.31
Map access 1 prop 0.35500 ns/op 0.28800 ns/op 1.23
Object get x1000 17.271 ns/op 17.387 ns/op 0.99
Map get x1000 1.0940 ns/op 0.97500 ns/op 1.12
Object set x1000 114.41 ns/op 103.87 ns/op 1.10
Map set x1000 81.756 ns/op 62.614 ns/op 1.31
Return object 10000 times 0.46040 ns/op 0.36830 ns/op 1.25
Throw Error 10000 times 7.4091 us/op 5.9682 us/op 1.24
enrSubnets - fastDeserialize 64 bits 1.2100 us/op 819.00 ns/op 1.48
enrSubnets - ssz BitVector 64 bits 920.00 ns/op 716.00 ns/op 1.28
enrSubnets - fastDeserialize 4 bits 466.00 ns/op 396.00 ns/op 1.18
enrSubnets - ssz BitVector 4 bits 920.00 ns/op 736.00 ns/op 1.25
RateTracker 1000000 limit, 1 obj count per request 200.08 ns/op 178.09 ns/op 1.12
RateTracker 1000000 limit, 2 obj count per request 156.58 ns/op 133.76 ns/op 1.17
RateTracker 1000000 limit, 4 obj count per request 130.69 ns/op 111.92 ns/op 1.17
RateTracker 1000000 limit, 8 obj count per request 112.82 ns/op 96.986 ns/op 1.16
RateTracker with prune 4.9170 us/op 3.6200 us/op 1.36
array of 16000 items push then shift 5.1407 us/op 3.1535 us/op 1.63
LinkedList of 16000 items push then shift 17.111 ns/op 16.806 ns/op 1.02
array of 16000 items push then pop 233.42 ns/op 203.50 ns/op 1.15
LinkedList of 16000 items push then pop 18.407 ns/op 16.003 ns/op 1.15
array of 24000 items push then shift 8.5497 us/op 4.5569 us/op 1.88
LinkedList of 24000 items push then shift 19.961 ns/op 19.109 ns/op 1.04
array of 24000 items push then pop 217.00 ns/op 188.57 ns/op 1.15
LinkedList of 24000 items push then pop 18.296 ns/op 17.838 ns/op 1.03

by benchmarkbot/action

@twoeths twoeths marked this pull request as ready for review April 14, 2022 09:35
@twoeths twoeths requested a review from a team as a code owner April 14, 2022 09:35
@dapplion dapplion merged commit 9f0c7a6 into master Apr 15, 2022
@dapplion dapplion deleted the tuyen/submit-attestations-one-go branch April 15, 2022 08:51
nflaig
nflaig reviewed Mar 9, 2023
View reviewed changes
packages/validator/src/services/attestation.ts
await sleep(this.clock.msToSlot(slot + 2 / 3), signal);
this.metrics?.attesterStepCallProduceAggregate.observe(this.clock.secFromSlot(slot + 2 / 3));

const dutiesByCommitteeIndex = groupAttDutiesByCommitteeIndex(this.dutiesService.getDutiesAtSlot(slot));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tuyennhv why do we need to call this.dutiesService.getDutiesAtSlot again here? I would assume we could just do

const dutiesByCommitteeIndex = groupAttDutiesByCommitteeIndex(duties);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes we can just use duties there, no need to call that again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@twoeths @dapplion @nflaig