Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement committee_hash lightclient api #4736

Merged
merged 8 commits into from
Nov 15, 2022
Merged

Implement committee_hash lightclient api #4736

merged 8 commits into from
Nov 15, 2022

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Nov 8, 2022

Motivation

Description

  • From a period, get best update
  • Extract attester header than beacon block root from there
  • Get sync committee witness from syncCommitteeWitness db
  • Get current sync committee from syncCommittee db

part of #4733

@github-actions
Copy link
Contributor

github-actions bot commented Nov 8, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: e55cd82 Previous: 1ceaa20 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.3464 ms/op 1.2088 ms/op 1.94
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 64.518 us/op 49.006 us/op 1.32
BLS verify - blst-native 2.1703 ms/op 1.5911 ms/op 1.36
BLS verifyMultipleSignatures 3 - blst-native 4.4753 ms/op 3.2743 ms/op 1.37
BLS verifyMultipleSignatures 8 - blst-native 9.6757 ms/op 7.0766 ms/op 1.37
BLS verifyMultipleSignatures 32 - blst-native 35.180 ms/op 25.687 ms/op 1.37
BLS aggregatePubkeys 32 - blst-native 46.760 us/op 35.227 us/op 1.33
BLS aggregatePubkeys 128 - blst-native 181.99 us/op 137.64 us/op 1.32
getAttestationsForBlock 84.617 ms/op 60.936 ms/op 1.39
isKnown best case - 1 super set check 490.00 ns/op 335.00 ns/op 1.46
isKnown normal case - 2 super set checks 474.00 ns/op 327.00 ns/op 1.45
isKnown worse case - 16 super set checks 475.00 ns/op 328.00 ns/op 1.45
CheckpointStateCache - add get delete 9.2150 us/op 7.0680 us/op 1.30
validate gossip signedAggregateAndProof - struct 5.0381 ms/op 3.6981 ms/op 1.36
validate gossip attestation - struct 2.3626 ms/op 1.7414 ms/op 1.36
pickEth1Vote - no votes 2.1461 ms/op 1.6398 ms/op 1.31
pickEth1Vote - max votes 19.785 ms/op 14.439 ms/op 1.37
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.031 ms/op 8.6904 ms/op 1.50
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 20.690 ms/op 15.633 ms/op 1.32
pickEth1Vote - Eth1Data fastSerialize value x2048 1.4802 ms/op 955.51 us/op 1.55
pickEth1Vote - Eth1Data fastSerialize tree x2048 13.742 ms/op 8.6387 ms/op 1.59
bytes32 toHexString 994.00 ns/op 725.00 ns/op 1.37
bytes32 Buffer.toString(hex) 751.00 ns/op 585.00 ns/op 1.28
bytes32 Buffer.toString(hex) from Uint8Array 1.0200 us/op 734.00 ns/op 1.39
bytes32 Buffer.toString(hex) + 0x 776.00 ns/op 544.00 ns/op 1.43
Object access 1 prop 0.40000 ns/op 0.30300 ns/op 1.32
Map access 1 prop 0.31000 ns/op 0.24000 ns/op 1.29
Object get x1000 11.275 ns/op 9.0340 ns/op 1.25
Map get x1000 0.93300 ns/op 0.85200 ns/op 1.10
Object set x1000 73.359 ns/op 59.837 ns/op 1.23
Map set x1000 49.416 ns/op 38.972 ns/op 1.27
Return object 10000 times 0.43280 ns/op 0.33050 ns/op 1.31
Throw Error 10000 times 6.1912 us/op 4.4936 us/op 1.38
fastMsgIdFn sha256 / 200 bytes 4.8490 us/op 3.3410 us/op 1.45
fastMsgIdFn h32 xxhash / 200 bytes 594.00 ns/op 405.00 ns/op 1.47
fastMsgIdFn h64 xxhash / 200 bytes 896.00 ns/op 566.00 ns/op 1.58
fastMsgIdFn sha256 / 1000 bytes 15.497 us/op 10.622 us/op 1.46
fastMsgIdFn h32 xxhash / 1000 bytes 785.00 ns/op 617.00 ns/op 1.27
fastMsgIdFn h64 xxhash / 1000 bytes 922.00 ns/op 643.00 ns/op 1.43
fastMsgIdFn sha256 / 10000 bytes 133.67 us/op 90.805 us/op 1.47
fastMsgIdFn h32 xxhash / 10000 bytes 2.6040 us/op 2.6940 us/op 0.97
fastMsgIdFn h64 xxhash / 10000 bytes 2.0090 us/op 1.8390 us/op 1.09
enrSubnets - fastDeserialize 64 bits 3.0190 us/op 1.7680 us/op 1.71
enrSubnets - ssz BitVector 64 bits 861.00 ns/op 571.00 ns/op 1.51
enrSubnets - fastDeserialize 4 bits 413.00 ns/op 267.00 ns/op 1.55
enrSubnets - ssz BitVector 4 bits 881.00 ns/op 582.00 ns/op 1.51
prioritizePeers score -10:0 att 32-0.1 sync 2-0 89.089 us/op 61.328 us/op 1.45
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 139.11 us/op 97.228 us/op 1.43
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 231.77 us/op 155.39 us/op 1.49
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 348.68 us/op 276.60 us/op 1.26
prioritizePeers score 0:0 att 64-1 sync 4-1 454.80 us/op 333.46 us/op 1.36
RateTracker 1000000 limit, 1 obj count per request 196.00 ns/op 139.03 ns/op 1.41
RateTracker 1000000 limit, 2 obj count per request 146.27 ns/op 104.78 ns/op 1.40
RateTracker 1000000 limit, 4 obj count per request 122.24 ns/op 88.920 ns/op 1.37
RateTracker 1000000 limit, 8 obj count per request 109.42 ns/op 78.398 ns/op 1.40
RateTracker with prune 4.8220 us/op 3.2470 us/op 1.49
array of 16000 items push then shift 51.632 us/op 2.1641 us/op 23.86
LinkedList of 16000 items push then shift 12.640 ns/op 9.3890 ns/op 1.35
array of 16000 items push then pop 205.46 ns/op 146.13 ns/op 1.41
LinkedList of 16000 items push then pop 12.042 ns/op 9.1740 ns/op 1.31
array of 24000 items push then shift 77.439 us/op 3.1084 us/op 24.91
LinkedList of 24000 items push then shift 13.072 ns/op 9.3990 ns/op 1.39
array of 24000 items push then pop 194.55 ns/op 100.20 ns/op 1.94
LinkedList of 24000 items push then pop 12.105 ns/op 9.0420 ns/op 1.34
intersect bitArray bitLen 8 10.814 ns/op 8.2020 ns/op 1.32
intersect array and set length 8 134.56 ns/op 102.38 ns/op 1.31
intersect bitArray bitLen 128 55.709 ns/op 40.161 ns/op 1.39
intersect array and set length 128 1.8373 us/op 1.4304 us/op 1.28
Buffer.concat 32 items 1.8120 ns/op 1.2920 ns/op 1.40
pass gossip attestations to forkchoice per slot 3.5563 ms/op 4.3639 ms/op 0.81
computeDeltas 4.7661 ms/op 3.9580 ms/op 1.20
computeProposerBoostScoreFromBalances 804.84 us/op 619.15 us/op 1.30
altair processAttestation - 250000 vs - 7PWei normalcase 3.6261 ms/op 2.5583 ms/op 1.42
altair processAttestation - 250000 vs - 7PWei worstcase 5.3605 ms/op 3.7799 ms/op 1.42
altair processAttestation - setStatus - 1/6 committees join 177.20 us/op 142.47 us/op 1.24
altair processAttestation - setStatus - 1/3 committees join 350.41 us/op 261.60 us/op 1.34
altair processAttestation - setStatus - 1/2 committees join 501.01 us/op 371.25 us/op 1.35
altair processAttestation - setStatus - 2/3 committees join 656.51 us/op 482.15 us/op 1.36
altair processAttestation - setStatus - 4/5 committees join 928.25 us/op 643.46 us/op 1.44
altair processAttestation - setStatus - 100% committees join 1.1062 ms/op 767.92 us/op 1.44
altair processBlock - 250000 vs - 7PWei normalcase 27.506 ms/op 19.012 ms/op 1.45
altair processBlock - 250000 vs - 7PWei normalcase hashState 37.340 ms/op 27.181 ms/op 1.37
altair processBlock - 250000 vs - 7PWei worstcase 79.372 ms/op 57.726 ms/op 1.37
altair processBlock - 250000 vs - 7PWei worstcase hashState 114.72 ms/op 75.406 ms/op 1.52
phase0 processBlock - 250000 vs - 7PWei normalcase 3.3864 ms/op 2.1326 ms/op 1.59
phase0 processBlock - 250000 vs - 7PWei worstcase 51.726 ms/op 36.549 ms/op 1.42
altair processEth1Data - 250000 vs - 7PWei normalcase 746.17 us/op 466.02 us/op 1.60
Tree 40 250000 create 752.10 ms/op 438.64 ms/op 1.71
Tree 40 250000 get(125000) 244.61 ns/op 194.05 ns/op 1.26
Tree 40 250000 set(125000) 2.2843 us/op 1.2665 us/op 1.80
Tree 40 250000 toArray() 27.951 ms/op 22.887 ms/op 1.22
Tree 40 250000 iterate all - toArray() + loop 28.128 ms/op 23.494 ms/op 1.20
Tree 40 250000 iterate all - get(i) 116.78 ms/op 80.252 ms/op 1.46
MutableVector 250000 create 13.786 ms/op 9.4587 ms/op 1.46
MutableVector 250000 get(125000) 10.667 ns/op 8.5360 ns/op 1.25
MutableVector 250000 set(125000) 581.10 ns/op 389.74 ns/op 1.49
MutableVector 250000 toArray() 6.6413 ms/op 4.0940 ms/op 1.62
MutableVector 250000 iterate all - toArray() + loop 6.8220 ms/op 3.7620 ms/op 1.81
MutableVector 250000 iterate all - get(i) 2.6898 ms/op 2.0687 ms/op 1.30
Array 250000 create 6.6412 ms/op 3.2874 ms/op 2.02
Array 250000 clone - spread 3.2521 ms/op 1.4777 ms/op 2.20
Array 250000 get(125000) 1.4990 ns/op 0.89400 ns/op 1.68
Array 250000 set(125000) 1.5330 ns/op 1.0000 ns/op 1.53
Array 250000 iterate all - loop 152.87 us/op 119.43 us/op 1.28
effectiveBalanceIncrements clone Uint8Array 300000 50.375 us/op 102.26 us/op 0.49
effectiveBalanceIncrements clone MutableVector 300000 983.00 ns/op 494.00 ns/op 1.99
effectiveBalanceIncrements rw all Uint8Array 300000 247.53 us/op 278.38 us/op 0.89
effectiveBalanceIncrements rw all MutableVector 300000 189.09 ms/op 100.64 ms/op 1.88
phase0 afterProcessEpoch - 250000 vs - 7PWei 188.72 ms/op 124.87 ms/op 1.51
phase0 beforeProcessEpoch - 250000 vs - 7PWei 58.941 ms/op 40.963 ms/op 1.44
altair processEpoch - mainnet_e81889 554.32 ms/op 378.31 ms/op 1.47
mainnet_e81889 - altair beforeProcessEpoch 86.981 ms/op 80.988 ms/op 1.07
mainnet_e81889 - altair processJustificationAndFinalization 17.403 us/op 16.471 us/op 1.06
mainnet_e81889 - altair processInactivityUpdates 8.7460 ms/op 7.6340 ms/op 1.15
mainnet_e81889 - altair processRewardsAndPenalties 119.71 ms/op 85.535 ms/op 1.40
mainnet_e81889 - altair processRegistryUpdates 2.5880 us/op 2.5950 us/op 1.00
mainnet_e81889 - altair processSlashings 637.00 ns/op 441.00 ns/op 1.44
mainnet_e81889 - altair processEth1DataReset 624.00 ns/op 491.00 ns/op 1.27
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.9279 ms/op 1.4370 ms/op 1.34
mainnet_e81889 - altair processSlashingsReset 4.4940 us/op 3.8880 us/op 1.16
mainnet_e81889 - altair processRandaoMixesReset 7.2880 us/op 4.1340 us/op 1.76
mainnet_e81889 - altair processHistoricalRootsUpdate 1.1140 us/op 653.00 ns/op 1.71
mainnet_e81889 - altair processParticipationFlagUpdates 3.0550 us/op 3.6360 us/op 0.84
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0200 us/op 491.00 ns/op 2.08
mainnet_e81889 - altair afterProcessEpoch 197.24 ms/op 147.13 ms/op 1.34
phase0 processEpoch - mainnet_e58758 494.67 ms/op 384.53 ms/op 1.29
mainnet_e58758 - phase0 beforeProcessEpoch 189.32 ms/op 131.26 ms/op 1.44
mainnet_e58758 - phase0 processJustificationAndFinalization 16.580 us/op 16.379 us/op 1.01
mainnet_e58758 - phase0 processRewardsAndPenalties 117.90 ms/op 69.271 ms/op 1.70
mainnet_e58758 - phase0 processRegistryUpdates 8.1490 us/op 6.8100 us/op 1.20
mainnet_e58758 - phase0 processSlashings 661.00 ns/op 532.00 ns/op 1.24
mainnet_e58758 - phase0 processEth1DataReset 566.00 ns/op 528.00 ns/op 1.07
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.6211 ms/op 1.1822 ms/op 1.37
mainnet_e58758 - phase0 processSlashingsReset 3.6030 us/op 3.6250 us/op 0.99
mainnet_e58758 - phase0 processRandaoMixesReset 4.1730 us/op 3.5570 us/op 1.17
mainnet_e58758 - phase0 processHistoricalRootsUpdate 636.00 ns/op 647.00 ns/op 0.98
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.5680 us/op 3.7170 us/op 0.96
mainnet_e58758 - phase0 afterProcessEpoch 162.89 ms/op 110.37 ms/op 1.48
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.9805 ms/op 1.4324 ms/op 1.38
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.2372 ms/op 1.9065 ms/op 1.17
altair processInactivityUpdates - 250000 normalcase 40.884 ms/op 24.880 ms/op 1.64
altair processInactivityUpdates - 250000 worstcase 34.529 ms/op 23.553 ms/op 1.47
phase0 processRegistryUpdates - 250000 normalcase 6.0780 us/op 7.0870 us/op 0.86
phase0 processRegistryUpdates - 250000 badcase_full_deposits 375.79 us/op 322.48 us/op 1.17
phase0 processRegistryUpdates - 250000 worstcase 0.5 193.75 ms/op 136.31 ms/op 1.42
altair processRewardsAndPenalties - 250000 normalcase 72.032 ms/op 72.132 ms/op 1.00
altair processRewardsAndPenalties - 250000 worstcase 112.03 ms/op 53.532 ms/op 2.09
phase0 getAttestationDeltas - 250000 normalcase 13.054 ms/op 7.4778 ms/op 1.75
phase0 getAttestationDeltas - 250000 worstcase 13.182 ms/op 7.4893 ms/op 1.76
phase0 processSlashings - 250000 worstcase 5.2277 ms/op 3.8177 ms/op 1.37
altair processSyncCommitteeUpdates - 250000 299.55 ms/op 208.59 ms/op 1.44
BeaconState.hashTreeRoot - No change 551.00 ns/op 369.00 ns/op 1.49
BeaconState.hashTreeRoot - 1 full validator 66.040 us/op 48.983 us/op 1.35
BeaconState.hashTreeRoot - 32 full validator 704.98 us/op 568.50 us/op 1.24
BeaconState.hashTreeRoot - 512 full validator 8.0104 ms/op 4.8586 ms/op 1.65
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 92.564 us/op 62.458 us/op 1.48
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3249 ms/op 888.81 us/op 1.49
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.646 ms/op 11.721 ms/op 1.51
BeaconState.hashTreeRoot - 1 balances 69.270 us/op 43.270 us/op 1.60
BeaconState.hashTreeRoot - 32 balances 733.26 us/op 457.05 us/op 1.60
BeaconState.hashTreeRoot - 512 balances 6.4333 ms/op 4.4924 ms/op 1.43
BeaconState.hashTreeRoot - 250000 balances 107.37 ms/op 72.195 ms/op 1.49
aggregationBits - 2048 els - zipIndexesInBitList 29.160 us/op 23.435 us/op 1.24
regular array get 100000 times 60.561 us/op 46.624 us/op 1.30
wrappedArray get 100000 times 61.112 us/op 60.856 us/op 1.00
arrayWithProxy get 100000 times 28.482 ms/op 23.571 ms/op 1.21
ssz.Root.equals 510.00 ns/op 352.00 ns/op 1.45
byteArrayEquals 500.00 ns/op 346.00 ns/op 1.45
shuffle list - 16384 els 11.358 ms/op 7.6047 ms/op 1.49
shuffle list - 250000 els 166.59 ms/op 112.98 ms/op 1.47
processSlot - 1 slots 13.707 us/op 9.5220 us/op 1.44
processSlot - 32 slots 2.0270 ms/op 1.3631 ms/op 1.49
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 432.36 us/op 317.04 us/op 1.36
getCommitteeAssignments - req 1 vs - 250000 vc 5.4273 ms/op 3.6838 ms/op 1.47
getCommitteeAssignments - req 100 vs - 250000 vc 8.0050 ms/op 5.3760 ms/op 1.49
getCommitteeAssignments - req 1000 vs - 250000 vc 8.6266 ms/op 5.7688 ms/op 1.50
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 8.2300 ns/op 6.2100 ns/op 1.33
state getBlockRootAtSlot - 250000 vs - 7PWei 1.1645 us/op 669.32 ns/op 1.74
computeProposers - vc 250000 17.952 ms/op 12.207 ms/op 1.47
computeEpochShuffling - vc 250000 170.18 ms/op 114.54 ms/op 1.49
getNextSyncCommittee - vc 250000 300.04 ms/op 203.63 ms/op 1.47

by benchmarkbot/action

@shresthagrawal
Copy link

Thanks for taking the task! I guess you haven't added the hashing part yet? The hash should be simply the keccak256 of the concatenated public keys of the sync committee. Here is the current prover implementation

@twoeths twoeths force-pushed the tuyen/committee_hash_api branch 2 times, most recently from 74ce77a to e7ae483 Compare November 11, 2022 01:34
@twoeths twoeths mentioned this pull request Nov 11, 2022
Closed
@twoeths twoeths marked this pull request as ready for review November 11, 2022 02:54
@twoeths twoeths requested a review from a team as a code owner November 11, 2022 02:54
dapplion
dapplion previously requested changes Nov 11, 2022
View reviewed changes
packages/api/src/beacon/routes/lightclient.ts Outdated Show resolved Hide resolved
@dadepo
Copy link
Contributor

dadepo commented Nov 11, 2022

Maybe consider adding tests that covers the added endpoint? apart from that LGTM.

@twoeths
Copy link
Contributor Author

twoeths commented Nov 14, 2022

Maybe consider adding tests that covers the added endpoint? apart from that LGTM.

it already covered the end point as long as we update the testData for lightclient, doubled checked it via the unit test output

  beacon / lightclient
onError: Error: aborted
    ✔ getStateProof
    ✔ getUpdates
    ✔ getOptimisticUpdate
    ✔ getFinalityUpdate
    ✔ getBootstrap
    ✔ getCommitteeHash

@dapplion
Copy link
Contributor

dapplion commented Nov 14, 2022 •
edited
Loading

Thanks for taking the task! I guess you haven't added the hashing part yet? The hash should be simply the keccak256 of the concatenated public keys of the sync committee. Here is the current prover implementation

@shresthagrawal Why don't you do the hashTreeRoot of the SyncCommittee instead of concat pubkeys + hash? It's kindof tangent to your classic object hashing in the beacon chain. Plus the hashTreeRoot of the SyncCommittee is usually already available due to state hashing.

@shresthagrawal
Copy link

@dapplion yes that works, I already mentioned that in the GitHub issue earlier #4733 (comment)

@dadepo
Copy link
Contributor

dadepo commented Nov 14, 2022

Maybe consider adding tests that covers the added endpoint? apart from that LGTM.

it already covered the end point as long as we update the testData for lightclient, doubled checked it via the unit test output

  beacon / lightclient
onError: Error: aborted
    ✔ getStateProof
    ✔ getUpdates
    ✔ getOptimisticUpdate
    ✔ getFinalityUpdate
    ✔ getBootstrap
    ✔ getCommitteeHash

Unfortunately that test only seem to test some of the type machinery and not the actual implementation.

I updated the implementation to throw an error like this:

    async getCommitteeHash(startPeriod: SyncPeriod, count: number) {
      throw new Error("error");
      const maxAllowedCount = Math.min(MAX_REQUEST_LIGHT_CLIENT_COMMITTEE_HASHES, count);
      const periods = Array.from({length: maxAllowedCount}, (_ignored, i) => i + startPeriod);
      const committeeHashes = await Promise.all(
        periods.map((period) => chain.lightClientServer.getCommitteeHash(period))
      );
      return {data: committeeHashes};
    },

Ran the test and also got:

  beacon / lightclient
    ✔ getStateProof
    ✔ getUpdates
    ✔ getOptimisticUpdate
    ✔ getFinalityUpdate
    ✔ getBootstrap
    ✔ getCommitteeHash

You can also confirm this.

What will be needed is e2e test, and the few existing ones can be found in packages/beacon-node/test/e2e/api/impl/beacon

@twoeths
Copy link
Contributor Author

twoeths commented Nov 15, 2022

What will be needed is e2e test, and the few existing ones can be found in packages/beacon-node/test/e2e/api/impl/beacon

@dadepo thanks, implemented it

wemeetagain
wemeetagain approved these changes Nov 15, 2022
View reviewed changes
Copy link
Member

@wemeetagain wemeetagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wemeetagain wemeetagain merged commit 5cbcca0 into unstable Nov 15, 2022
@wemeetagain wemeetagain deleted the tuyen/committee_hash_api branch November 15, 2022 14:37
dapplion pushed a commit that referenced this pull request Nov 18, 2022
@twoeths @dapplion
Implement committee_hash lightclient api (#4736)
dbad2d7 
* Implement committee_hash lightclient api

* Fix check-types in light-client package

* Return committee hash instead of committee itself

* Fix lint in light-client

* Tested successfuly on local server

* Address PR comments

* Add lightclient e2e tests

* Switch to committee root
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion left review comments

@shresthagrawal shresthagrawal shresthagrawal left review comments

@dadepo dadepo dadepo left review comments

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@twoeths @shresthagrawal @dadepo @dapplion @wemeetagain