Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify INetwork Interface #5187

Merged
merged 1 commit into from
Feb 23, 2023
Merged

Simplify INetwork Interface #5187

merged 1 commit into from
Feb 23, 2023

Conversation

wemeetagain
Copy link
Member

Motivation

Another step towards #4856
The INetwork class should be simplified in preparation for moving it to a worker.

Description

  • remove unnecessary methods/properties
    • many properties were internal details, unused, or factored away
  • replace large interfaces with small interfaces
    • create a simplified gossipsub interface, GossipsubBeaconNode
  • most methods become async/return promises
    • things that will be easy to retain outside the worker can stay sync
  • avoid reaching into deep internals in favor of dumpXXX methods

@wemeetagain wemeetagain requested a review from a team as a code owner February 21, 2023 01:07
@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 52f71e3 Previous: f1b8c5f Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 477.80 us/op 848.45 us/op 0.56
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 44.958 us/op 53.231 us/op 0.84
BLS verify - blst-native 1.2127 ms/op 1.2328 ms/op 0.98
BLS verifyMultipleSignatures 3 - blst-native 2.4596 ms/op 2.5074 ms/op 0.98
BLS verifyMultipleSignatures 8 - blst-native 5.2739 ms/op 5.4075 ms/op 0.98
BLS verifyMultipleSignatures 32 - blst-native 19.286 ms/op 19.580 ms/op 0.98
BLS aggregatePubkeys 32 - blst-native 25.922 us/op 26.469 us/op 0.98
BLS aggregatePubkeys 128 - blst-native 100.53 us/op 101.82 us/op 0.99
getAttestationsForBlock 56.591 ms/op 60.315 ms/op 0.94
isKnown best case - 1 super set check 256.00 ns/op 278.00 ns/op 0.92
isKnown normal case - 2 super set checks 249.00 ns/op 277.00 ns/op 0.90
isKnown worse case - 16 super set checks 260.00 ns/op 274.00 ns/op 0.95
CheckpointStateCache - add get delete 5.4130 us/op 5.3390 us/op 1.01
validate gossip signedAggregateAndProof - struct 2.7632 ms/op 2.8420 ms/op 0.97
validate gossip attestation - struct 1.3131 ms/op 1.3454 ms/op 0.98
pickEth1Vote - no votes 1.2674 ms/op 1.3653 ms/op 0.93
pickEth1Vote - max votes 10.277 ms/op 11.995 ms/op 0.86
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.4592 ms/op 9.5803 ms/op 0.99
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.589 ms/op 15.806 ms/op 0.92
pickEth1Vote - Eth1Data fastSerialize value x2048 635.26 us/op 744.46 us/op 0.85
pickEth1Vote - Eth1Data fastSerialize tree x2048 4.5106 ms/op 7.2013 ms/op 0.63
bytes32 toHexString 490.00 ns/op 551.00 ns/op 0.89
bytes32 Buffer.toString(hex) 339.00 ns/op 404.00 ns/op 0.84
bytes32 Buffer.toString(hex) from Uint8Array 549.00 ns/op 626.00 ns/op 0.88
bytes32 Buffer.toString(hex) + 0x 352.00 ns/op 432.00 ns/op 0.81
Object access 1 prop 0.17400 ns/op 0.19600 ns/op 0.89
Map access 1 prop 0.16800 ns/op 0.16100 ns/op 1.04
Object get x1000 6.5390 ns/op 6.9290 ns/op 0.94
Map get x1000 0.59400 ns/op 0.67400 ns/op 0.88
Object set x1000 51.633 ns/op 59.324 ns/op 0.87
Map set x1000 42.488 ns/op 51.997 ns/op 0.82
Return object 10000 times 0.23280 ns/op 0.24740 ns/op 0.94
Throw Error 10000 times 4.0449 us/op 4.3189 us/op 0.94
fastMsgIdFn sha256 / 200 bytes 3.4330 us/op 3.6050 us/op 0.95
fastMsgIdFn h32 xxhash / 200 bytes 275.00 ns/op 307.00 ns/op 0.90
fastMsgIdFn h64 xxhash / 200 bytes 376.00 ns/op 454.00 ns/op 0.83
fastMsgIdFn sha256 / 1000 bytes 11.549 us/op 11.838 us/op 0.98
fastMsgIdFn h32 xxhash / 1000 bytes 397.00 ns/op 426.00 ns/op 0.93
fastMsgIdFn h64 xxhash / 1000 bytes 454.00 ns/op 506.00 ns/op 0.90
fastMsgIdFn sha256 / 10000 bytes 103.85 us/op 104.69 us/op 0.99
fastMsgIdFn h32 xxhash / 10000 bytes 1.8710 us/op 1.9880 us/op 0.94
fastMsgIdFn h64 xxhash / 10000 bytes 1.3360 us/op 1.4630 us/op 0.91
enrSubnets - fastDeserialize 64 bits 1.3530 us/op 1.4040 us/op 0.96
enrSubnets - ssz BitVector 64 bits 532.00 ns/op 541.00 ns/op 0.98
enrSubnets - fastDeserialize 4 bits 183.00 ns/op 211.00 ns/op 0.87
enrSubnets - ssz BitVector 4 bits 513.00 ns/op 570.00 ns/op 0.90
prioritizePeers score -10:0 att 32-0.1 sync 2-0 101.24 us/op 107.93 us/op 0.94
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 126.80 us/op 131.37 us/op 0.97
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 182.54 us/op 186.52 us/op 0.98
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 344.73 us/op 339.09 us/op 1.02
prioritizePeers score 0:0 att 64-1 sync 4-1 362.12 us/op 400.52 us/op 0.90
array of 16000 items push then shift 1.5732 us/op 1.7134 us/op 0.92
LinkedList of 16000 items push then shift 8.7550 ns/op 9.2640 ns/op 0.95
array of 16000 items push then pop 81.260 ns/op 108.06 ns/op 0.75
LinkedList of 16000 items push then pop 8.7300 ns/op 8.9980 ns/op 0.97
array of 24000 items push then shift 2.4142 us/op 2.4382 us/op 0.99
LinkedList of 24000 items push then shift 9.0220 ns/op 9.0700 ns/op 0.99
array of 24000 items push then pop 84.524 ns/op 83.356 ns/op 1.01
LinkedList of 24000 items push then pop 8.9190 ns/op 8.9380 ns/op 1.00
intersect bitArray bitLen 8 13.582 ns/op 13.630 ns/op 1.00
intersect array and set length 8 82.381 ns/op 88.599 ns/op 0.93
intersect bitArray bitLen 128 44.561 ns/op 44.592 ns/op 1.00
intersect array and set length 128 1.1450 us/op 1.1766 us/op 0.97
Buffer.concat 32 items 2.7320 us/op 3.0150 us/op 0.91
Uint8Array.set 32 items 3.0140 us/op 2.5160 us/op 1.20
pass gossip attestations to forkchoice per slot 2.3404 ms/op 2.3785 ms/op 0.98
computeDeltas 2.9962 ms/op 2.9615 ms/op 1.01
computeProposerBoostScoreFromBalances 1.7836 ms/op 1.8030 ms/op 0.99
altair processAttestation - 250000 vs - 7PWei normalcase 2.1653 ms/op 2.1806 ms/op 0.99
altair processAttestation - 250000 vs - 7PWei worstcase 3.4286 ms/op 4.4699 ms/op 0.77
altair processAttestation - setStatus - 1/6 committees join 139.54 us/op 147.92 us/op 0.94
altair processAttestation - setStatus - 1/3 committees join 278.88 us/op 280.99 us/op 0.99
altair processAttestation - setStatus - 1/2 committees join 373.77 us/op 386.25 us/op 0.97
altair processAttestation - setStatus - 2/3 committees join 472.94 us/op 484.06 us/op 0.98
altair processAttestation - setStatus - 4/5 committees join 657.01 us/op 674.81 us/op 0.97
altair processAttestation - setStatus - 100% committees join 771.50 us/op 787.12 us/op 0.98
altair processBlock - 250000 vs - 7PWei normalcase 18.512 ms/op 19.310 ms/op 0.96
altair processBlock - 250000 vs - 7PWei normalcase hashState 25.253 ms/op 26.925 ms/op 0.94
altair processBlock - 250000 vs - 7PWei worstcase 54.965 ms/op 51.591 ms/op 1.07
altair processBlock - 250000 vs - 7PWei worstcase hashState 68.008 ms/op 73.810 ms/op 0.92
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0718 ms/op 2.1923 ms/op 0.95
phase0 processBlock - 250000 vs - 7PWei worstcase 29.975 ms/op 29.166 ms/op 1.03
altair processEth1Data - 250000 vs - 7PWei normalcase 512.48 us/op 518.56 us/op 0.99
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 8.2670 us/op 11.245 us/op 0.74
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 20.461 us/op 30.520 us/op 0.67
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 9.3830 us/op 10.526 us/op 0.89
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 7.0360 us/op 8.0980 us/op 0.87
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 106.62 us/op 105.19 us/op 1.01
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 663.93 us/op 726.55 us/op 0.91
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 904.96 us/op 915.06 us/op 0.99
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 905.31 us/op 894.38 us/op 1.01
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.4253 ms/op 2.3781 ms/op 1.02
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.5634 ms/op 1.7138 ms/op 0.91
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.8103 ms/op 3.9885 ms/op 0.96
Tree 40 250000 create 302.97 ms/op 316.17 ms/op 0.96
Tree 40 250000 get(125000) 179.11 ns/op 198.20 ns/op 0.90
Tree 40 250000 set(125000) 961.33 ns/op 1.0324 us/op 0.93
Tree 40 250000 toArray() 17.451 ms/op 19.877 ms/op 0.88
Tree 40 250000 iterate all - toArray() + loop 18.608 ms/op 20.137 ms/op 0.92
Tree 40 250000 iterate all - get(i) 71.091 ms/op 70.043 ms/op 1.01
MutableVector 250000 create 10.456 ms/op 10.092 ms/op 1.04
MutableVector 250000 get(125000) 6.4950 ns/op 6.3760 ns/op 1.02
MutableVector 250000 set(125000) 263.12 ns/op 254.27 ns/op 1.03
MutableVector 250000 toArray() 3.3077 ms/op 2.9913 ms/op 1.11
MutableVector 250000 iterate all - toArray() + loop 3.2807 ms/op 3.0209 ms/op 1.09
MutableVector 250000 iterate all - get(i) 1.5255 ms/op 1.5840 ms/op 0.96
Array 250000 create 2.7632 ms/op 2.5910 ms/op 1.07
Array 250000 clone - spread 1.1874 ms/op 1.1358 ms/op 1.05
Array 250000 get(125000) 0.60100 ns/op 0.58400 ns/op 1.03
Array 250000 set(125000) 0.68600 ns/op 0.65700 ns/op 1.04
Array 250000 iterate all - loop 112.87 us/op 84.291 us/op 1.34
effectiveBalanceIncrements clone Uint8Array 300000 30.327 us/op 29.511 us/op 1.03
effectiveBalanceIncrements clone MutableVector 300000 374.00 ns/op 371.00 ns/op 1.01
effectiveBalanceIncrements rw all Uint8Array 300000 170.50 us/op 170.03 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 82.922 ms/op 84.710 ms/op 0.98
phase0 afterProcessEpoch - 250000 vs - 7PWei 113.29 ms/op 116.74 ms/op 0.97
phase0 beforeProcessEpoch - 250000 vs - 7PWei 35.199 ms/op 39.227 ms/op 0.90
altair processEpoch - mainnet_e81889 296.03 ms/op 333.96 ms/op 0.89
mainnet_e81889 - altair beforeProcessEpoch 60.479 ms/op 68.056 ms/op 0.89
mainnet_e81889 - altair processJustificationAndFinalization 15.757 us/op 18.533 us/op 0.85
mainnet_e81889 - altair processInactivityUpdates 5.4256 ms/op 5.7018 ms/op 0.95
mainnet_e81889 - altair processRewardsAndPenalties 55.190 ms/op 71.585 ms/op 0.77
mainnet_e81889 - altair processRegistryUpdates 2.8240 us/op 2.5780 us/op 1.10
mainnet_e81889 - altair processSlashings 680.00 ns/op 577.00 ns/op 1.18
mainnet_e81889 - altair processEth1DataReset 573.00 ns/op 1.0110 us/op 0.57
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2252 ms/op 1.2751 ms/op 0.96
mainnet_e81889 - altair processSlashingsReset 4.8660 us/op 7.1350 us/op 0.68
mainnet_e81889 - altair processRandaoMixesReset 4.4720 us/op 8.3560 us/op 0.54
mainnet_e81889 - altair processHistoricalRootsUpdate 663.00 ns/op 963.00 ns/op 0.69
mainnet_e81889 - altair processParticipationFlagUpdates 2.4000 us/op 3.0820 us/op 0.78
mainnet_e81889 - altair processSyncCommitteeUpdates 433.00 ns/op 1.1130 us/op 0.39
mainnet_e81889 - altair afterProcessEpoch 112.55 ms/op 132.07 ms/op 0.85
phase0 processEpoch - mainnet_e58758 315.30 ms/op 393.56 ms/op 0.80
mainnet_e58758 - phase0 beforeProcessEpoch 124.01 ms/op 163.27 ms/op 0.76
mainnet_e58758 - phase0 processJustificationAndFinalization 15.592 us/op 18.298 us/op 0.85
mainnet_e58758 - phase0 processRewardsAndPenalties 40.479 ms/op 68.510 ms/op 0.59
mainnet_e58758 - phase0 processRegistryUpdates 7.6000 us/op 8.4360 us/op 0.90
mainnet_e58758 - phase0 processSlashings 505.00 ns/op 502.00 ns/op 1.01
mainnet_e58758 - phase0 processEth1DataReset 500.00 ns/op 498.00 ns/op 1.00
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 957.99 us/op 1.0261 ms/op 0.93
mainnet_e58758 - phase0 processSlashingsReset 3.1730 us/op 4.0700 us/op 0.78
mainnet_e58758 - phase0 processRandaoMixesReset 4.5220 us/op 5.0540 us/op 0.89
mainnet_e58758 - phase0 processHistoricalRootsUpdate 572.00 ns/op 655.00 ns/op 0.87
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.2040 us/op 4.4450 us/op 0.95
mainnet_e58758 - phase0 afterProcessEpoch 96.819 ms/op 99.381 ms/op 0.97
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2271 ms/op 1.2566 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4200 ms/op 1.6494 ms/op 0.86
altair processInactivityUpdates - 250000 normalcase 19.451 ms/op 26.821 ms/op 0.73
altair processInactivityUpdates - 250000 worstcase 22.637 ms/op 29.907 ms/op 0.76
phase0 processRegistryUpdates - 250000 normalcase 6.6040 us/op 6.4110 us/op 1.03
phase0 processRegistryUpdates - 250000 badcase_full_deposits 248.95 us/op 271.61 us/op 0.92
phase0 processRegistryUpdates - 250000 worstcase 0.5 116.71 ms/op 127.59 ms/op 0.91
altair processRewardsAndPenalties - 250000 normalcase 49.820 ms/op 70.922 ms/op 0.70
altair processRewardsAndPenalties - 250000 worstcase 48.299 ms/op 70.203 ms/op 0.69
phase0 getAttestationDeltas - 250000 normalcase 6.4536 ms/op 6.8454 ms/op 0.94
phase0 getAttestationDeltas - 250000 worstcase 6.4795 ms/op 6.7463 ms/op 0.96
phase0 processSlashings - 250000 worstcase 3.6055 ms/op 3.5313 ms/op 1.02
altair processSyncCommitteeUpdates - 250000 170.48 ms/op 180.01 ms/op 0.95
BeaconState.hashTreeRoot - No change 266.00 ns/op 272.00 ns/op 0.98
BeaconState.hashTreeRoot - 1 full validator 51.429 us/op 55.574 us/op 0.93
BeaconState.hashTreeRoot - 32 full validator 494.74 us/op 532.16 us/op 0.93
BeaconState.hashTreeRoot - 512 full validator 4.9018 ms/op 5.4378 ms/op 0.90
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 61.556 us/op 64.288 us/op 0.96
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 867.70 us/op 920.66 us/op 0.94
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.024 ms/op 12.156 ms/op 0.91
BeaconState.hashTreeRoot - 1 balances 50.286 us/op 53.909 us/op 0.93
BeaconState.hashTreeRoot - 32 balances 435.49 us/op 489.94 us/op 0.89
BeaconState.hashTreeRoot - 512 balances 4.2891 ms/op 4.5711 ms/op 0.94
BeaconState.hashTreeRoot - 250000 balances 71.776 ms/op 74.140 ms/op 0.97
aggregationBits - 2048 els - zipIndexesInBitList 15.352 us/op 16.131 us/op 0.95
regular array get 100000 times 33.269 us/op 33.763 us/op 0.99
wrappedArray get 100000 times 32.268 us/op 33.373 us/op 0.97
arrayWithProxy get 100000 times 15.331 ms/op 16.150 ms/op 0.95
ssz.Root.equals 543.00 ns/op 544.00 ns/op 1.00
byteArrayEquals 532.00 ns/op 540.00 ns/op 0.99
shuffle list - 16384 els 6.6749 ms/op 6.9586 ms/op 0.96
shuffle list - 250000 els 97.995 ms/op 102.52 ms/op 0.96
processSlot - 1 slots 8.6060 us/op 8.4830 us/op 1.01
processSlot - 32 slots 1.3964 ms/op 1.3217 ms/op 1.06
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 188.68 us/op 192.30 us/op 0.98
getCommitteeAssignments - req 1 vs - 250000 vc 2.8945 ms/op 2.9487 ms/op 0.98
getCommitteeAssignments - req 100 vs - 250000 vc 4.1358 ms/op 4.1904 ms/op 0.99
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4454 ms/op 4.4937 ms/op 0.99
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.4300 ns/op 4.7600 ns/op 0.93
state getBlockRootAtSlot - 250000 vs - 7PWei 754.80 ns/op 549.02 ns/op 1.37
computeProposers - vc 250000 10.304 ms/op 10.433 ms/op 0.99
computeEpochShuffling - vc 250000 101.09 ms/op 102.27 ms/op 0.99
getNextSyncCommittee - vc 250000 170.24 ms/op 173.16 ms/op 0.98

by benchmarkbot/action

Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does look a lot cleaner and I did not find any big issues but can't really approve this as I am not familiar with the code and the impact of those changes.

@@ -122,6 +122,21 @@ export type GossipModules = {
chain: IBeaconChain;
};

export type GossipBeaconNode = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this rather be an interface?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely could be an interface. I don't know that we have clear project-wide opinions on the use of one vs the either.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO if it is implemented by a class within the project directly or even outside in an external library I would use interface otherwise should just use type

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, type here is a bit sloppy. Can clean it up in a future PR.
I'm continuing to do some work to clean this up, aiming to move the network to a worker, so will continue touching these pieces of code.

Comment on lines +52 to +58
networkStub.getMetadata.returns(
Promise.resolve({
attnets: BitArray.fromBoolArray([true]),
syncnets: BitArray.fromBitLen(0),
seqNumber: BigInt(1),
})
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might just be able to do

Suggested change
networkStub.getMetadata.returns(
Promise.resolve({
attnets: BitArray.fromBoolArray([true]),
syncnets: BitArray.fromBitLen(0),
seqNumber: BigInt(1),
})
);
networkStub.getMetadata.resolves(
attnets: BitArray.fromBoolArray([true]),
syncnets: BitArray.fromBitLen(0),
seqNumber: BigInt(1),
);

there are more cases below

@dapplion
Copy link
Contributor

Noted we should probably have two classes, one interacting with everything else in the main thread and one for the worker - main thread comms

Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@wemeetagain wemeetagain merged commit 93fc4ee into unstable Feb 23, 2023
@wemeetagain wemeetagain deleted the cayman/network-interface branch February 23, 2023 19:37
@wemeetagain
Copy link
Member Author

🎉 This PR is included in v1.6.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants