Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for justified block id in Beacon Node API #5459

Merged
merged 3 commits into from
May 8, 2023

Conversation

acuarica
Copy link
Contributor

@acuarica acuarica commented May 3, 2023

Motivation

This PR closes #4255.

Description

The ethereum-metrics-exporter uses justified as block id to retrieve block details https://ethereum.github.io/beacon-APIs/?urls.primaryName=v2.4.0#/Beacon/getBlockV2.

This PR adds support for the justified block id, even if it is not explicitly stated in the Beacon API.

@acuarica acuarica requested a review from a team as a code owner May 3, 2023 18:23
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions
Copy link
Contributor

github-actions bot commented May 3, 2023

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 622d8f5 Previous: 81b9998 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 757.52 us/op 785.80 us/op 0.96
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 46.992 us/op 43.888 us/op 1.07
BLS verify - blst-native 1.2058 ms/op 1.1703 ms/op 1.03
BLS verifyMultipleSignatures 3 - blst-native 2.4550 ms/op 2.3762 ms/op 1.03
BLS verifyMultipleSignatures 8 - blst-native 5.2979 ms/op 5.1025 ms/op 1.04
BLS verifyMultipleSignatures 32 - blst-native 19.135 ms/op 18.574 ms/op 1.03
BLS aggregatePubkeys 32 - blst-native 25.690 us/op 24.577 us/op 1.05
BLS aggregatePubkeys 128 - blst-native 100.44 us/op 96.296 us/op 1.04
getAttestationsForBlock 55.643 ms/op 50.807 ms/op 1.10
isKnown best case - 1 super set check 260.00 ns/op 254.00 ns/op 1.02
isKnown normal case - 2 super set checks 248.00 ns/op 251.00 ns/op 0.99
isKnown worse case - 16 super set checks 246.00 ns/op 248.00 ns/op 0.99
CheckpointStateCache - add get delete 4.9840 us/op 4.7910 us/op 1.04
validate gossip signedAggregateAndProof - struct 2.7464 ms/op 2.6810 ms/op 1.02
validate gossip attestation - struct 1.3518 ms/op 1.2616 ms/op 1.07
pickEth1Vote - no votes 1.3344 ms/op 1.2038 ms/op 1.11
pickEth1Vote - max votes 10.445 ms/op 11.233 ms/op 0.93
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.8759 ms/op 8.6673 ms/op 1.02
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.159 ms/op 14.652 ms/op 0.97
pickEth1Vote - Eth1Data fastSerialize value x2048 700.80 us/op 634.32 us/op 1.10
pickEth1Vote - Eth1Data fastSerialize tree x2048 4.6568 ms/op 7.6972 ms/op 0.60
bytes32 toHexString 477.00 ns/op 477.00 ns/op 1.00
bytes32 Buffer.toString(hex) 378.00 ns/op 336.00 ns/op 1.13
bytes32 Buffer.toString(hex) from Uint8Array 606.00 ns/op 539.00 ns/op 1.12
bytes32 Buffer.toString(hex) + 0x 428.00 ns/op 335.00 ns/op 1.28
Object access 1 prop 0.18700 ns/op 0.16100 ns/op 1.16
Map access 1 prop 0.16300 ns/op 0.15200 ns/op 1.07
Object get x1000 6.5280 ns/op 6.0340 ns/op 1.08
Map get x1000 0.56700 ns/op 0.57500 ns/op 0.99
Object set x1000 58.421 ns/op 48.935 ns/op 1.19
Map set x1000 48.696 ns/op 41.643 ns/op 1.17
Return object 10000 times 0.23990 ns/op 0.22680 ns/op 1.06
Throw Error 10000 times 4.1374 us/op 4.0412 us/op 1.02
fastMsgIdFn sha256 / 200 bytes 3.4850 us/op 3.3410 us/op 1.04
fastMsgIdFn h32 xxhash / 200 bytes 313.00 ns/op 269.00 ns/op 1.16
fastMsgIdFn h64 xxhash / 200 bytes 423.00 ns/op 378.00 ns/op 1.12
fastMsgIdFn sha256 / 1000 bytes 11.501 us/op 11.229 us/op 1.02
fastMsgIdFn h32 xxhash / 1000 bytes 418.00 ns/op 411.00 ns/op 1.02
fastMsgIdFn h64 xxhash / 1000 bytes 519.00 ns/op 447.00 ns/op 1.16
fastMsgIdFn sha256 / 10000 bytes 102.19 us/op 101.53 us/op 1.01
fastMsgIdFn h32 xxhash / 10000 bytes 1.9510 us/op 1.8990 us/op 1.03
fastMsgIdFn h64 xxhash / 10000 bytes 1.4660 us/op 1.3860 us/op 1.06
enrSubnets - fastDeserialize 64 bits 1.4350 us/op 1.2950 us/op 1.11
enrSubnets - ssz BitVector 64 bits 531.00 ns/op 481.00 ns/op 1.10
enrSubnets - fastDeserialize 4 bits 174.00 ns/op 169.00 ns/op 1.03
enrSubnets - ssz BitVector 4 bits 491.00 ns/op 474.00 ns/op 1.04
prioritizePeers score -10:0 att 32-0.1 sync 2-0 110.60 us/op 106.03 us/op 1.04
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 151.72 us/op 130.33 us/op 1.16
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 179.97 us/op 163.67 us/op 1.10
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 353.67 us/op 299.25 us/op 1.18
prioritizePeers score 0:0 att 64-1 sync 4-1 408.76 us/op 354.77 us/op 1.15
array of 16000 items push then shift 1.6322 us/op 1.6200 us/op 1.01
LinkedList of 16000 items push then shift 8.7490 ns/op 8.7520 ns/op 1.00
array of 16000 items push then pop 92.893 ns/op 95.967 ns/op 0.97
LinkedList of 16000 items push then pop 8.5780 ns/op 8.5530 ns/op 1.00
array of 24000 items push then shift 2.3934 us/op 2.3366 us/op 1.02
LinkedList of 24000 items push then shift 9.0420 ns/op 8.7250 ns/op 1.04
array of 24000 items push then pop 76.205 ns/op 84.167 ns/op 0.91
LinkedList of 24000 items push then pop 8.9950 ns/op 8.7370 ns/op 1.03
intersect bitArray bitLen 8 13.231 ns/op 13.208 ns/op 1.00
intersect array and set length 8 78.982 ns/op 81.232 ns/op 0.97
intersect bitArray bitLen 128 43.904 ns/op 45.157 ns/op 0.97
intersect array and set length 128 1.0554 us/op 1.0507 us/op 1.00
Buffer.concat 32 items 2.9560 us/op 2.8990 us/op 1.02
Uint8Array.set 32 items 2.9020 us/op 2.3660 us/op 1.23
pass gossip attestations to forkchoice per slot 3.1916 ms/op 3.0414 ms/op 1.05
computeDeltas 3.1108 ms/op 2.9987 ms/op 1.04
computeProposerBoostScoreFromBalances 1.7069 ms/op 1.8028 ms/op 0.95
altair processAttestation - 250000 vs - 7PWei normalcase 2.1303 ms/op 2.6547 ms/op 0.80
altair processAttestation - 250000 vs - 7PWei worstcase 3.2244 ms/op 3.7380 ms/op 0.86
altair processAttestation - setStatus - 1/6 committees join 134.98 us/op 142.30 us/op 0.95
altair processAttestation - setStatus - 1/3 committees join 267.86 us/op 286.77 us/op 0.93
altair processAttestation - setStatus - 1/2 committees join 367.70 us/op 410.02 us/op 0.90
altair processAttestation - setStatus - 2/3 committees join 463.47 us/op 468.80 us/op 0.99
altair processAttestation - setStatus - 4/5 committees join 643.98 us/op 664.15 us/op 0.97
altair processAttestation - setStatus - 100% committees join 724.46 us/op 764.05 us/op 0.95
altair processBlock - 250000 vs - 7PWei normalcase 18.504 ms/op 18.628 ms/op 0.99
altair processBlock - 250000 vs - 7PWei normalcase hashState 26.052 ms/op 27.326 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase 48.791 ms/op 51.143 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase hashState 70.105 ms/op 68.648 ms/op 1.02
phase0 processBlock - 250000 vs - 7PWei normalcase 1.9756 ms/op 2.0587 ms/op 0.96
phase0 processBlock - 250000 vs - 7PWei worstcase 27.888 ms/op 29.115 ms/op 0.96
altair processEth1Data - 250000 vs - 7PWei normalcase 474.63 us/op 460.27 us/op 1.03
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 6.8060 us/op 7.7260 us/op 0.88
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 20.655 us/op 24.942 us/op 0.83
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 8.6030 us/op 12.680 us/op 0.68
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 7.7760 us/op 9.0060 us/op 0.86
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 110.12 us/op 98.811 us/op 1.11
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 637.99 us/op 734.18 us/op 0.87
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 918.02 us/op 951.81 us/op 0.96
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 863.64 us/op 887.78 us/op 0.97
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3568 ms/op 3.3394 ms/op 0.71
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.7589 ms/op 1.7606 ms/op 1.00
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.9403 ms/op 3.9045 ms/op 1.01
Tree 40 250000 create 314.69 ms/op 320.08 ms/op 0.98
Tree 40 250000 get(125000) 194.94 ns/op 185.57 ns/op 1.05
Tree 40 250000 set(125000) 1.0359 us/op 1.0014 us/op 1.03
Tree 40 250000 toArray() 19.659 ms/op 19.300 ms/op 1.02
Tree 40 250000 iterate all - toArray() + loop 19.905 ms/op 21.277 ms/op 0.94
Tree 40 250000 iterate all - get(i) 71.997 ms/op 76.433 ms/op 0.94
MutableVector 250000 create 12.052 ms/op 11.839 ms/op 1.02
MutableVector 250000 get(125000) 6.2570 ns/op 6.3770 ns/op 0.98
MutableVector 250000 set(125000) 263.62 ns/op 261.03 ns/op 1.01
MutableVector 250000 toArray() 3.2547 ms/op 3.3237 ms/op 0.98
MutableVector 250000 iterate all - toArray() + loop 3.3530 ms/op 3.3440 ms/op 1.00
MutableVector 250000 iterate all - get(i) 1.5062 ms/op 1.5387 ms/op 0.98
Array 250000 create 3.5511 ms/op 3.1093 ms/op 1.14
Array 250000 clone - spread 1.3824 ms/op 1.1856 ms/op 1.17
Array 250000 get(125000) 0.88400 ns/op 0.58600 ns/op 1.51
Array 250000 set(125000) 0.92300 ns/op 0.67700 ns/op 1.36
Array 250000 iterate all - loop 86.864 us/op 83.620 us/op 1.04
effectiveBalanceIncrements clone Uint8Array 300000 51.610 us/op 28.089 us/op 1.84
effectiveBalanceIncrements clone MutableVector 300000 381.00 ns/op 365.00 ns/op 1.04
effectiveBalanceIncrements rw all Uint8Array 300000 173.56 us/op 165.68 us/op 1.05
effectiveBalanceIncrements rw all MutableVector 300000 105.69 ms/op 85.511 ms/op 1.24
phase0 afterProcessEpoch - 250000 vs - 7PWei 118.18 ms/op 116.06 ms/op 1.02
phase0 beforeProcessEpoch - 250000 vs - 7PWei 43.313 ms/op 42.266 ms/op 1.02
altair processEpoch - mainnet_e81889 356.09 ms/op 339.66 ms/op 1.05
mainnet_e81889 - altair beforeProcessEpoch 55.183 ms/op 74.122 ms/op 0.74
mainnet_e81889 - altair processJustificationAndFinalization 21.779 us/op 21.596 us/op 1.01
mainnet_e81889 - altair processInactivityUpdates 6.3180 ms/op 6.0586 ms/op 1.04
mainnet_e81889 - altair processRewardsAndPenalties 72.376 ms/op 48.169 ms/op 1.50
mainnet_e81889 - altair processRegistryUpdates 4.5820 us/op 2.6380 us/op 1.74
mainnet_e81889 - altair processSlashings 951.00 ns/op 631.00 ns/op 1.51
mainnet_e81889 - altair processEth1DataReset 889.00 ns/op 566.00 ns/op 1.57
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3694 ms/op 1.4519 ms/op 0.94
mainnet_e81889 - altair processSlashingsReset 7.2480 us/op 5.6150 us/op 1.29
mainnet_e81889 - altair processRandaoMixesReset 10.538 us/op 6.8640 us/op 1.54
mainnet_e81889 - altair processHistoricalRootsUpdate 2.5900 us/op 717.00 ns/op 3.61
mainnet_e81889 - altair processParticipationFlagUpdates 4.7060 us/op 5.0730 us/op 0.93
mainnet_e81889 - altair processSyncCommitteeUpdates 1.4890 us/op 621.00 ns/op 2.40
mainnet_e81889 - altair afterProcessEpoch 147.83 ms/op 131.99 ms/op 1.12
phase0 processEpoch - mainnet_e58758 414.43 ms/op 376.23 ms/op 1.10
mainnet_e58758 - phase0 beforeProcessEpoch 156.23 ms/op 153.46 ms/op 1.02
mainnet_e58758 - phase0 processJustificationAndFinalization 18.663 us/op 25.496 us/op 0.73
mainnet_e58758 - phase0 processRewardsAndPenalties 67.261 ms/op 69.201 ms/op 0.97
mainnet_e58758 - phase0 processRegistryUpdates 11.872 us/op 10.895 us/op 1.09
mainnet_e58758 - phase0 processSlashings 761.00 ns/op 494.00 ns/op 1.54
mainnet_e58758 - phase0 processEth1DataReset 967.00 ns/op 686.00 ns/op 1.41
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.1245 ms/op 1.6733 ms/op 0.67
mainnet_e58758 - phase0 processSlashingsReset 5.5420 us/op 5.4920 us/op 1.01
mainnet_e58758 - phase0 processRandaoMixesReset 6.1940 us/op 7.8630 us/op 0.79
mainnet_e58758 - phase0 processHistoricalRootsUpdate 752.00 ns/op 798.00 ns/op 0.94
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.2710 us/op 4.1800 us/op 1.02
mainnet_e58758 - phase0 afterProcessEpoch 103.88 ms/op 101.33 ms/op 1.03
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.4034 ms/op 1.2682 ms/op 1.11
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.8012 ms/op 1.6217 ms/op 1.11
altair processInactivityUpdates - 250000 normalcase 25.121 ms/op 19.457 ms/op 1.29
altair processInactivityUpdates - 250000 worstcase 25.707 ms/op 24.651 ms/op 1.04
phase0 processRegistryUpdates - 250000 normalcase 7.3730 us/op 7.3440 us/op 1.00
phase0 processRegistryUpdates - 250000 badcase_full_deposits 304.59 us/op 309.04 us/op 0.99
phase0 processRegistryUpdates - 250000 worstcase 0.5 123.07 ms/op 137.94 ms/op 0.89
altair processRewardsAndPenalties - 250000 normalcase 67.884 ms/op 64.442 ms/op 1.05
altair processRewardsAndPenalties - 250000 worstcase 67.215 ms/op 71.363 ms/op 0.94
phase0 getAttestationDeltas - 250000 normalcase 6.9374 ms/op 7.0702 ms/op 0.98
phase0 getAttestationDeltas - 250000 worstcase 7.0528 ms/op 6.9573 ms/op 1.01
phase0 processSlashings - 250000 worstcase 4.0524 ms/op 3.5724 ms/op 1.13
altair processSyncCommitteeUpdates - 250000 189.20 ms/op 191.78 ms/op 0.99
BeaconState.hashTreeRoot - No change 290.00 ns/op 274.00 ns/op 1.06
BeaconState.hashTreeRoot - 1 full validator 52.970 us/op 54.731 us/op 0.97
BeaconState.hashTreeRoot - 32 full validator 578.65 us/op 572.31 us/op 1.01
BeaconState.hashTreeRoot - 512 full validator 5.7984 ms/op 5.1010 ms/op 1.14
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 64.916 us/op 63.113 us/op 1.03
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 978.84 us/op 907.68 us/op 1.08
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 13.676 ms/op 12.605 ms/op 1.08
BeaconState.hashTreeRoot - 1 balances 52.457 us/op 48.281 us/op 1.09
BeaconState.hashTreeRoot - 32 balances 445.47 us/op 451.63 us/op 0.99
BeaconState.hashTreeRoot - 512 balances 4.9569 ms/op 4.3663 ms/op 1.14
BeaconState.hashTreeRoot - 250000 balances 77.861 ms/op 76.133 ms/op 1.02
aggregationBits - 2048 els - zipIndexesInBitList 18.134 us/op 16.160 us/op 1.12
regular array get 100000 times 33.418 us/op 43.451 us/op 0.77
wrappedArray get 100000 times 34.368 us/op 33.014 us/op 1.04
arrayWithProxy get 100000 times 18.295 ms/op 16.273 ms/op 1.12
ssz.Root.equals 617.00 ns/op 562.00 ns/op 1.10
byteArrayEquals 607.00 ns/op 558.00 ns/op 1.09
shuffle list - 16384 els 7.5711 ms/op 6.9073 ms/op 1.10
shuffle list - 250000 els 109.04 ms/op 101.37 ms/op 1.08
processSlot - 1 slots 9.6540 us/op 9.1060 us/op 1.06
processSlot - 32 slots 1.5951 ms/op 1.3823 ms/op 1.15
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 36.486 ms/op 34.790 ms/op 1.05
getCommitteeAssignments - req 1 vs - 250000 vc 3.0524 ms/op 2.8784 ms/op 1.06
getCommitteeAssignments - req 100 vs - 250000 vc 4.1971 ms/op 4.0977 ms/op 1.02
getCommitteeAssignments - req 1000 vs - 250000 vc 4.6170 ms/op 4.4639 ms/op 1.03
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.5800 ns/op 5.0300 ns/op 1.11
state getBlockRootAtSlot - 250000 vs - 7PWei 827.99 ns/op 616.04 ns/op 1.34
computeProposers - vc 250000 11.683 ms/op 10.837 ms/op 1.08
computeEpochShuffling - vc 250000 112.90 ms/op 103.80 ms/op 1.09
getNextSyncCommittee - vc 250000 203.49 ms/op 186.94 ms/op 1.09
computeSigningRoot for AttestationData 15.426 us/op 14.373 us/op 1.07
hash AttestationData serialized data then Buffer.toString(base64) 2.6716 us/op 2.4669 us/op 1.08
toHexString serialized data 1.6669 us/op 1.0733 us/op 1.55
Buffer.toString(base64) 366.65 ns/op 324.69 ns/op 1.13

by benchmarkbot/action

@acuarica acuarica changed the title Add support for justified block id in Beacon Node API feat: add support for justified block id in Beacon Node API May 3, 2023
it("should resolve justified", async function () {
const expected = 0;
forkChoiceStub.getJustifiedBlock.returns(expectedSummary);
await resolveBlockId(forkChoiceStub, dbStub, "justified").catch(() => {});
Copy link
Contributor

@dapplion dapplion May 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the silent catch all errors here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why, all other tests for resolveBlockId have the same structure, so I didn't want to make it different in case I'm missing something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reviewing this with @nazarhussain, we found that without the catch all these tests fail here

throw new ApiError(404, `No block found for id '${blockId}'`);

@@ -65,6 +65,13 @@ async function resolveBlockIdOrNull(
};
}

if (blockId === "justified") {
return {
block: await db.blockArchive.get(forkChoice.getJustifiedBlock().slot),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non finalized blocks are not in the archive, but in the block hot db. blockArchive will most likely return null for this query. Instead grab by root from the hot db.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed a fix for this, thanks @nazarhussain for helping me to validate the solution.

const justified = forkChoice.getJustifiedBlock();
return {
block: await db.block.get(fromHexString(justified.blockRoot)),
executionOptimistic: false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can compute the correct executionOptimistic boolean value with the forkChoice returned data structure. Check other API endpoints can't recall the exact helper fn name

const justified = forkChoice.getJustifiedBlock();
return {
block: await db.block.get(fromHexString(justified.blockRoot)),
executionOptimistic: isOptimisticBlock(justified),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @dapplion, is this what you meant regarding #5459 (comment)?

@wemeetagain wemeetagain merged commit c4a7fde into unstable May 8, 2023
@wemeetagain wemeetagain deleted the acuarica/justified-block-id branch May 8, 2023 14:09
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Invalid block id 'justified'
4 participants