Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: snapshot apis for EIP-4881 #400

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

feat: snapshot apis for EIP-4881 #400

wants to merge 12 commits into from

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Sep 9, 2024

Motivation

Implement snapshot apis for eip-4881

Description

  • persistent-merkle-tree: implement toSnapshot/fromSnapshot apis
  • ssz:
    • define new PartialListCompositeType type along with ViewDU (and no View since we lacks a lot of methods there and dont maintain it anymore)
    • new toPartialViewDU() api as the entry point
    • consume persistent-merkle-tree toSnapshot, fromSnapshot apis
  • notes for consumers (lodestar)
    • new DepositDataRootPartialList = new PartialListCompositeType(Root, 2 ** DEPOSIT_CONTRACT_TREE_DEPTH);
    • download snapshot
    • then create the deposit root tree like depositRootTree = DepositDataRootPartialList.toPartialViewDU(snapshot)

Closes #293
a candidate to replace #377

TODOs

  • spec tests

Copy link

github-actions bot commented Sep 9, 2024

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: d119dcc Previous: 1dc50ef Ratio
digestTwoHashObjects 50023 times 47.951 ms/op 48.133 ms/op 1.00
digest64 50023 times 53.708 ms/op 53.792 ms/op 1.00
digest 50023 times 56.108 ms/op 57.269 ms/op 0.98
input length 32 1.2440 us/op 1.2580 us/op 0.99
input length 64 1.3600 us/op 1.3750 us/op 0.99
input length 128 2.2890 us/op 2.3320 us/op 0.98
input length 256 3.4130 us/op 3.6010 us/op 0.95
input length 512 5.8380 us/op 5.8540 us/op 1.00
input length 1024 11.125 us/op 11.249 us/op 0.99
digest 1000000 times 926.49 ms/op 928.44 ms/op 1.00
hashObjectToByteArray 50023 times 1.4275 ms/op 1.4270 ms/op 1.00
byteArrayToHashObject 50023 times 2.5223 ms/op 2.5396 ms/op 0.99
digest64 200092 times 222.91 ms/op 233.71 ms/op 0.95
hash 200092 times using batchHash4UintArray64s 245.10 ms/op 262.76 ms/op 0.93
digest64HashObjects 200092 times 197.32 ms/op 197.43 ms/op 1.00
hash 200092 times using batchHash4HashObjectInputs 203.33 ms/op 206.47 ms/op 0.98
getGindicesAtDepth 4.3250 us/op 4.3090 us/op 1.00
iterateAtDepth 7.7100 us/op 7.6000 us/op 1.01
getGindexBits 482.00 ns/op 473.00 ns/op 1.02
gindexIterator 1.0550 us/op 1.0450 us/op 1.01
HashComputationLevel.push then loop 26.215 ms/op 28.201 ms/op 0.93
HashComputation[] push then loop 48.587 ms/op 49.163 ms/op 0.99
hash 2 Uint8Array 500000 times - as-sha256 551.52 ms/op 548.74 ms/op 1.01
hashTwoObjects 500000 times - as-sha256 501.92 ms/op 500.19 ms/op 1.00
executeHashComputations - as-sha256 46.774 ms/op 48.276 ms/op 0.97
hash 2 Uint8Array 500000 times - noble 1.1138 s/op 1.1227 s/op 0.99
hashTwoObjects 500000 times - noble 1.6089 s/op 1.6548 s/op 0.97
executeHashComputations - noble 40.522 ms/op 42.226 ms/op 0.96
hash 2 Uint8Array 500000 times - hashtree 231.33 ms/op 232.33 ms/op 1.00
hashTwoObjects 500000 times - hashtree 213.22 ms/op 210.23 ms/op 1.01
executeHashComputations - hashtree 11.268 ms/op 11.083 ms/op 1.02
getHashComputations 3.1522 ms/op 2.9669 ms/op 1.06
executeHashComputations 11.461 ms/op 12.854 ms/op 0.89
get root 17.415 ms/op 17.981 ms/op 0.97
getNodeH() x7812.5 avg hindex 11.955 us/op 11.952 us/op 1.00
getNodeH() x7812.5 index 0 6.2950 us/op 6.2460 us/op 1.01
getNodeH() x7812.5 index 7 6.2730 us/op 6.3690 us/op 0.98
getNodeH() x7812.5 index 7 with key array 6.2620 us/op 6.2960 us/op 0.99
new LeafNode() x7812.5 14.669 us/op 14.752 us/op 0.99
getHashComputations 250000 nodes 20.195 ms/op 21.030 ms/op 0.96
batchHash 250000 nodes 87.087 ms/op 94.098 ms/op 0.93
get root 250000 nodes 117.71 ms/op 119.27 ms/op 0.99
getHashComputations 500000 nodes 30.397 ms/op 32.594 ms/op 0.93
batchHash 500000 nodes 157.47 ms/op 172.95 ms/op 0.91
get root 500000 nodes 233.25 ms/op 249.47 ms/op 0.93
getHashComputations 1000000 nodes 66.292 ms/op 77.094 ms/op 0.86
batchHash 1000000 nodes 358.94 ms/op 367.39 ms/op 0.98
get root 1000000 nodes 466.05 ms/op 489.52 ms/op 0.95
multiproof - depth 15, 1 requested leaves 9.4970 us/op 8.5110 us/op 1.12
tree offset multiproof - depth 15, 1 requested leaves 20.745 us/op 18.203 us/op 1.14
compact multiproof - depth 15, 1 requested leaves 3.8510 us/op 3.5740 us/op 1.08
multiproof - depth 15, 2 requested leaves 14.737 us/op 12.007 us/op 1.23
tree offset multiproof - depth 15, 2 requested leaves 24.754 us/op 22.849 us/op 1.08
compact multiproof - depth 15, 2 requested leaves 4.3800 us/op 3.7190 us/op 1.18
multiproof - depth 15, 3 requested leaves 20.272 us/op 16.744 us/op 1.21
tree offset multiproof - depth 15, 3 requested leaves 32.106 us/op 29.753 us/op 1.08
compact multiproof - depth 15, 3 requested leaves 5.4770 us/op 4.3070 us/op 1.27
multiproof - depth 15, 4 requested leaves 24.742 us/op 23.344 us/op 1.06
tree offset multiproof - depth 15, 4 requested leaves 39.025 us/op 36.846 us/op 1.06
compact multiproof - depth 15, 4 requested leaves 6.3410 us/op 5.8380 us/op 1.09
packedRootsBytesToLeafNodes bytes 4000 offset 0 2.2870 us/op 2.0270 us/op 1.13
packedRootsBytesToLeafNodes bytes 4000 offset 1 2.2900 us/op 1.9980 us/op 1.15
packedRootsBytesToLeafNodes bytes 4000 offset 2 2.1960 us/op 2.0120 us/op 1.09
packedRootsBytesToLeafNodes bytes 4000 offset 3 2.2540 us/op 2.0840 us/op 1.08
subtreeFillToContents depth 40 count 250000 43.024 ms/op 48.258 ms/op 0.89
setRoot - gindexBitstring 10.231 ms/op 10.272 ms/op 1.00
setRoot - gindex 10.514 ms/op 11.098 ms/op 0.95
getRoot - gindexBitstring 2.6653 ms/op 2.6634 ms/op 1.00
getRoot - gindex 3.5238 ms/op 3.5613 ms/op 0.99
getHashObject then setHashObject 10.508 ms/op 11.668 ms/op 0.90
setNodeWithFn 8.1469 ms/op 9.6807 ms/op 0.84
getNodeAtDepth depth 0 x100000 1.1139 ms/op 1.1138 ms/op 1.00
setNodeAtDepth depth 0 x100000 3.0447 ms/op 2.8359 ms/op 1.07
getNodesAtDepth depth 0 x100000 1.0535 ms/op 1.0533 ms/op 1.00
setNodesAtDepth depth 0 x100000 1.5161 ms/op 1.5176 ms/op 1.00
getNodeAtDepth depth 1 x100000 1.1970 ms/op 1.1837 ms/op 1.01
setNodeAtDepth depth 1 x100000 5.3963 ms/op 6.0382 ms/op 0.89
getNodesAtDepth depth 1 x100000 1.1903 ms/op 1.1800 ms/op 1.01
setNodesAtDepth depth 1 x100000 4.6893 ms/op 6.2059 ms/op 0.76
getNodeAtDepth depth 2 x100000 1.4833 ms/op 1.4547 ms/op 1.02
setNodeAtDepth depth 2 x100000 9.3417 ms/op 10.217 ms/op 0.91
getNodesAtDepth depth 2 x100000 19.151 ms/op 21.592 ms/op 0.89
setNodesAtDepth depth 2 x100000 14.432 ms/op 16.512 ms/op 0.87
tree.getNodesAtDepth - gindexes 9.1649 ms/op 8.8271 ms/op 1.04
tree.getNodesAtDepth - push all nodes 2.0024 ms/op 2.3076 ms/op 0.87
tree.getNodesAtDepth - navigation 232.50 us/op 243.84 us/op 0.95
tree.setNodesAtDepth - indexes 492.78 us/op 457.54 us/op 1.08
set at depth 8 508.00 ns/op 512.00 ns/op 0.99
set at depth 16 629.00 ns/op 695.00 ns/op 0.91
set at depth 32 979.00 ns/op 1.1500 us/op 0.85
iterateNodesAtDepth 8 256 14.443 us/op 14.931 us/op 0.97
getNodesAtDepth 8 256 3.6270 us/op 3.7970 us/op 0.96
iterateNodesAtDepth 16 65536 4.5080 ms/op 4.7069 ms/op 0.96
getNodesAtDepth 16 65536 1.5610 ms/op 1.9283 ms/op 0.81
iterateNodesAtDepth 32 250000 16.128 ms/op 16.770 ms/op 0.96
getNodesAtDepth 32 250000 4.4259 ms/op 4.6775 ms/op 0.95
iterateNodesAtDepth 40 250000 15.799 ms/op 17.133 ms/op 0.92
getNodesAtDepth 40 250000 4.4616 ms/op 8.1189 ms/op 0.55
250000 validators root getter 117.63 ms/op 121.88 ms/op 0.97
250000 validators batchHash() 85.657 ms/op 109.73 ms/op 0.78
250000 validators hashComputations 14.254 ms/op 17.362 ms/op 0.82
bitlist bytes to struct (120,90) 955.00 ns/op 998.00 ns/op 0.96
bitlist bytes to tree (120,90) 3.7570 us/op 3.8180 us/op 0.98
bitlist bytes to struct (2048,2048) 1.3360 us/op 1.4070 us/op 0.95
bitlist bytes to tree (2048,2048) 4.1810 us/op 4.4550 us/op 0.94
ByteListType - deserialize 8.6924 ms/op 9.1992 ms/op 0.94
BasicListType - deserialize 17.616 ms/op 18.276 ms/op 0.96
ByteListType - serialize 8.4616 ms/op 8.0877 ms/op 1.05
BasicListType - serialize 11.155 ms/op 11.164 ms/op 1.00
BasicListType - tree_convertToStruct 30.143 ms/op 30.343 ms/op 0.99
List[uint8, 68719476736] len 300000 ViewDU.getAll() + iterate 4.8874 ms/op 4.8622 ms/op 1.01
List[uint8, 68719476736] len 300000 ViewDU.get(i) 4.0702 ms/op 3.9857 ms/op 1.02
Array.push len 300000 empty Array - number 7.0436 ms/op 7.6708 ms/op 0.92
Array.set len 300000 from new Array - number 2.2548 ms/op 1.8425 ms/op 1.22
Array.set len 300000 - number 6.5625 ms/op 6.4571 ms/op 1.02
Uint8Array.set len 300000 382.83 us/op 393.08 us/op 0.97
Uint32Array.set len 300000 456.03 us/op 488.37 us/op 0.93
Container({a: uint8, b: uint8}) getViewDU x300000 46.557 ms/op 50.716 ms/op 0.92
ContainerNodeStruct({a: uint8, b: uint8}) getViewDU x300000 11.186 ms/op 11.636 ms/op 0.96
List(Container) len 300000 ViewDU.getAllReadonly() + iterate 220.81 ms/op 239.62 ms/op 0.92
List(Container) len 300000 ViewDU.getAllReadonlyValues() + iterate 262.29 ms/op 321.41 ms/op 0.82
List(Container) len 300000 ViewDU.get(i) 7.0460 ms/op 6.9628 ms/op 1.01
List(Container) len 300000 ViewDU.getReadonly(i) 6.5498 ms/op 6.8634 ms/op 0.95
List(ContainerNodeStruct) len 300000 ViewDU.getAllReadonly() + iterate 36.678 ms/op 37.302 ms/op 0.98
List(ContainerNodeStruct) len 300000 ViewDU.getAllReadonlyValues() + iterate 5.5840 ms/op 8.2226 ms/op 0.68
List(ContainerNodeStruct) len 300000 ViewDU.get(i) 6.3838 ms/op 6.5813 ms/op 0.97
List(ContainerNodeStruct) len 300000 ViewDU.getReadonly(i) 6.1845 ms/op 6.4747 ms/op 0.96
Array.push len 300000 empty Array - object 6.9517 ms/op 7.6359 ms/op 0.91
Array.set len 300000 from new Array - object 2.3696 ms/op 2.5378 ms/op 0.93
Array.set len 300000 - object 7.1413 ms/op 8.3293 ms/op 0.86
cachePermanentRootStruct no cache 5.4400 us/op 5.8100 us/op 0.94
cachePermanentRootStruct with cache 223.00 ns/op 261.00 ns/op 0.85
epochParticipation len 250000 rws 7813 2.3114 ms/op 2.3850 ms/op 0.97
BeaconState ViewDU hashTreeRoot() vc=200000 104.69 ms/op 104.21 ms/op 1.00
BeaconState ViewDU recursive hash - commit step vc=200000 4.2898 ms/op 5.1522 ms/op 0.83
BeaconState ViewDU validator tree creation vc=10000 36.316 ms/op 37.073 ms/op 0.98
BeaconState ViewDU batchHashTreeRoot vc=200000 93.944 ms/op 96.108 ms/op 0.98
BeaconState ViewDU hashTreeRoot - commit step vc=200000 84.123 ms/op 82.669 ms/op 1.02
BeaconState ViewDU hashTreeRoot - hash step vc=200000 16.095 ms/op 16.121 ms/op 1.00
deserialize Attestation - tree 4.3360 us/op 4.6290 us/op 0.94
deserialize Attestation - struct 2.0960 us/op 1.9490 us/op 1.08
deserialize SignedAggregateAndProof - tree 4.0760 us/op 3.9820 us/op 1.02
deserialize SignedAggregateAndProof - struct 3.0350 us/op 3.1310 us/op 0.97
deserialize SyncCommitteeMessage - tree 1.1160 us/op 1.0940 us/op 1.02
deserialize SyncCommitteeMessage - struct 1.1870 us/op 1.0870 us/op 1.09
deserialize SignedContributionAndProof - tree 2.1760 us/op 2.1800 us/op 1.00
deserialize SignedContributionAndProof - struct 2.3990 us/op 2.3870 us/op 1.01
deserialize SignedBeaconBlock - tree 229.82 us/op 220.70 us/op 1.04
deserialize SignedBeaconBlock - struct 124.66 us/op 126.03 us/op 0.99
BeaconState vc 300000 - deserialize tree 625.35 ms/op 640.71 ms/op 0.98
BeaconState vc 300000 - serialize tree 174.19 ms/op 143.01 ms/op 1.22
BeaconState.historicalRoots vc 300000 - deserialize tree 815.00 ns/op 820.00 ns/op 0.99
BeaconState.historicalRoots vc 300000 - serialize tree 686.00 ns/op 696.00 ns/op 0.99
BeaconState.validators vc 300000 - deserialize tree 601.83 ms/op 633.49 ms/op 0.95
BeaconState.validators vc 300000 - serialize tree 102.39 ms/op 96.018 ms/op 1.07
BeaconState.balances vc 300000 - deserialize tree 42.126 ms/op 22.892 ms/op 1.84
BeaconState.balances vc 300000 - serialize tree 3.6998 ms/op 3.7395 ms/op 0.99
BeaconState.previousEpochParticipation vc 300000 - deserialize tree 469.71 us/op 454.19 us/op 1.03
BeaconState.previousEpochParticipation vc 300000 - serialize tree 275.51 us/op 280.32 us/op 0.98
BeaconState.currentEpochParticipation vc 300000 - deserialize tree 465.86 us/op 391.68 us/op 1.19
BeaconState.currentEpochParticipation vc 300000 - serialize tree 277.64 us/op 273.28 us/op 1.02
BeaconState.inactivityScores vc 300000 - deserialize tree 25.514 ms/op 25.757 ms/op 0.99
BeaconState.inactivityScores vc 300000 - serialize tree 3.9320 ms/op 3.8948 ms/op 1.01
hashTreeRoot Attestation - struct 21.055 us/op 21.234 us/op 0.99
hashTreeRoot Attestation - tree 9.2700 us/op 9.0710 us/op 1.02
hashTreeRoot SignedAggregateAndProof - struct 25.440 us/op 25.597 us/op 0.99
hashTreeRoot SignedAggregateAndProof - tree 12.781 us/op 12.800 us/op 1.00
hashTreeRoot SyncCommitteeMessage - struct 6.4490 us/op 6.6670 us/op 0.97
hashTreeRoot SyncCommitteeMessage - tree 3.1410 us/op 3.1200 us/op 1.01
hashTreeRoot SignedContributionAndProof - struct 15.626 us/op 15.231 us/op 1.03
hashTreeRoot SignedContributionAndProof - tree 10.683 us/op 8.7460 us/op 1.22
hashTreeRoot SignedBeaconBlock - struct 1.4138 ms/op 1.5448 ms/op 0.92
hashTreeRoot SignedBeaconBlock - tree 761.30 us/op 774.26 us/op 0.98
hashTreeRoot Validator - struct 8.3510 us/op 8.6410 us/op 0.97
hashTreeRoot Validator - tree 6.7410 us/op 7.2210 us/op 0.93
BeaconState vc 300000 - hashTreeRoot tree 2.1743 s/op 2.1489 s/op 1.01
BeaconState vc 300000 - batchHashTreeRoot tree 3.5572 s/op 3.7418 s/op 0.95
BeaconState.historicalRoots vc 300000 - hashTreeRoot tree 1.0990 us/op 1.0590 us/op 1.04
BeaconState.validators vc 300000 - hashTreeRoot tree 2.1557 s/op 2.2721 s/op 0.95
BeaconState.balances vc 300000 - hashTreeRoot tree 32.833 ms/op 35.865 ms/op 0.92
BeaconState.previousEpochParticipation vc 300000 - hashTreeRoot tree 4.2844 ms/op 3.9650 ms/op 1.08
BeaconState.currentEpochParticipation vc 300000 - hashTreeRoot tree 4.0779 ms/op 3.9582 ms/op 1.03
BeaconState.inactivityScores vc 300000 - hashTreeRoot tree 32.859 ms/op 36.002 ms/op 0.91
hash64 x18 9.5520 us/op 9.4170 us/op 1.01
hashTwoObjects x18 8.9070 us/op 8.5700 us/op 1.04
hash64 x1740 905.57 us/op 941.11 us/op 0.96
hashTwoObjects x1740 798.36 us/op 801.55 us/op 1.00
hash64 x2700000 1.3898 s/op 1.3917 s/op 1.00
hashTwoObjects x2700000 1.2495 s/op 1.2413 s/op 1.01
get_exitEpoch - ContainerType 253.00 ns/op 277.00 ns/op 0.91
get_exitEpoch - ContainerNodeStructType 258.00 ns/op 263.00 ns/op 0.98
set_exitEpoch - ContainerType 254.00 ns/op 262.00 ns/op 0.97
set_exitEpoch - ContainerNodeStructType 259.00 ns/op 280.00 ns/op 0.93
get_pubkey - ContainerType 1.0850 us/op 919.00 ns/op 1.18
get_pubkey - ContainerNodeStructType 246.00 ns/op 300.00 ns/op 0.82
hashTreeRoot - ContainerType 473.00 ns/op 468.00 ns/op 1.01
hashTreeRoot - ContainerNodeStructType 514.00 ns/op 491.00 ns/op 1.05
createProof - ContainerType 5.0670 us/op 4.2960 us/op 1.18
createProof - ContainerNodeStructType 25.050 us/op 22.385 us/op 1.12
serialize - ContainerType 2.1580 us/op 2.0100 us/op 1.07
serialize - ContainerNodeStructType 1.7050 us/op 1.6020 us/op 1.06
set_exitEpoch_and_hashTreeRoot - ContainerType 2.8030 us/op 2.8950 us/op 0.97
set_exitEpoch_and_hashTreeRoot - ContainerNodeStructType 7.6270 us/op 7.7510 us/op 0.98
Array - for of 5.9580 us/op 9.7260 us/op 0.61
Array - for(;;) 5.9130 us/op 10.013 us/op 0.59
basicListValue.readonlyValuesArray() 7.2032 ms/op 4.6683 ms/op 1.54
basicListValue.readonlyValuesArray() + loop all 7.2978 ms/op 5.3708 ms/op 1.36
compositeListValue.readonlyValuesArray() 31.122 ms/op 32.117 ms/op 0.97
compositeListValue.readonlyValuesArray() + loop all 30.744 ms/op 29.924 ms/op 1.03
Number64UintType - get balances list 4.7938 ms/op 4.8945 ms/op 0.98
Number64UintType - set balances list 10.188 ms/op 10.200 ms/op 1.00
Number64UintType - get and increase 10 then set 41.672 ms/op 43.590 ms/op 0.96
Number64UintType - increase 10 using applyDelta 16.381 ms/op 16.929 ms/op 0.97
Number64UintType - increase 10 using applyDeltaInBatch 16.372 ms/op 16.861 ms/op 0.97
tree_newTreeFromUint64Deltas 17.198 ms/op 17.580 ms/op 0.98
unsafeUint8ArrayToTree 34.377 ms/op 33.866 ms/op 1.02
bitLength(50) 246.00 ns/op 245.00 ns/op 1.00
bitLengthStr(50) 233.00 ns/op 248.00 ns/op 0.94
bitLength(8000) 233.00 ns/op 260.00 ns/op 0.90
bitLengthStr(8000) 277.00 ns/op 277.00 ns/op 1.00
bitLength(250000) 258.00 ns/op 255.00 ns/op 1.01
bitLengthStr(250000) 297.00 ns/op 313.00 ns/op 0.95
floor - Math.floor (53) 1.2369 ns/op 1.2383 ns/op 1.00
floor - << 0 (53) 1.2420 ns/op 1.2375 ns/op 1.00
floor - Math.floor (512) 1.2370 ns/op 1.2543 ns/op 0.99
floor - << 0 (512) 1.2365 ns/op 1.2384 ns/op 1.00
fnIf(0) 1.5512 ns/op 1.5484 ns/op 1.00
fnSwitch(0) 2.1642 ns/op 2.1658 ns/op 1.00
fnObj(0) 1.5494 ns/op 1.5493 ns/op 1.00
fnArr(0) 1.5489 ns/op 1.5529 ns/op 1.00
fnIf(4) 2.1723 ns/op 2.1653 ns/op 1.00
fnSwitch(4) 2.1636 ns/op 2.1991 ns/op 0.98
fnObj(4) 1.5536 ns/op 1.5483 ns/op 1.00
fnArr(4) 1.5532 ns/op 1.5706 ns/op 0.99
fnIf(9) 3.0912 ns/op 3.0906 ns/op 1.00
fnSwitch(9) 2.1653 ns/op 2.1987 ns/op 0.98
fnObj(9) 1.5518 ns/op 1.5592 ns/op 1.00
fnArr(9) 1.5500 ns/op 1.5456 ns/op 1.00
Container {a,b,vec} - as struct x100000 123.85 us/op 123.89 us/op 1.00
Container {a,b,vec} - as tree x100000 340.55 us/op 341.07 us/op 1.00
Container {a,vec,b} - as struct x100000 154.91 us/op 154.78 us/op 1.00
Container {a,vec,b} - as tree x100000 402.22 us/op 402.43 us/op 1.00
get 2 props x1000000 - rawObject 309.47 us/op 309.28 us/op 1.00
get 2 props x1000000 - proxy 72.710 ms/op 72.986 ms/op 1.00
get 2 props x1000000 - customObj 309.57 us/op 311.03 us/op 1.00
Simple object binary -> struct 596.00 ns/op 596.00 ns/op 1.00
Simple object binary -> tree_backed 1.1330 us/op 1.0660 us/op 1.06
Simple object struct -> tree_backed 1.7410 us/op 1.6350 us/op 1.06
Simple object tree_backed -> struct 1.7200 us/op 1.6250 us/op 1.06
Simple object struct -> binary 896.00 ns/op 829.00 ns/op 1.08
Simple object tree_backed -> binary 1.4920 us/op 1.4530 us/op 1.03
aggregationBits binary -> struct 524.00 ns/op 511.00 ns/op 1.03
aggregationBits binary -> tree_backed 2.2050 us/op 2.0860 us/op 1.06
aggregationBits struct -> tree_backed 2.6260 us/op 2.4640 us/op 1.07
aggregationBits tree_backed -> struct 1.1380 us/op 1.0230 us/op 1.11
aggregationBits struct -> binary 738.00 ns/op 792.00 ns/op 0.93
aggregationBits tree_backed -> binary 1.0070 us/op 977.00 ns/op 1.03
List(uint8) 100000 binary -> struct 1.4669 ms/op 1.4808 ms/op 0.99
List(uint8) 100000 binary -> tree_backed 96.885 us/op 96.612 us/op 1.00
List(uint8) 100000 struct -> tree_backed 1.1736 ms/op 1.1618 ms/op 1.01
List(uint8) 100000 tree_backed -> struct 1.0419 ms/op 1.0415 ms/op 1.00
List(uint8) 100000 struct -> binary 1.0067 ms/op 1.0312 ms/op 0.98
List(uint8) 100000 tree_backed -> binary 90.543 us/op 92.142 us/op 0.98
List(uint64Number) 100000 binary -> struct 1.2270 ms/op 1.2798 ms/op 0.96
List(uint64Number) 100000 binary -> tree_backed 3.4753 ms/op 3.8945 ms/op 0.89
List(uint64Number) 100000 struct -> tree_backed 4.8831 ms/op 5.8436 ms/op 0.84
List(uint64Number) 100000 tree_backed -> struct 2.2968 ms/op 2.3376 ms/op 0.98
List(uint64Number) 100000 struct -> binary 1.5222 ms/op 1.6151 ms/op 0.94
List(uint64Number) 100000 tree_backed -> binary 856.94 us/op 901.30 us/op 0.95
List(Uint64Bigint) 100000 binary -> struct 3.8153 ms/op 4.0316 ms/op 0.95
List(Uint64Bigint) 100000 binary -> tree_backed 3.3245 ms/op 3.7637 ms/op 0.88
List(Uint64Bigint) 100000 struct -> tree_backed 5.6097 ms/op 6.4457 ms/op 0.87
List(Uint64Bigint) 100000 tree_backed -> struct 4.9698 ms/op 5.0072 ms/op 0.99
List(Uint64Bigint) 100000 struct -> binary 2.0774 ms/op 2.0659 ms/op 1.01
List(Uint64Bigint) 100000 tree_backed -> binary 991.80 us/op 1.0920 ms/op 0.91
Vector(Root) 100000 binary -> struct 33.414 ms/op 35.527 ms/op 0.94
Vector(Root) 100000 binary -> tree_backed 36.562 ms/op 33.557 ms/op 1.09
Vector(Root) 100000 struct -> tree_backed 44.736 ms/op 44.978 ms/op 0.99
Vector(Root) 100000 tree_backed -> struct 50.777 ms/op 50.142 ms/op 1.01
Vector(Root) 100000 struct -> binary 2.7700 ms/op 2.7928 ms/op 0.99
Vector(Root) 100000 tree_backed -> binary 9.2043 ms/op 10.251 ms/op 0.90
List(Validator) 100000 binary -> struct 100.37 ms/op 100.76 ms/op 1.00
List(Validator) 100000 binary -> tree_backed 321.97 ms/op 324.06 ms/op 0.99
List(Validator) 100000 struct -> tree_backed 333.68 ms/op 339.28 ms/op 0.98
List(Validator) 100000 tree_backed -> struct 217.38 ms/op 224.19 ms/op 0.97
List(Validator) 100000 struct -> binary 27.157 ms/op 27.280 ms/op 1.00
List(Validator) 100000 tree_backed -> binary 113.98 ms/op 118.06 ms/op 0.97
List(Validator-NS) 100000 binary -> struct 102.23 ms/op 115.13 ms/op 0.89
List(Validator-NS) 100000 binary -> tree_backed 152.98 ms/op 161.92 ms/op 0.94
List(Validator-NS) 100000 struct -> tree_backed 191.23 ms/op 203.00 ms/op 0.94
List(Validator-NS) 100000 tree_backed -> struct 170.87 ms/op 176.98 ms/op 0.97
List(Validator-NS) 100000 struct -> binary 26.882 ms/op 27.957 ms/op 0.96
List(Validator-NS) 100000 tree_backed -> binary 33.358 ms/op 34.879 ms/op 0.96
get epochStatuses - MutableVector 111.45 us/op 116.49 us/op 0.96
get epochStatuses - ViewDU 202.50 us/op 199.36 us/op 1.02
set epochStatuses - ListTreeView 1.6117 ms/op 1.7212 ms/op 0.94
set epochStatuses - ListTreeView - set() 460.65 us/op 426.97 us/op 1.08
set epochStatuses - ListTreeView - commit() 609.63 us/op 1.6358 ms/op 0.37
bitstring 650.95 ns/op 652.32 ns/op 1.00
bit mask 13.802 ns/op 14.226 ns/op 0.97
struct - increase slot to 1000000 927.61 us/op 928.26 us/op 1.00
UintNumberType - increase slot to 1000000 21.989 ms/op 27.291 ms/op 0.81
UintBigintType - increase slot to 1000000 177.75 ms/op 196.10 ms/op 0.91
UintBigint8 x 100000 tree_deserialize 4.9469 ms/op 5.1091 ms/op 0.97
UintBigint8 x 100000 tree_serialize 1.0981 ms/op 1.1100 ms/op 0.99
UintBigint16 x 100000 tree_deserialize 5.0301 ms/op 5.2915 ms/op 0.95
UintBigint16 x 100000 tree_serialize 1.2702 ms/op 1.3759 ms/op 0.92
UintBigint32 x 100000 tree_deserialize 6.0171 ms/op 5.4499 ms/op 1.10
UintBigint32 x 100000 tree_serialize 1.2893 ms/op 1.3623 ms/op 0.95
UintBigint64 x 100000 tree_deserialize 6.1901 ms/op 5.6278 ms/op 1.10
UintBigint64 x 100000 tree_serialize 1.6550 ms/op 1.7370 ms/op 0.95
UintBigint8 x 100000 value_deserialize 432.99 us/op 433.04 us/op 1.00
UintBigint8 x 100000 value_serialize 698.03 us/op 808.02 us/op 0.86
UintBigint16 x 100000 value_deserialize 464.23 us/op 464.58 us/op 1.00
UintBigint16 x 100000 value_serialize 764.99 us/op 848.00 us/op 0.90
UintBigint32 x 100000 value_deserialize 432.75 us/op 433.00 us/op 1.00
UintBigint32 x 100000 value_serialize 747.15 us/op 845.72 us/op 0.88
UintBigint64 x 100000 value_deserialize 496.01 us/op 496.47 us/op 1.00
UintBigint64 x 100000 value_serialize 930.05 us/op 969.44 us/op 0.96
UintBigint8 x 100000 deserialize 3.3405 ms/op 3.7772 ms/op 0.88
UintBigint8 x 100000 serialize 1.6332 ms/op 1.7460 ms/op 0.94
UintBigint16 x 100000 deserialize 3.1618 ms/op 3.7574 ms/op 0.84
UintBigint16 x 100000 serialize 1.5641 ms/op 1.4925 ms/op 1.05
UintBigint32 x 100000 deserialize 3.0936 ms/op 4.0283 ms/op 0.77
UintBigint32 x 100000 serialize 2.8676 ms/op 2.9962 ms/op 0.96
UintBigint64 x 100000 deserialize 3.8758 ms/op 4.1962 ms/op 0.92
UintBigint64 x 100000 serialize 1.5146 ms/op 1.5403 ms/op 0.98
UintBigint128 x 100000 deserialize 5.4288 ms/op 5.2387 ms/op 1.04
UintBigint128 x 100000 serialize 14.668 ms/op 14.658 ms/op 1.00
UintBigint256 x 100000 deserialize 8.5575 ms/op 8.5338 ms/op 1.00
UintBigint256 x 100000 serialize 43.374 ms/op 43.360 ms/op 1.00
Slice from Uint8Array x25000 1.2318 ms/op 1.2160 ms/op 1.01
Slice from ArrayBuffer x25000 16.524 ms/op 15.788 ms/op 1.05
Slice from ArrayBuffer x25000 + new Uint8Array 18.202 ms/op 17.359 ms/op 1.05
Copy Uint8Array 100000 iterate 1.6703 ms/op 1.7065 ms/op 0.98
Copy Uint8Array 100000 slice 125.58 us/op 131.01 us/op 0.96
Copy Uint8Array 100000 Uint8Array.prototype.slice.call 125.05 us/op 119.91 us/op 1.04
Copy Buffer 100000 Uint8Array.prototype.slice.call 126.09 us/op 116.62 us/op 1.08
Copy Uint8Array 100000 slice + set 245.04 us/op 221.07 us/op 1.11
Copy Uint8Array 100000 subarray + set 128.13 us/op 114.50 us/op 1.12
Copy Uint8Array 100000 slice arrayBuffer 119.28 us/op 117.04 us/op 1.02
Uint64 deserialize 100000 - iterate Uint8Array 1.9912 ms/op 1.8882 ms/op 1.05
Uint64 deserialize 100000 - by Uint32A 2.0155 ms/op 2.0486 ms/op 0.98
Uint64 deserialize 100000 - by DataView.getUint32 x2 1.9725 ms/op 2.1332 ms/op 0.92
Uint64 deserialize 100000 - by DataView.getBigUint64 5.3426 ms/op 5.3102 ms/op 1.01
Uint64 deserialize 100000 - by byte 40.448 ms/op 40.211 ms/op 1.01

by benchmarkbot/action

@twoeths twoeths marked this pull request as ready for review September 10, 2024 08:14
@twoeths twoeths requested a review from a team as a code owner September 10, 2024 08:14
* - special handle for index === snapshot.count - 1, we restore from snapshot
*/
sliceTo(index: number): this {
if (index < this.snapshot.count - 1) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

g11tech
g11tech previously approved these changes Sep 10, 2024
Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks super amazing 👍

i think we will now be saving the DepositTree in db by the finalized gindex nodes and then values post that?

@twoeths
Copy link
Contributor Author

twoeths commented Sep 10, 2024

looks super amazing 👍

i think we will now be saving the DepositTree in db by the finalized gindex nodes and then values post that?

now instead of saving all roots, we'll just save serialized snapshot periodically. Something like:

const treeSnapshot = depositRootTree.toSnapshot(depositCount);
const depositSnapshot = {
    finalized: treeSnapshot.finalized,
    depositRoot: treeSnapshot.root,
    depositCount: treeSnapshot.count,
    executionBlockHash,
    executionBlockHeight,
};
db.put(executionBlockHeight, depositSnapshot);

upon restart we download snapshot from checkpointSyncUrl, compare to db's snapshot and use either of them, then reconstruct the partial tree from snapshot

const depositRootTree = DepositDataRootPartialList.toPartialViewDU({
  finalized: snapshot.finalized,
  root: snapshot.depositRoot,
  count: snapshot.depositCount,
});

then the partial tree grows from there similar to the current full tree

@@ -1 +1,8 @@
export type Require<T, K extends keyof T> = T & Required<Pick<T, K>>;

/** Snapshot for PartialListCompositeType */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add more comments here?

Eg:
A snapshot contains the minimum amount of information needed to reconstruct a merkleized list, for the purposes of appending more items. Note: This does not contain list elements, rather only contains intermediate merkle nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Snapshot apis
3 participants