Skip to content

Commit

Permalink
Initial squashed v2 commit
Browse files Browse the repository at this point in the history
Replace lodestarTypes

Update tests

Add Union and None types

Update spec tests

Remove old src and lodestar dependencies

Improve JSON parsing for tests

Load YAML with lodestar utils schema

Commit ArrayComposite before getAll

Re-write toJson fromJson test

Fix uint json conversions

Print tree in valid test if requested

Fix hashTreeRoot for ByteList

Add RENDER_ROOTS option

Fix typos

Use numerical sort in array commit

Fix value_serializedSizeArrayComposite

Pass ssz_generic tests

Allow to select tests to run in ssz_static

Fix container bytes offset

Rename fixedLen to fixedSize

Sort deserailization methods

Print json stringified in test

Review logic

Clean up tests

Define JSON casing at constructor time only

Add casing maps for merge types

Update casing in unit tests

Re-organize utils

Fix merge casing

FIx offset calculation

Pass all spec test pre-merge

Bump merge test

Extend timeout for mainnet tests

Pass all unit tests

Return defaultValue in simpleserialize random

Remove @chainsafe/lodestar-spec-test-util dependency

Copy yaml schema from lodestar-utils

Fix benchmark type issues

Skip createProof benchmark

Skip old benchmark without runner

Remove postinstall script

Re-add UintBigint optimization

Fix set_exitEpoch_and_hashTreeRoot benchmark

Add List of Number benchmark

Use DataViews for faster deserialization

Use DataViews for tree serialization too

Update workflows build after install

Update packedNode tests

Review Tree API

Validate length in ByteArrays

FIx benchmarks in persistent-merkle-tree

Simplify LeafNode constructor

Refactor subtreeFillToContents

Run struct <-> tree_backed benchmarks

Update test types

Fix Uint64 DataView benchmarks

Add benchmarks for full state serialization

Use consistent initialization in DataView

Optimize toView for ByteArray

Add clone method for safer ContainerTreeViewDU

Add documentation to all public methods

Update SSZ README

Simplify testTypes

Update persistent-merkle-tree README

Add unit test push x5

Fix heigh typo

Add note Supports index up to Number.MAX_SAFE_INTEGER.

Address PR comments

Add more benchmarks (#227)

Remove merkleizeSingleBuff

Add more documentation

Add typeName to all containers

Add eth2 JSON casing

Rename merge to bellatrix

Fix missing renames of merge to bellatrix

Update ssz readme

Export byteArrayEquals

Add more BitArray functionality

Various fixes

Test type UX of allForks types

Abstract logic into BitArray and ByteArray

Add .equals functionality

Don't run allForks code

Rename receiptRoot

Update ssz docstrings

Multiple fixes

Remove non-existent variable

Add proof api

Fix int conversion in node.getUintBigint

Add Proofs functionality

Run Proof tests on ssz_static minimal

Add length node for proofs of lists

Require rootNode only when necessary in Array proofs

Fix rebase issues

Move Node navigation to functions

Add treeZeroAfterIndex fn

Add sliceTo method in CompositeList ViewDU

Use Uint64NumInf only where necessary

Add Eth1Block type

Use defaultView and defaultViewDU methods

Support BranchNodeStruct in proofs

Run proof tests

Post process proof nodes

Fix proof generation for all types

Fix ContainerNodeStructType proof unit test

Stop tracking complimentary benchmarks

Fix typo in Union maxSize

Add test for maxSize and minSize for all data structures

Export helper functions

Fix length param in Vector

Prevent setting values in ViewDU beyond length

Add more test cases

Improve ListBasicTreeViewDU.push logic

Fix Composite ViewDU commit nodes logic

Fix cache logic in ViewDU

Use type guards and simply set composite

Fix allForks test

Fix recursive call in BitArray

Ensure data consistency in ArrayCompositeViewDU

Better error in BitArray.set

FIx typo in populateAllNodes

Add more BitArray unit tests

TreeViewDU.commit should not return node

Better error messages in fromHexString

Auto-commit on ViewDU

Fix perf types

Fix proof generation for Validator type

Benchmark more eth2 ssz objects

Track raw cost of hashing vc list

Add more coverage WIP

More coverage WIP

Prevent pushing over length

More coverage

Ensure consistent mutability behaviour

Ensure correct mutability in ContainerNodeStruct

Move isViewMutable to CompositeType

Prevent keeping references for immutable views

Fix ContainerNodeStruct valid tests

Fix types in Uint constructor

Increase bitArray coverage

Guard against new BitList checks

Fix readVariableOffsetsArrayComposite
  • Loading branch information
dapplion committed Feb 12, 2022
1 parent e7c4905 commit 59598e4
Show file tree
Hide file tree
Showing 246 changed files with 16,187 additions and 9,599 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
- uses: actions/setup-node@v2-beta
with:
node-version: "14.16.0"
- run: yarn
- run: yarn && yarn build

- name: Run benchmarks
run: yarn benchmark
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ jobs:
- uses: actions/checkout@v1
- name: Bootstrap
run: yarn
- name: Check types
run: yarn check-types
- name: Build
run: yarn build
- name: Check types
run: yarn check-types
- name: Lint
run: yarn lint
- name: Tests
Expand Down Expand Up @@ -47,9 +47,9 @@ jobs:
- name: Spec tests altair-mainnet
run: LODESTAR_PRESET=mainnet LODESTAR_FORK=altair ../../node_modules/.bin/mocha test/spec/ssz_static.test.ts
working-directory: packages/ssz
- name: Spec tests merge-minimal
run: LODESTAR_PRESET=minimal LODESTAR_FORK=merge ../../node_modules/.bin/mocha test/spec/ssz_static.test.ts
- name: Spec tests bellatrix-minimal
run: LODESTAR_PRESET=minimal LODESTAR_FORK=bellatrix ../../node_modules/.bin/mocha test/spec/ssz_static.test.ts
working-directory: packages/ssz
- name: Spec tests merge-mainnet
run: LODESTAR_PRESET=mainnet LODESTAR_FORK=merge ../../node_modules/.bin/mocha test/spec/ssz_static.test.ts
- name: Spec tests bellatrix-mainnet
run: LODESTAR_PRESET=mainnet LODESTAR_FORK=bellatrix ../../node_modules/.bin/mocha test/spec/ssz_static.test.ts
working-directory: packages/ssz
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ yarn-error.log
.idea/
.vscode

coverage
lib
packages/*/package-lock.json
packages/ssz/spec-tests

benchmark_data/
.failedTest.txt
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@
"packages/*"
],
"scripts": {
"postinstall": "yarn build",
"clean": "rm -rf ./packages/*/lib ./packages/*/*.tsbuildinfo",
"build": "yarn clean && lerna run build",
"lint": "lerna run lint --no-bail",
"check-types": "lerna run check-types --no-bail",
"coverage": "lerna run coverage --no-bail",
"test": "lerna run test --no-bail",
"benchmark": "NODE_OPTIONS=--max_old_space_size=4096 benchmark --config .benchrc.yaml 'packages/*/test/perf/**/*.test.ts'",
"benchmark:files": "NODE_OPTIONS=--max_old_space_size=4096 benchmark --config .benchrc.yaml",
"benchmark": "yarn benchmark:files 'packages/*/test/perf/**/*.test.ts'",
"benchmark:local": "yarn benchmark --local",
"publish:release": "lerna publish from-package --yes --no-verify-access",
"release": "lerna version --no-push --sign-git-commit",
Expand Down
46 changes: 43 additions & 3 deletions packages/persistent-merkle-tree/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ A binary merkle tree implemented as a [persistent data structure](https://en.wik

import {LeafNode, BranchNode} from "@chainsafe/persistent-merkle-tree";

const leaf = new LeafNode(Uint8Array.from([...]));
const otherLeaf = new LeafNode(Uint8Array.from([...]));
const leaf = LeafNode.fromRoot(Buffer.alloc(32, 0xaa));
const otherLeaf = LeafNode.fromRoot(Buffer.alloc(32, 0xbb));

const branch = new BranchNode(leaf, otherLeaf);

Expand Down Expand Up @@ -61,7 +61,7 @@ const gindex = BigInt(...);

const n: Node = tree.getNode(gindex); // the Node at gindex
const rrr: Uint8Array = tree.getRoot(gindex); // the Uint8Array root at gindex
const subtree: Tree = tree.getSubtree(gindex); // the Tree wrapping the Node at gindex
const subtree: Tree = tree.getSubtree(gindex); // the Tree wrapping the Node at gindex. Updates to `subtree` will be propagated to `tree`

// A merkle proof for a gindex can be generated

Expand Down Expand Up @@ -115,6 +115,46 @@ Any update to a tree (either to a leaf or intermediate node) is performed as a r

A `Tree` object wraps `Node` and provides an API for tree navigation and transparent rebinding on updates.

#### Navigation by gindex or (depth, index)

Many tree methods allow navigation with a gindex. A gindex (or generalized index) describes a path through the tree, starting from the root and nagivating downwards.

```
1
/ \
2 3
/ \ / \
4 5 6 7
```

It can also be interpreted as a bitstring, starting with "1", then appending "0" for each navigation left, or "1" for each navigation right.

```
1
/ \
10 11
/ \ / \
100 101 110 111
```

Alternatively, several tree methods, with names ending with `AtDepth`, allow navigation by (depth, index). Depth and index navigation works by first navigating down levels into the tree from the top, starting at 0 (depth), and indexing nodes from the left, starting at 0 (index).

```
0 <- depth 0
/ \
0 1 <- depth 1
/ \ / \
0 1 2 3 <- depth 2
```

#### Memory efficiency

As an implementation detail, nodes hold their values as a `HashObject` (a javascript object with `h0`, ... `h7` uint32 `number` values) internally, rather than as a 32 byte `Uint8Array`. Memory benchmarking shows that this results in a ~3x reduction in memory over `Uint8Array`. This optimization allows this library to practically scale to trees with millions of nodes.

#### Navigation efficiency

In performance-critical applications performing many reads and writes to trees, being smart with tree navigation is crucial. This library correctly provides tree navigation methods that handle several important optimized cases: multi-node get and set, and get-then-set operations.

## See also:

https://github.com/protolambda/remerkleable
Expand Down
8 changes: 3 additions & 5 deletions packages/persistent-merkle-tree/src/gindex.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,19 +23,17 @@ export function toGindexBitstring(depth: number, index: number): GindexBitstring
}

export function convertGindexToBitstring(gindex: Gindex | GindexBitstring): GindexBitstring {
let bitstring: string;
if (typeof gindex === "string") {
if (!gindex.length) {
if (gindex.length === 0) {
throw new Error(ERR_INVALID_GINDEX);
}
bitstring = gindex;
return gindex;
} else {
if (gindex < 1) {
throw new Error(ERR_INVALID_GINDEX);
}
bitstring = gindex.toString(2);
return gindex.toString(2);
}
return bitstring;
}

// Get the depth (root starting at 0) necessary to cover a subtree of `count` elements.
Expand Down
5 changes: 3 additions & 2 deletions packages/persistent-merkle-tree/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
export * from "./gindex";
export * from "./hash";
export * from "./node";
export * from "./zeroNode";
export * from "./packedNode";
export * from "./proof";
export * from "./subtree";
export * from "./tree";
export * from "./proof";
export * from "./zeroNode";
Loading

0 comments on commit 59598e4

Please sign in to comment.