Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: switch bls to napi rebuild #6616

Merged
merged 71 commits into from
Apr 24, 2024
Merged
Show file tree
Hide file tree
Changes from 65 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
a74b8f5
chore: update package.json deps and run yarn
matthewkeil Mar 31, 2024
f692d1b
feat: remove unused blst config from vitest.browser.*.config.ts
matthewkeil Mar 31, 2024
e2dfc26
test: move G2@INF test to base blst repo. No longer exported.
matthewkeil Mar 31, 2024
923daad
fix: add Uint8Array.from for Buffer return value in unit test so type…
matthewkeil Mar 31, 2024
f53a496
refactor: update imports of CoordType to use bls instead of blst
matthewkeil Mar 31, 2024
03be472
feat: add blsAddVerificationRandomness cli option
matthewkeil Mar 31, 2024
bbd3436
feat: add randomBytesNonZero to utils
matthewkeil Mar 31, 2024
0b363ae
feat: implement verification randomness for same message
matthewkeil Mar 31, 2024
23f263c
feat: implement napi blst and switch to libuv worker pool
matthewkeil Mar 31, 2024
d152dfc
chore: lint
matthewkeil Mar 31, 2024
8dbc8cc
fix: update comments for availableParallelism
matthewkeil Mar 31, 2024
c2158f2
feat: set UV_THREADPOOL_SIZE for perf tests
matthewkeil Mar 31, 2024
01cfbac
feat: set UV_THREADPOOL_SIZE=$(nproc) in lodestar entrance script
matthewkeil Mar 31, 2024
de1e3c5
feat: remove case for serialized PublicKey in deserializeSet
matthewkeil Apr 1, 2024
efede8c
feat: add back workBusy count logic to canAcceptWork
matthewkeil Apr 3, 2024
9757f43
feat: automatically set threadpool size in beaconHandler
matthewkeil Apr 4, 2024
420b081
refactor: rename flag to disableSameMessageVerificationRandomness
matthewkeil Apr 9, 2024
afaef65
fix: remove UV_THREADPOOL_SIZE from ./lodestar
matthewkeil Apr 9, 2024
e0f3ff3
docs: add TODO about updating metrics
matthewkeil Apr 9, 2024
2578604
fix: make benchmark multi threading cross-compatible
matthewkeil Apr 9, 2024
65cbf11
chore: add [email protected]
matthewkeil Apr 12, 2024
3d99932
feat: default to added randomness
matthewkeil Apr 12, 2024
a3ec634
feat: add warning for no same message randomness
matthewkeil Apr 12, 2024
2afb0cc
fix: passing to updated bls functions
matthewkeil Apr 12, 2024
1ee3514
feat: add blst peerDep
matthewkeil Apr 12, 2024
f1d5e6c
fix: default to same message randomness
matthewkeil Apr 12, 2024
0137f39
Merge branch 'unstable' into mkeil/napi-rebuild
matthewkeil Apr 16, 2024
1eaa8b0
chore: update yarn.lock
matthewkeil Apr 16, 2024
f16a6f1
chore: remove unused function to clear lint
matthewkeil Apr 16, 2024
c57580f
feat: break out setThreadPoolSize
matthewkeil Apr 17, 2024
9e74c7a
docs: update packages/cli/src/cmds/beacon/setThreadPoolSize.ts
matthewkeil Apr 17, 2024
ee21968
chore: update location for blst dep
matthewkeil Apr 17, 2024
ecd5402
chore: run lint again
matthewkeil Apr 17, 2024
3684b1b
fix: remove peerDep from light-client
matthewkeil Apr 17, 2024
98d04ab
test: fix browser based tests
matthewkeil Apr 17, 2024
dd3b72d
fix: bad relative imports
matthewkeil Apr 17, 2024
b992f5c
docs: add note to readme about switchable bls
matthewkeil Apr 17, 2024
b345aa6
docs: typo
matthewkeil Apr 17, 2024
1acce0c
chore: lint readme
matthewkeil Apr 17, 2024
fa7893e
fix: light-client unit tests
matthewkeil Apr 17, 2024
23c11f0
fix: light-client e2e test
matthewkeil Apr 17, 2024
1f0f2b3
feat: remove disableSameMessageVerificationRandomness
matthewkeil Apr 17, 2024
bf65cc9
refactor: remove unused type
matthewkeil Apr 17, 2024
cc59be9
docs: update wordlist
matthewkeil Apr 17, 2024
f71acd7
fix: e2e tests
matthewkeil Apr 17, 2024
b56389f
Lint package.json
nflaig Apr 19, 2024
04f49c7
Merge branch 'unstable' into mkeil/napi-rebuild
matthewkeil Apr 21, 2024
2cdf203
Revert "fix: e2e tests"
matthewkeil Apr 21, 2024
a36c983
Revert "fix: light-client e2e test"
matthewkeil Apr 21, 2024
0891abb
Revert "fix: light-client unit tests"
matthewkeil Apr 21, 2024
85522d2
Revert "test: fix browser based tests"
matthewkeil Apr 21, 2024
9bd2880
docs: update readme to remove switchable bls argument
matthewkeil Apr 21, 2024
c4141f6
fix: replace optimizeDeps in vitest.browser.config
matthewkeil Apr 21, 2024
8ccd5d1
chore: updated to bls 8.1.0
matthewkeil Apr 22, 2024
df49036
feat: move setting threadpool size and warnings to applyPreset.ts
matthewkeil Apr 23, 2024
1fa4955
fix: remove setThreadpoolSize from beacon handler
matthewkeil Apr 23, 2024
13e2677
feat: move libuv size logging into handler to use logger
matthewkeil Apr 23, 2024
01ec025
Merge branch 'unstable' into mkeil/napi-rebuild
matthewkeil Apr 23, 2024
221b0de
refactor: rename applyPreset to preInitialization
matthewkeil Apr 23, 2024
065180a
refactor: logging about pool size
matthewkeil Apr 23, 2024
e20d5ec
Update packages/cli/src/options/globalOptions.ts
matthewkeil Apr 23, 2024
9d34194
refactor: rename prover applyPreset
matthewkeil Apr 23, 2024
2321c16
docs: change comment in cli/index
matthewkeil Apr 23, 2024
0d284c0
fix: update logging for
matthewkeil Apr 23, 2024
480034e
fix: throw error for invalid uvThreadpoolSize
matthewkeil Apr 23, 2024
524261a
Update packages/cli/src/preInitialization.ts
matthewkeil Apr 23, 2024
37cc6b2
Fix lint issues
nflaig Apr 23, 2024
ee0c04f
Remove unused import
nflaig Apr 23, 2024
63a5441
refactor: move bls pool logging back to multithread.ts
matthewkeil Apr 24, 2024
f00955d
Merge branch 'unstable' into mkeil/napi-rebuild
matthewkeil Apr 24, 2024
b720eab
fix: log of threadcount
matthewkeil Apr 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ orchestrator
osx
overriden
params
peerDependency
pid
plaintext
pre
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@
"test-coverage:e2e": "c8 --config .c8rc.json --report-dir coverage/e2e/ --all npm run test:e2e",
"test-coverage:e2e-sim": "c8 --config .c8rc.json --report-dir coverage/e2e-sim/ --all npm run test:e2e:sim",
"test-coverage:spec": "c8 --config .c8rc.json --report-dir coverage/spec/ --all npm run test:spec",
"benchmark": "yarn benchmark:files 'packages/*/test/perf/**/*.test.ts'",
"benchmark:files": "NODE_OPTIONS='--max-old-space-size=4096 --loader=ts-node/esm' benchmark --config .benchrc.yaml --defaultBranch unstable",
"benchmark": "UV_THREADPOOL_SIZE=$(node -e 'console.log(require(`os`).availableParallelism())') yarn benchmark:files 'packages/*/test/perf/**/*.test.ts'",
"benchmark:files": "UV_THREADPOOL_SIZE=$(node -e 'console.log(require(`os`).availableParallelism())') NODE_OPTIONS='--max-old-space-size=4096 --loader=ts-node/esm' benchmark --config .benchrc.yaml --defaultBranch unstable",
"release:create-rc": "node scripts/release/create_rc.mjs",
"release:tag-rc": "node scripts/release/tag_rc.mjs",
"release:tag-stable": "node scripts/release/tag_stable.mjs",
Expand Down
4 changes: 2 additions & 2 deletions packages/beacon-node/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,8 @@
},
"dependencies": {
"@chainsafe/as-sha256": "^0.4.1",
"@chainsafe/bls": "7.1.3",
"@chainsafe/blst": "^0.2.10",
"@chainsafe/bls": "^8.1.0",
"@chainsafe/blst": "^1.0.0",
"@chainsafe/discv5": "^9.0.0",
"@chainsafe/enr": "^3.0.0",
"@chainsafe/libp2p-gossipsub": "^11.2.1",
Expand Down
5 changes: 3 additions & 2 deletions packages/beacon-node/src/chain/bls/index.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
export type {IBlsVerifier} from "./interface.js";
export type {BlsMultiThreadWorkerPoolModules, JobQueueItemType} from "./multithread/index.js";
export {BlsMultiThreadWorkerPool} from "./multithread/index.js";
export type {JobQueueItemType} from "./jobItem.js";
export type {BlsMultiThreadWorkerPoolModules} from "./multiThread.js";
export {BlsMultiThreadWorkerPool} from "./multiThread.js";
export {BlsSingleThreadVerifier} from "./singleThread.js";
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
import bls from "@chainsafe/bls";
import {CoordType, PointFormat, PublicKey} from "@chainsafe/bls/types";
import {ISignatureSet, SignatureSetType} from "@lodestar/state-transition";
import {VerifySignatureOpts} from "../interface.js";
import {getAggregatedPubkey} from "../utils.js";
import {LinkedList} from "../../../util/array.js";
import {Metrics} from "../../../metrics/metrics.js";
import {LinkedList} from "../../util/array.js";
import {Metrics} from "../../metrics/metrics.js";
import {VerifySignatureOpts} from "./interface.js";
import {getAggregatedPubkey} from "./utils.js";
import {BlsWorkReq} from "./types.js";
import {randomBytesNonZero} from "./utils.js";

export type JobQueueItem = JobQueueItemDefault | JobQueueItemSameMessage;

Expand Down Expand Up @@ -56,7 +57,7 @@ export function jobItemWorkReq(job: JobQueueItem, format: PointFormat, metrics:
opts: job.opts,
sets: job.sets.map((set) => ({
// this can throw, handled in the consumer code
publicKey: getAggregatedPubkey(set, metrics).toBytes(format),
publicKey: getAggregatedPubkey(set, metrics),
signature: set.signature,
message: set.signingRoot,
})),
Expand All @@ -73,13 +74,19 @@ export function jobItemWorkReq(job: JobQueueItem, format: PointFormat, metrics:
const signatures = job.sets.map((set) => bls.Signature.fromBytes(set.signature, CoordType.affine, true));
timer?.();

// adding verification randomness is napi specific. must not attempt with herumi until
// @chainsafe/bls is updated to support it with herumi
const randomness: Uint8Array[] = [];
for (let i = 0; i < job.sets.length; i++) {
randomness.push(randomBytesNonZero(8));
}
return {
opts: job.opts,
sets: [
{
publicKey: bls.PublicKey.aggregate(job.sets.map((set) => set.publicKey)).toBytes(format),
signature: bls.Signature.aggregate(signatures).toBytes(format),
message: job.message,
publicKey: bls.PublicKey.aggregate(job.sets.map((set, i) => set.publicKey.multiplyBy(randomness[i]))),
signature: bls.Signature.aggregate(signatures.map((sig, i) => sig.multiplyBy(randomness[i]))),
},
],
};
Expand Down
46 changes: 0 additions & 46 deletions packages/beacon-node/src/chain/bls/maybeBatch.ts

This file was deleted.

Loading
Loading