Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perform kurtosis runs for lodestar coupled with Kiln ready EL/Cl clients #3751

Closed
5 of 6 tasks
Tracked by #3731
g11tech opened this issue Feb 11, 2022 · 4 comments
Closed
5 of 6 tasks
Tracked by #3731

Perform kurtosis runs for lodestar coupled with Kiln ready EL/Cl clients #3751

g11tech opened this issue Feb 11, 2022 · 4 comments
Assignees
@g11tech
Labels
scope-security Issues that fix security issues: DOS, key leak, CVEs.

Comments

@g11tech
Copy link
Contributor

g11tech commented Feb 11, 2022 •
edited
Loading

Merge Scenarios

Build kurtosis enclave image: https://github.com/kurtosis-tech/eth2-merge-kurtosis-module and do a successful run
(thanks @parithosh for providing this bit ❤️ ❤️ ❤️ with some image building fun mixed in 🤣 ) 🚀

  • Scenario 1 - lodestar <> geth, lodestar geth <> Observations:
    kurtosis module exec --enclave-id eth2-multi-cl parithoshj/kurtosis:kiln-x86 --execute-params '{"participants":[{"elType":"geth","elImage":"parithoshj/geth:merge-0569213","clType":"lodestar","clImage":"chainsafe/lodestar:next"},{"elType":"geth","elImage":"parithoshj/geth:merge-0569213","clType":"lodestar","clImage":"chainsafe/lodestar:next"}],"network":{"networkId":"3151908","depositContractAddress":"0x4242424242424242424242424242424242424242","secondsPerSlot":3,"slotsPerEpoch":4,"altairForkEpoch":1,"mergeForkEpoch":2,"totalTerminalDifficulty":1000000,"numValidatorKeysPerNode":64,"preregisteredValidatorKeysMnemonic":"giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete"},"waitForMining":false,"waitForFinalization":true,"logLevel":"info"}'
    • Observation 1: Sometimes EL dont sync up, for e.g. both geth nodes kept on giving ACCEPTED ad nauseam leading to lodestar beacons blocking each other
    • Observation 2: Other times, EL synced up with the other after giving one ACCEPTED
  • Scenario 2- lodestar<> nethermind, lodestar <> geth
    kurtosis module exec --enclave-id eth2-multi-cl parithoshj/kurtosis:kiln-x86 --execute-params '{"participants":[{"elType":"geth","elImage":"parithoshj/geth:merge-0569213","clType":"lodestar","clImage":"chainsafe/lodestar:next"},{"elType":"nethermind","elImage":"nethermindeth/nethermind:kiln_0.1","clType":"lodestar","clImage":"chainsafe/lodestar:next"}],"network":{"networkId":"3151908","depositContractAddress":"0x4242424242424242424242424242424242424242","secondsPerSlot":3,"slotsPerEpoch":4,"altairForkEpoch":1,"mergeForkEpoch":2,"totalTerminalDifficulty":1000000,"numValidatorKeysPerNode":64,"preregisteredValidatorKeysMnemonic":"giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete"},"waitForMining":false,"waitForFinalization":true,"logLevel":"info"}'
  • Scenario 3 - multi client runs
    kurtosis module exec --enclave-id eth2-multi-cl parithoshj/kurtosis:kiln --execute-params '{"participants":[{"elType":"geth","elImage":"parithoshj/geth:merge-0569213","clType":"lodestar","clImage":"chainsafe/lodestar:next"},{"elType":"geth","elImage":"parithoshj/geth:merge-0569213","clType":"teku","clImage":"consensys/teku:develop","beaconExtraParams":["--Xee-version kiln"]},{"elType":"geth","elImage":"parithoshj/geth:merge-0569213","clType":"teku","clImage":"consensys/teku:develop","beaconExtraParams":["--Xee-version kiln"]},{"elType":"nethermind","elImage":"nethermindeth/nethermind:kiln_0.1","clType":"teku","clImage":"consensys/teku:develop","beaconExtraParams":["--Xee-version kiln"]}],"network":{"networkId":"3151908","depositContractAddress":"0x4242424242424242424242424242424242424242","secondsPerSlot":12,"slotsPerEpoch":32,"altairForkEpoch":1,"mergeForkEpoch":3,"totalTerminalDifficulty":100000000,"numValidatorKeysPerNode":64,"preregisteredValidatorKeysMnemonic":"giant issue aisle success illegal bike spike question tent bar rely arctic volcano long crawl hungry vocal artwork sniff fantasy very lucky have athlete"},"waitForMining":false,"waitForFinalization":true,"logLevel":"info"}'

Lodestar behavior checklist

  • don't accept on ACCEPTED when merge is not yet justified 
    Feb-11 17:14:50.411 [CHAIN]           error: Block error slot=64, errCode=BLOCK_ERROR_EXECUTION_ERROR code=BLOCK_ERROR_EXECUTION_ERROR, errorMessage=not safe to import not yet validated payload within 128 of currentSlot, status=ACCEPTED
Error: BLOCK_ERROR_EXECUTION_ERROR
    at verifyBlockStateTransition (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:240:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at verifyBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:46:40)
  • If ELs sync up, the beacons should accept each other payloads and move forward completing the merge 
    Feb-11 17:14:50.411 [CHAIN]           error: Block error slot=64, errCode=BLOCK_ERROR_EXECUTION_ERROR code=BLOCK_ERROR_EXECUTION_ERROR, errorMessage=not safe to import not yet validated payload within 128 of currentSlot, status=ACCEPTED
Error: BLOCK_ERROR_EXECUTION_ERROR
    at verifyBlockStateTransition (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:240:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at verifyBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:46:40)
    at processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/index.ts:64:32)
    at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)
Feb-11 17:14:50.412 [NETWORK]         error: Error receiving block slot=64, peer=16Uiu2HAmBvTmyuJqK18phuABiSdA8fAnY8xwuSu5wSRf1v9Mym6Z code=BLOCK_ERROR_EXECUTION_ERROR, errorMessage=not safe to import not yet validated payload within 128 of currentSlot, status=ACCEPTED
Error: BLOCK_ERROR_EXECUTION_ERROR
    at verifyBlockStateTransition (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:240:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at verifyBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:46:40)
    at processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/index.ts:64:32)
    at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)
Feb-11 17:14:51.500 []                 info: Synced - slot: 64 (skipped 1) - head: 63 0x25a1…cfb5 - finalized: 0x0000…0000:0 - peers: 1
Feb-11 17:14:54.501 []                 info: Synced - slot: 65 - head: 65 0xf359…5e80 - execution: valid(0x170e…d7bd) - finalized: 0x0000…0000:0 - peers: 1
Feb-11 17:14:57.501 []                 info: Synced - slot: 66 - head: 66 0xd550…b952 - execution: valid(0x3661…440a) - finalized: 0x0000…0000:0 - peers: 1
Feb-11 17:15:00.500 []                 info: Synced - slot: 67 - head: 67 0xa2de…c7af - execution: valid(0x551a…552b) - finalized: 0x0000…0000:0 - peers: 1
Feb-11 17:15:03.506 []                 info: Synced - slot: 68 - head: 68 0x7cf5…a0db - execution: valid(0x45c7…6cd1) - finalized: 0x0000…0000:0 - peers: 1
  • If ELs don't sync up don't ban each other (Scenario 1 observation 1) as Optimistic sync now recommends not to block peers on valid/invalid/syncing responses.
@g11tech g11tech mentioned this issue Feb 11, 2022
Closed
14 tasks
@g11tech g11tech self-assigned this Feb 11, 2022
@g11tech
Copy link
Contributor Author

g11tech commented Feb 14, 2022

UPDATE SCENARIO 2

Both geth & nethermind gave same terminal POW:
geth:

Feb-14 09:05:36.200 [ETH1]             info: Starting search for terminal POW block TERMINAL_TOTAL_DIFFICULTY=1000000
Feb-14 09:05:37.774 [ETH1]             info: Terminal POW block found! hash=0x594e9d4193a4cc0f8b7ce0037b2e989123662c43a733d1d352d380d8762078a5, number=8, totalDifficulty=1049921

nethermind:

Feb-14 09:05:48.366 [ETH1]             info: Starting search for terminal POW block TERMINAL_TOTAL_DIFFICULTY=1000000
Feb-14 09:05:49.018 [ETH1]             info: Terminal POW block found! hash=0x594e9d4193a4cc0f8b7ce0037b2e989123662c43a733d1d352d380d8762078a5, number=8, totalDifficulty=1049921

So the CLs synced up perfect on the merge:
lodestar 0:

Feb-14 09:12:10.500 []                 info: Synced - slot: 64 - head: 64 0xd8fb…3de9 - execution: valid(0xd56e…80c7) - finalized: 0x0000…0000:0 - peers: 1
Feb-14 09:12:13.501 []                 info: Synced - slot: 65 - head: 65 0x853f…db7b - execution: valid(0x18f0…0ccd) - finalized: 0x0000…0000:0 - peers

lodestar 1:

eb-14 09:12:07.500 []                 info: Synced - slot: 63 - head: 63 0xefc3…55d6 - finalized: 0x0000…0000:0 - peers: 1
Feb-14 09:12:10.500 []                 info: Synced - slot: 64 - head: 64 0xd8fb…3de9 - execution: valid(0xd56e…80c7) - finalized: 0x0000…0000:0 - peers: 1
Feb-14 09:12:13.500 []                 info: Synced - slot: 65 - head: 65 0x853f…db7b - execution: valid(0x18f0…0ccd) - finalized: 0x0000…0000:0 -

@g11tech
Copy link
Contributor Author

g11tech commented Feb 14, 2022

UPDATE Scenario 3:

Multi client scenario , lodestar <> geth, 2 teku <> geth, teku <> nethermind

after couple of ACCEPTED statuses which lodestar correctly rejected,

Feb-14 15:45:16.003 [NETWORK]          info: Unsuscribing to random attnets from prev fork prevFork=phase0
Feb-14 15:45:22.001 []                 info: Synced - slot: 96 - head: 96 0xcbd6…5d38 - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:45:28.195 [CHAIN]           error: Block error slot=97, errCode=BLOCK_ERROR_EXECUTION_ERROR code=BLOCK_ERROR_EXECUTION_ERROR, errorMessage=not safe to import not yet validated payload within 128 of currentSlot, status=ACCEPTED
Error: BLOCK_ERROR_EXECUTION_ERROR
    at verifyBlockStateTransition (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:239:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at verifyBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:45:40)
    at processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/index.ts:64:32)
    at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)
Feb-14 15:45:28.195 [NETWORK]         error: Error receiving block slot=97, peer=16Uiu2HAm8hryNHHuXxbS3rPMNTqehYaJgCGbcMctj5mTuGkZKAht code=BLOCK_ERROR_EXECUTION_ERROR, errorMessage=not safe to import not yet validated payload within 128 of currentSlot, status=ACCEPTED
Error: BLOCK_ERROR_EXECUTION_ERROR
    at verifyBlockStateTransition (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:239:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at verifyBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:45:40)
    at processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/index.ts:64:32)
    at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)
Feb-14 15:45:34.000 []                 info: Synced - slot: 97 (skipped 1) - head: 96 0xcbd6…5d38 - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:45:46.001 []                 info: Synced - slot: 98 - head: 98 0xf6eb…a9fa - execution: valid(0xcd96…41fb) - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:45:52.214 [CHAIN]           error: Block error slot=97, errCode=BLOCK_ERROR_EXECUTION_ERROR code=BLOCK_ERROR_EXECUTION_ERROR, errorMessage=not safe to import not yet validated payload within 128 of currentSlot, status=ACCEPTED
Error: BLOCK_ERROR_EXECUTION_ERROR
    at verifyBlockStateTransition (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:239:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at verifyBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/verifyBlock.ts:45:40)
    at processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/index.ts:64:32)
    at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)
Feb-14 15:45:52.214 [SYNC]            error: Removing unknown parent block root=0x2748924ae00db4db9c70f73438fb14620c413c67fcb1d545f913fe0232dbc651, slot=97
Feb-14 15:45:52.214 [SYNC]            error: Removing unknown parent block root=0xe2048d1517bbad645ff6331b0e1c6b8fb780df697c487e83eabbed105d963586, slot=99
Feb-14 15:45:58.001 []                 info: Synced - slot: 99 (skipped 1) - head: 98 0xf6eb…a9fa - execution: valid(0xcd96…41fb) - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:46:10.001 []                 info: Synced - slot: 100 - head: 100 0x5837…e692 - execution: valid(0xba49…3e43) - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:46:16.042 [CHAIN]            warn: Error processing attestations from block slot=101, erroredAttestations=2 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"UNKNOWN_HEAD_BLOCK","beaconBlockRoot":"0x2748924ae00db4db9c70f73438fb14620c413c67fcb1d545f913fe0232dbc651"}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/usr/app/node_modules/@chainsafe/lodestar-fork-choice/src/forkChoice/forkChoice.ts:880:13)
    at ForkChoice.validateOnAttestation (/usr/app/node_modules/@chainsafe/lodestar-fork-choice/src/forkChoice/forkChoice.ts:812:10)
    at ForkChoice.onAttestation (/usr/app/node_modules/@chainsafe/lodestar-fork-choice/src/forkChoice/forkChoice.ts:466:10)
    at importBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/importBlock.ts:139:26)
    at processBlock (/usr/app/node_modules/@chainsafe/lodestar/src/chain/blocks/index.ts:65:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Timeout.JobItemQueue.runJob [as _onTimeout] (/usr/app/node_modules/@chainsafe/lodestar/src/util/queue/itemQueue.ts:85:22)
Feb-14 15:46:22.000 []                 info: Synced - slot: 101 - head: 101 0xe973…784b - execution: valid(0x3a55…d8fb) - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:46:34.000 []                 info: Synced - slot: 102 - head: 102 0x4c4d…aa17 - execution: valid(0xcb2b…5948) - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:46:46.000 []                 info: Synced - slot: 103 - head: 103 0x3c81…5614 - execution: valid(0x0536…ebed) - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:46:58.001 []                 info: Synced - slot: 104 - head: 104 0xc9bd…bb1a - execution: valid(0x8b7b…9512) - finalized: 0x0000…0000:0 - peers: 3
Feb-14 15:47:10.001 []                 info: Synced - s

ELs synced up and the merge completed

image

@dapplion dapplion added the scope-security Issues that fix security issues: DOS, key leak, CVEs. label May 10, 2022
@g11tech
Copy link
Contributor Author

g11tech commented Sep 13, 2022

closing as issue as issue not relavant for now

@g11tech g11tech closed this as completed Sep 13, 2022
@dapplion
Copy link
Contributor

Leaving note for future reference: there's an effort to run Kurtosis in house, but not as part of this repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@g11tech g11tech

Labels
scope-security Issues that fix security issues: DOS, key leak, CVEs.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dapplion @g11tech