clients/besu: add Shanghai fork and fix issue with duplicated node keys #674
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fjl
changed the title
|
Is the nodekey deterministic in besu? |
|
If you place a key in the data directory besu will read that and use it to generate the node id. If you use the same key for the bootnode and node 2 (usually used in hive to test the sync) they will have the same node address and node id. Which eventually will prevent these nodes from peering with each other. |
|
I understand. AFAIK, hive does not place a key in the data directory, and besu should generate a random key by itself. |
gfukushima
changed the base branch from
engine-withdrawals-2
to
all-withdrawals
|
Hi @fjl, is there anything this PR is missing to be accepted? Right now everyone who runs the hive withdrawals suite need to add those changes manually for Besu. Thanks |
|
Sorry, I just wanted to confirm why the nodekey is used in clique mining mode. But it seems this is not easy to fix in besu, so we can apply this workaround for now. |
fjl
changed the title
fjl
pushed a commit
that referenced
this pull request
Jan 23, 2023
…ys (#674) * Add bootnode check to avoid duplicated enode address * Add shanghaiTime to mapper.jq Co-authored-by: gfukushima <[email protected]>
Rjected
pushed a commit
to Rjected/hive
that referenced
this pull request
Feb 7, 2023
…ys (ethereum#674) * Add bootnode check to avoid duplicated enode address * Add shanghaiTime to mapper.jq Co-authored-by: gfukushima <[email protected]>
alexshliu
added a commit
to taikoxyz/hive_old
that referenced
this pull request
Feb 22, 2023
* all: Go workspace support and dependency sync (ethereum#664) This change adds a top-level go.work file containing all Go-based simulators as well as the hive module. Joining the modules in a workspace makes it possible to easily share code among simulators. When simulators share code, the imported simulator modules need to be made available in the docker build environment. To facilitate this, the docker image build context can now be configured using a special file in the simulator directory. Co-authored-by: Felix Lange <[email protected]> * internal/libdocker: add optional registry authentication in builder (ethereum#662) Hive can now authenticate to the registry when building images. This allows users to run tests against private docker images in cases where they may not want a pre-prod image to be exposed publicly. Co-authored-by: Felix Lange <[email protected]> * simulators/ethereum/consensus: Fix Consensus Tests (ethereum#668) simulators/ethereum/consensus: Add TTD to Merge config * clients/besu: change sync mode to snap sync instead of fast sync (ethereum#666) * clients/besu: update config to address rpc and send-legacy-transaction rpc-compat failure (ethereum#661) Signed-off-by: mark-terry <[email protected]> Co-authored-by: Felix Lange <[email protected]> * test.sh: fix typo (ethereum#670) * go.work: disable top-level go.work by default * simulators/ethereum/engine: adding tests for Shanghai (ethereum#676) * simulators/ethereum/engine: Withdrawals Sanity test * simulators/ethereum/engine: CLMock changes * simulators/ethereum/engine: Add timestamp global * simulators/ethereum/engine: Withdrawals required changes * simulators/ethereum/engine: Fix request payload * simulators/ethereum/engine: Withdrawals tests * simulators/ethereum/engine: Withdrawals tests update * simulators/ethereum/engine: Test spec interface * simulators/ethereum/engine: Add withdrawals test cases * simulators/ethereum/engine: Withdraw many accounts fix * simulators/ethereum/engine: Withdrawals sync tests * simulators/ethereum/engine: Withdrawals re-org * simulators/ethereum/engine: More withdrawals tests * simulators/ethereum/engine: Variable timestamp increments on CLMock * simulators/ethereum/engine: Withdrawals reorg changes * simulators/ethereum/engine: withdrawals reorg tests * simulators/ethereum/engine: Add LVH check * simulators/ethereum/engine: Dynamic genesis * simulators/ethereum/engine: Fix HTTP Transport usage * simulators/ethereum/engine: Roundtrip change * simulator/ethereum/engine: Add txs to withdrawal tests * simulators/ethereum/engine: Decrease withdrawals count * simulators/ethereum/engine: Withdrawals test change * simulators/ethereum/engine: Withdrawals, comments * simulators/ethereum/engine: fix lvh in withdrawals * simulators/ethereum/engine: go mod tidy * simulators/ethereum/engine: Fix lvh check on genesis * simulators/ethereum/engine: Fix long sync test * simulators/ethereum/engine: Fix sync tests take 2 * simulators/ethereum/engine: Add blockValue to getPayload * simulators/ethereum/engine: withdrawals, wei to gwei * simulators/ethereum/engine: fix genesis block * simulators/ethereum/engine: implement error code expect check * simulators/ethereum/engine: withdrawals, expect errors on invalid version * simulators/ethereum/engine: Withdrawals readme * simulators/ethereum/engine: update withdrawals readme * simulators/ethereum/engine: update test parameter, description * simulators/ethereum/engine: update branch * simulators/ethereum/engine: Support querying txs from client * simulators/ethereum/engine: Add eip 3860 invalid tx test * simulators/ethereum/engine: fix gaslimit everywhere Co-authored-by: marioevz <[email protected]> * clients/nethermind: add Shanghai fork timestamp * clients/go-ethereum: add Shanghai for timestamp * clients/erigon: add Shanghai fork timestamp * simulators/ethereum/engine: fix nil pointer exception (ethereum#678) * clients/ethereumjs: fix build and add Shanghai fork configuration (ethereum#679) * clients/besu: add Shanghai fork and fix issue with duplicated node keys (ethereum#674) * Add bootnode check to avoid duplicated enode address * Add shanghaiTime to mapper.jq Co-authored-by: gfukushima <[email protected]> * simulators/eth2: add common library and tests for withdrawals (ethereum#681) This adds a simulator and tests for the Capella beacon chain fork (including withdrawals). There are no client changes here. Some CL client definitions will not pass these tests in their default branch and will require an explicit capella branch specification. As part of factoring out the eth2 testnet setup code into a standalone library, this change introduces a 'Go workspace' containing all eth2 simulator sub-modules. When hive builds the simulator, it will now use ./simulators/eth2 as the base directory and will locate the ./simulators/eth2/common module via the go.work file. When building the simulators outside of docker, the go tool locates the go.work file as well. Co-authored-by: marioevz <[email protected]> * clients/lodestar: new beacon node and validator client definition (ethereum#680) * simulators/eth2/withdrawals: rename suite * simulators/eth2/common: avoid <nil> value for HIVE_SHANGHAI_TIMESTAMP * simulators/eth2/withdrawals: clean up go.mod * simulators/eth2/testnet: clean up go.mod * simulators/eth2/engine: clean up go.mod * simulators/eth2: go mod tidy * simulators/ethereum/engine: set HIVE_MERGE_BLOCK_ID before shanghai fork (ethereum#684) * README.md, docs: fix typos in documentation (ethereum#683) * simulators/eth2/common: handle client branch when getting validator name * clients/nethermind: add shanghai EIPs to mapper (ethereum#686) * simulators/eth2/withdrawals: check exact partial withdrawn balance expected (ethereum#685) This adds precise checking of EL balances after withdrawals have been processed. * clients/nethermind: use terminal block hash/number in config (ethereum#687) * hivesim: make test matching case insensitive (ethereum#689) * simulators/eth2/withdrawals: handle missed slots in test (ethereum#690) There was an issue in the withdrawals calculations so that, when the chain missed a slot, it aborted with error, but it's normal to miss a block in a slot. Now the computing method allows to skip a slot, and the calculations are still correct, the test passes for client combinations that might occasionally miss a slot. * simulators/ethereum/engine: Remove PoW config from withdrawals (ethereum#691) * clients/nimbus: fix nimbus (ethereum#693) * simulators/eth2: support 12 second slots, add test for large withdrawal amounts, more fixes (ethereum#694) * cmd/hiveview: viewer app update (ethereum#696) A big update for cmd/hiveview. - All external resources are now bundled in hiveview, so the app can run without Internet access. - External CSS/JS libraries have been updated to the latest version. - The index page and suite page are now separate documents. This makes browser navigation work better and is also a bit cleaner in the JS implementation. - When loading suite data and large log files, a progress bar is shown. - The 'expand' button in the test list was a bit confusing because it had a green icon when closed, and a red icon when opened. Since this was the only colored element on the page, it could be mistaken for test status. In the new version, the list of tests now shows red arrows for failed tests and gray ones for passed tests. - It is now possible to link to a test. When clicking on a test in the list, the test becomes 'selected', which is indicated by a yellow background. The selected test is stored in the page URL. - Output from tests is now limited to 50 lines to avoid freezing the browser. If there is more output, a link to the log viewer is shown. The log viewer can now display the output of a single test. - In the log viewer, a link back to the test is shown. This is useful when sharing a client log file with someone else, since they can go and check out the test case that generated the log. - Test output lines starting with `FAIL:`, `ERROR:` or `Error:` are shown in bold text. - The suite page now shows the duration of each test, as well as the overall suite duration and start time. - The new page header shows the hive commit and build date. This information is taken from `hive.json` in the workspace, so it will update whenever hive runs. - When a simulation run times out, hive now stores a `timeout` flag in the test results. The viewer app shows timeouts as a special kind of failure. - All timestamps are once again shown as local time. They were changed to UTC in commit beebe58, but I've since figured out how to sort by the ISO string and only convert to local time for display. * simulators/ethereum/consensus: update for Shanghai * cmd/hiveview: fix hourglass in Safari * simulators/ethereum/graphql: align tests with current spec (ethereum#692) Use long for timestamp and remove two SyncState fields. * cmd/hiveview: responsive layout for table views (ethereum#697) This enables the 'responsive' option for DataTable. With these changes, the viewer app is somewhat usable on small screens without horizontal scrolling. * simulators/ethereum/engine: INVALID_BLOCK_HASH is deprecated in Shanghai (ethereum#699) INVALID_BLOCK_HASH is deprecated in Shanghai * simulators/ethereum/engine: tests for getPayloadBodies*, exchangeCapabilities (ethereum#698) Adds tests for engine_getPayloadBodiesByRangeV1, engine_getPayloadBodiesByHashV1 and engine_exchangeCapabilities. * cmd/hiveview: small UI improvements (ethereum#701) This fixes a few small issues I noticed after deploying the new UI to hivetests.ethdevops.io. * cmd/hiveview: improve error highlight * cmd/hiveview: fix trim of /results/ in log title * cmd/hiveview: improve line highlight in viewer * cmd/hiveview: improve column widths on index page * cmd/hiveview: add scroll-margin for suite table highlight * cmd/hiveview: fixing more stuff in the UI (ethereum#702) * cmd/hiveview: fix some more CSS issues in viewer * cmd/hiveview: fix crash in viewer JS for big files * cmd/hiveview: fix margin * cmd/hiveview: fix row highlight * cmd/hiveview: prevent wrap of status * cmd/hiveview: enlarge suite name column * cmd/hiveview: prevent JSON-RPC error message highlight * cmd/hiveview: render hidden columns in test details * cmd/hiveview: print listing generator errors in -serve mode * cmd/hiveview: skip hive.json * cmd/hiveview: hide client list in details box when test has no clients * simulators/ethereum/engine: increase tests coverage (ethereum#700) Increase tests coverage to match expectations of checklist here: https://hackmd.io/@n0ble/rywPcPTci * README.md: update description of hivetests2 server * docs: add info about ethereum/engine simulator * docs: remove accidental \ * cmd/hiveview: UI improvements (ethereum#705) Another round of fixes for minor issues in the hiveview UI. * cmd/hiveview: fix typo in loader * cmd/hiveview: add spinner on index page * cmd/hiveview: include test id in client log link This makes it possible to go back to the test that created the client log. * cmd/hiveview: show test number in link back to suite * cmd/hiveview: improve line highlight * cmd/hiveview: fix scroll-margin in tests table * clients/nimbus: clean Dockerfile, accept branch arg (ethereum#704) * cmd/hiveview: JS/CSS bundling with esbuild (ethereum#708) This should fix issues with browser caching in the frontend. Ever since the app was split up into multiple JS files, the page would occasionally fail to load because the browser did not refresh some of the JS files when they were modified. In `hiveview -serve` mode, the new bundling system works like this: - In deploy.go, we maintain a static list of 'bundle targets'. The targets correspond with the app's entry point JS/CSS files. - Whenever one of the HTML files is requested, the server traverses the document and replaces references to bundle targets with a path to the built bundle. For example, <script src="/lib/app.js"> is replaced by <script src="/bundle/app.XXXX.js"> where XXXX is the bundle hash. - Bundle outputs are stored in memory, and bundles are rebuilt whenever necessary, i.e. whenever one of their inputs has changed. Making the JS code work with esbuild required some dependency re-structuring. All external JS libraries were updated to the ES module version provided by upstream. I had previously worked around the lack of ES module compatibility by providing a small 'module wrapper' for each library, but this workaround broke under esbuild. However, upgrading everything to ES modules created another issue: most libraries use named imports (`import "jquery"`), which can't be resolved directly by the browser. Making the app load from un-bundled sources requires an importmap. So, when use of the bundle is disabled by the -assets.nobundle flag, hiveview writes an importmap into the document. The same importmap definition is also used by esbuild to resolve named imports at build time. All this means the app can no longer be served directly from assets/. If you just pointed nginx at the assets/ directory and opened index.html, loading of app.js would fail because the document doesn't have an importmap. And it wouldn't use the bundle either. Serving the frontend with a server that isn't `hiveview -serve`, like we do for production hive, now requires running `hiveview -deploy <dir>` to install the fully-built app into a new directory. * cmd/hiveview: remove arrow in test table header * cmd/hiveview: improve esbuild setup (ethereum#710) In ethereum#708, I introduced a lot of logic to create CSS/JS bundles on demand. It turns out that the way I implemented bundling was a bit misguided, because I was operating under the basic assumption that esbuild will always create exactly one output file for every entrypoint. This breaks when it tries to copy assets or performs code splitting. So, instead of doing incremental builds ourself, just let esbuild do its thing and copy all of the files it produces into the output FS. As a bonus, enabling code splitting makes it possible to have one entry point for each page. * cmd/hiveview: disable caching for hive.json fetch * cmd/hiveview: enable sourcemaps in bundle output * cmd/hiveview: fix file name in error messages * cmd/hiveview: link to history instead of diff in header * simulators/ethereum/engine: test using getPayloadBodiesByRange/Hash after client sync (ethereum#709) Adds a test that verifies engine_getPayloadBodiesByRangeV1 and engine_getPayloadBodiesByHashV1 behavior after client syncs to a given canonical chain. * simulators/eth2/common,withdrawals: Verify Engine API Calls from Beacon Client (ethereum#712) * simulators/eth2/common: Add library to check expired endpoints based on timestamp * simulators/eth2/withdrawals: verify V1 endpoints indeed deprecated by beacon clients at capella * simulators/eth2/common: fix check on empty payload attributes * simulators/ethereum/engine: send test transactions in batch (ethereum#713) - Test transactions are now sent in batch for the withdrawals re-org tests, which leads to faster execution times. - Re-Org tests now have a higher timeout limit (EthereumJS was failing the tests only due to timeout). * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/graphql (ethereum#718) build(deps): bump golang.org/x/net in /simulators/ethereum/graphql Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/smoke/network (ethereum#719) build(deps): bump golang.org/x/net in /simulators/smoke/network Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/rpc (ethereum#720) build(deps): bump golang.org/x/net in /simulators/ethereum/rpc Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/consensus (ethereum#721) build(deps): bump golang.org/x/net in /simulators/ethereum/consensus Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/smoke/clique (ethereum#725) build(deps): bump golang.org/x/net in /simulators/smoke/clique Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/engine (ethereum#726) build(deps): bump golang.org/x/net in /simulators/ethereum/engine Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/rpc-compat (ethereum#722) build(deps): bump golang.org/x/net in /simulators/ethereum/rpc-compat Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/devp2p (ethereum#723) build(deps): bump golang.org/x/net in /simulators/devp2p Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bump golang.org/x/net from 0.5.0 to 0.7.0 (ethereum#724) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.5.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.5.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/containerd/containerd from 1.6.6 to 1.6.18 (ethereum#716) Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.6.6 to 1.6.18. - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.6.6...v1.6.18) --- updated-dependencies: - dependency-name: github.com/containerd/containerd dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/smoke/genesis (ethereum#727) build(deps): bump golang.org/x/net in /simulators/smoke/genesis Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/sync (ethereum#728) build(deps): bump golang.org/x/net in /simulators/ethereum/sync Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * clients/ethereumjs: add support for loading chain.rlp (ethereum#729) --------- Signed-off-by: mark-terry <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: protolambda <[email protected]> Co-authored-by: Felix Lange <[email protected]> Co-authored-by: mrod502 <[email protected]> Co-authored-by: Mario Vega <[email protected]> Co-authored-by: Gabriel-Trintinalia <[email protected]> Co-authored-by: mark-terry <[email protected]> Co-authored-by: Bjerg <[email protected]> Co-authored-by: Gabriel Fukushima <[email protected]> Co-authored-by: gfukushima <[email protected]> Co-authored-by: omahs <[email protected]> Co-authored-by: Danno Ferrin <[email protected]> Co-authored-by: Andrew Ashikhmin <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: acolytec3 <[email protected]>
alexshliu
added a commit
to taikoxyz/hive_old
that referenced
this pull request
Feb 22, 2023
* all: Go workspace support and dependency sync (ethereum#664) This change adds a top-level go.work file containing all Go-based simulators as well as the hive module. Joining the modules in a workspace makes it possible to easily share code among simulators. When simulators share code, the imported simulator modules need to be made available in the docker build environment. To facilitate this, the docker image build context can now be configured using a special file in the simulator directory. Co-authored-by: Felix Lange <[email protected]> * internal/libdocker: add optional registry authentication in builder (ethereum#662) Hive can now authenticate to the registry when building images. This allows users to run tests against private docker images in cases where they may not want a pre-prod image to be exposed publicly. Co-authored-by: Felix Lange <[email protected]> * simulators/ethereum/consensus: Fix Consensus Tests (ethereum#668) simulators/ethereum/consensus: Add TTD to Merge config * clients/besu: change sync mode to snap sync instead of fast sync (ethereum#666) * clients/besu: update config to address rpc and send-legacy-transaction rpc-compat failure (ethereum#661) Signed-off-by: mark-terry <[email protected]> Co-authored-by: Felix Lange <[email protected]> * test.sh: fix typo (ethereum#670) * go.work: disable top-level go.work by default * simulators/ethereum/engine: adding tests for Shanghai (ethereum#676) * simulators/ethereum/engine: Withdrawals Sanity test * simulators/ethereum/engine: CLMock changes * simulators/ethereum/engine: Add timestamp global * simulators/ethereum/engine: Withdrawals required changes * simulators/ethereum/engine: Fix request payload * simulators/ethereum/engine: Withdrawals tests * simulators/ethereum/engine: Withdrawals tests update * simulators/ethereum/engine: Test spec interface * simulators/ethereum/engine: Add withdrawals test cases * simulators/ethereum/engine: Withdraw many accounts fix * simulators/ethereum/engine: Withdrawals sync tests * simulators/ethereum/engine: Withdrawals re-org * simulators/ethereum/engine: More withdrawals tests * simulators/ethereum/engine: Variable timestamp increments on CLMock * simulators/ethereum/engine: Withdrawals reorg changes * simulators/ethereum/engine: withdrawals reorg tests * simulators/ethereum/engine: Add LVH check * simulators/ethereum/engine: Dynamic genesis * simulators/ethereum/engine: Fix HTTP Transport usage * simulators/ethereum/engine: Roundtrip change * simulator/ethereum/engine: Add txs to withdrawal tests * simulators/ethereum/engine: Decrease withdrawals count * simulators/ethereum/engine: Withdrawals test change * simulators/ethereum/engine: Withdrawals, comments * simulators/ethereum/engine: fix lvh in withdrawals * simulators/ethereum/engine: go mod tidy * simulators/ethereum/engine: Fix lvh check on genesis * simulators/ethereum/engine: Fix long sync test * simulators/ethereum/engine: Fix sync tests take 2 * simulators/ethereum/engine: Add blockValue to getPayload * simulators/ethereum/engine: withdrawals, wei to gwei * simulators/ethereum/engine: fix genesis block * simulators/ethereum/engine: implement error code expect check * simulators/ethereum/engine: withdrawals, expect errors on invalid version * simulators/ethereum/engine: Withdrawals readme * simulators/ethereum/engine: update withdrawals readme * simulators/ethereum/engine: update test parameter, description * simulators/ethereum/engine: update branch * simulators/ethereum/engine: Support querying txs from client * simulators/ethereum/engine: Add eip 3860 invalid tx test * simulators/ethereum/engine: fix gaslimit everywhere Co-authored-by: marioevz <[email protected]> * clients/nethermind: add Shanghai fork timestamp * clients/go-ethereum: add Shanghai for timestamp * clients/erigon: add Shanghai fork timestamp * simulators/ethereum/engine: fix nil pointer exception (ethereum#678) * clients/ethereumjs: fix build and add Shanghai fork configuration (ethereum#679) * clients/besu: add Shanghai fork and fix issue with duplicated node keys (ethereum#674) * Add bootnode check to avoid duplicated enode address * Add shanghaiTime to mapper.jq Co-authored-by: gfukushima <[email protected]> * simulators/eth2: add common library and tests for withdrawals (ethereum#681) This adds a simulator and tests for the Capella beacon chain fork (including withdrawals). There are no client changes here. Some CL client definitions will not pass these tests in their default branch and will require an explicit capella branch specification. As part of factoring out the eth2 testnet setup code into a standalone library, this change introduces a 'Go workspace' containing all eth2 simulator sub-modules. When hive builds the simulator, it will now use ./simulators/eth2 as the base directory and will locate the ./simulators/eth2/common module via the go.work file. When building the simulators outside of docker, the go tool locates the go.work file as well. Co-authored-by: marioevz <[email protected]> * clients/lodestar: new beacon node and validator client definition (ethereum#680) * simulators/eth2/withdrawals: rename suite * simulators/eth2/common: avoid <nil> value for HIVE_SHANGHAI_TIMESTAMP * simulators/eth2/withdrawals: clean up go.mod * simulators/eth2/testnet: clean up go.mod * simulators/eth2/engine: clean up go.mod * simulators/eth2: go mod tidy * simulators/ethereum/engine: set HIVE_MERGE_BLOCK_ID before shanghai fork (ethereum#684) * README.md, docs: fix typos in documentation (ethereum#683) * simulators/eth2/common: handle client branch when getting validator name * clients/nethermind: add shanghai EIPs to mapper (ethereum#686) * simulators/eth2/withdrawals: check exact partial withdrawn balance expected (ethereum#685) This adds precise checking of EL balances after withdrawals have been processed. * clients/nethermind: use terminal block hash/number in config (ethereum#687) * hivesim: make test matching case insensitive (ethereum#689) * simulators/eth2/withdrawals: handle missed slots in test (ethereum#690) There was an issue in the withdrawals calculations so that, when the chain missed a slot, it aborted with error, but it's normal to miss a block in a slot. Now the computing method allows to skip a slot, and the calculations are still correct, the test passes for client combinations that might occasionally miss a slot. * simulators/ethereum/engine: Remove PoW config from withdrawals (ethereum#691) * clients/nimbus: fix nimbus (ethereum#693) * simulators/eth2: support 12 second slots, add test for large withdrawal amounts, more fixes (ethereum#694) * cmd/hiveview: viewer app update (ethereum#696) A big update for cmd/hiveview. - All external resources are now bundled in hiveview, so the app can run without Internet access. - External CSS/JS libraries have been updated to the latest version. - The index page and suite page are now separate documents. This makes browser navigation work better and is also a bit cleaner in the JS implementation. - When loading suite data and large log files, a progress bar is shown. - The 'expand' button in the test list was a bit confusing because it had a green icon when closed, and a red icon when opened. Since this was the only colored element on the page, it could be mistaken for test status. In the new version, the list of tests now shows red arrows for failed tests and gray ones for passed tests. - It is now possible to link to a test. When clicking on a test in the list, the test becomes 'selected', which is indicated by a yellow background. The selected test is stored in the page URL. - Output from tests is now limited to 50 lines to avoid freezing the browser. If there is more output, a link to the log viewer is shown. The log viewer can now display the output of a single test. - In the log viewer, a link back to the test is shown. This is useful when sharing a client log file with someone else, since they can go and check out the test case that generated the log. - Test output lines starting with `FAIL:`, `ERROR:` or `Error:` are shown in bold text. - The suite page now shows the duration of each test, as well as the overall suite duration and start time. - The new page header shows the hive commit and build date. This information is taken from `hive.json` in the workspace, so it will update whenever hive runs. - When a simulation run times out, hive now stores a `timeout` flag in the test results. The viewer app shows timeouts as a special kind of failure. - All timestamps are once again shown as local time. They were changed to UTC in commit beebe58, but I've since figured out how to sort by the ISO string and only convert to local time for display. * simulators/ethereum/consensus: update for Shanghai * cmd/hiveview: fix hourglass in Safari * simulators/ethereum/graphql: align tests with current spec (ethereum#692) Use long for timestamp and remove two SyncState fields. * cmd/hiveview: responsive layout for table views (ethereum#697) This enables the 'responsive' option for DataTable. With these changes, the viewer app is somewhat usable on small screens without horizontal scrolling. * simulators/ethereum/engine: INVALID_BLOCK_HASH is deprecated in Shanghai (ethereum#699) INVALID_BLOCK_HASH is deprecated in Shanghai * simulators/ethereum/engine: tests for getPayloadBodies*, exchangeCapabilities (ethereum#698) Adds tests for engine_getPayloadBodiesByRangeV1, engine_getPayloadBodiesByHashV1 and engine_exchangeCapabilities. * cmd/hiveview: small UI improvements (ethereum#701) This fixes a few small issues I noticed after deploying the new UI to hivetests.ethdevops.io. * cmd/hiveview: improve error highlight * cmd/hiveview: fix trim of /results/ in log title * cmd/hiveview: improve line highlight in viewer * cmd/hiveview: improve column widths on index page * cmd/hiveview: add scroll-margin for suite table highlight * cmd/hiveview: fixing more stuff in the UI (ethereum#702) * cmd/hiveview: fix some more CSS issues in viewer * cmd/hiveview: fix crash in viewer JS for big files * cmd/hiveview: fix margin * cmd/hiveview: fix row highlight * cmd/hiveview: prevent wrap of status * cmd/hiveview: enlarge suite name column * cmd/hiveview: prevent JSON-RPC error message highlight * cmd/hiveview: render hidden columns in test details * cmd/hiveview: print listing generator errors in -serve mode * cmd/hiveview: skip hive.json * cmd/hiveview: hide client list in details box when test has no clients * simulators/ethereum/engine: increase tests coverage (ethereum#700) Increase tests coverage to match expectations of checklist here: https://hackmd.io/@n0ble/rywPcPTci * README.md: update description of hivetests2 server * docs: add info about ethereum/engine simulator * docs: remove accidental \ * cmd/hiveview: UI improvements (ethereum#705) Another round of fixes for minor issues in the hiveview UI. * cmd/hiveview: fix typo in loader * cmd/hiveview: add spinner on index page * cmd/hiveview: include test id in client log link This makes it possible to go back to the test that created the client log. * cmd/hiveview: show test number in link back to suite * cmd/hiveview: improve line highlight * cmd/hiveview: fix scroll-margin in tests table * clients/nimbus: clean Dockerfile, accept branch arg (ethereum#704) * cmd/hiveview: JS/CSS bundling with esbuild (ethereum#708) This should fix issues with browser caching in the frontend. Ever since the app was split up into multiple JS files, the page would occasionally fail to load because the browser did not refresh some of the JS files when they were modified. In `hiveview -serve` mode, the new bundling system works like this: - In deploy.go, we maintain a static list of 'bundle targets'. The targets correspond with the app's entry point JS/CSS files. - Whenever one of the HTML files is requested, the server traverses the document and replaces references to bundle targets with a path to the built bundle. For example, <script src="/lib/app.js"> is replaced by <script src="/bundle/app.XXXX.js"> where XXXX is the bundle hash. - Bundle outputs are stored in memory, and bundles are rebuilt whenever necessary, i.e. whenever one of their inputs has changed. Making the JS code work with esbuild required some dependency re-structuring. All external JS libraries were updated to the ES module version provided by upstream. I had previously worked around the lack of ES module compatibility by providing a small 'module wrapper' for each library, but this workaround broke under esbuild. However, upgrading everything to ES modules created another issue: most libraries use named imports (`import "jquery"`), which can't be resolved directly by the browser. Making the app load from un-bundled sources requires an importmap. So, when use of the bundle is disabled by the -assets.nobundle flag, hiveview writes an importmap into the document. The same importmap definition is also used by esbuild to resolve named imports at build time. All this means the app can no longer be served directly from assets/. If you just pointed nginx at the assets/ directory and opened index.html, loading of app.js would fail because the document doesn't have an importmap. And it wouldn't use the bundle either. Serving the frontend with a server that isn't `hiveview -serve`, like we do for production hive, now requires running `hiveview -deploy <dir>` to install the fully-built app into a new directory. * cmd/hiveview: remove arrow in test table header * cmd/hiveview: improve esbuild setup (ethereum#710) In ethereum#708, I introduced a lot of logic to create CSS/JS bundles on demand. It turns out that the way I implemented bundling was a bit misguided, because I was operating under the basic assumption that esbuild will always create exactly one output file for every entrypoint. This breaks when it tries to copy assets or performs code splitting. So, instead of doing incremental builds ourself, just let esbuild do its thing and copy all of the files it produces into the output FS. As a bonus, enabling code splitting makes it possible to have one entry point for each page. * cmd/hiveview: disable caching for hive.json fetch * cmd/hiveview: enable sourcemaps in bundle output * cmd/hiveview: fix file name in error messages * cmd/hiveview: link to history instead of diff in header * simulators/ethereum/engine: test using getPayloadBodiesByRange/Hash after client sync (ethereum#709) Adds a test that verifies engine_getPayloadBodiesByRangeV1 and engine_getPayloadBodiesByHashV1 behavior after client syncs to a given canonical chain. * simulators/eth2/common,withdrawals: Verify Engine API Calls from Beacon Client (ethereum#712) * simulators/eth2/common: Add library to check expired endpoints based on timestamp * simulators/eth2/withdrawals: verify V1 endpoints indeed deprecated by beacon clients at capella * simulators/eth2/common: fix check on empty payload attributes * simulators/ethereum/engine: send test transactions in batch (ethereum#713) - Test transactions are now sent in batch for the withdrawals re-org tests, which leads to faster execution times. - Re-Org tests now have a higher timeout limit (EthereumJS was failing the tests only due to timeout). * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/graphql (ethereum#718) build(deps): bump golang.org/x/net in /simulators/ethereum/graphql Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/smoke/network (ethereum#719) build(deps): bump golang.org/x/net in /simulators/smoke/network Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/rpc (ethereum#720) build(deps): bump golang.org/x/net in /simulators/ethereum/rpc Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/consensus (ethereum#721) build(deps): bump golang.org/x/net in /simulators/ethereum/consensus Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/smoke/clique (ethereum#725) build(deps): bump golang.org/x/net in /simulators/smoke/clique Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/engine (ethereum#726) build(deps): bump golang.org/x/net in /simulators/ethereum/engine Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/rpc-compat (ethereum#722) build(deps): bump golang.org/x/net in /simulators/ethereum/rpc-compat Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/devp2p (ethereum#723) build(deps): bump golang.org/x/net in /simulators/devp2p Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bump golang.org/x/net from 0.5.0 to 0.7.0 (ethereum#724) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.5.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.5.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/containerd/containerd from 1.6.6 to 1.6.18 (ethereum#716) Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.6.6 to 1.6.18. - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.6.6...v1.6.18) --- updated-dependencies: - dependency-name: github.com/containerd/containerd dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/smoke/genesis (ethereum#727) build(deps): bump golang.org/x/net in /simulators/smoke/genesis Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0 in /simulators/ethereum/sync (ethereum#728) build(deps): bump golang.org/x/net in /simulators/ethereum/sync Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * clients/ethereumjs: add support for loading chain.rlp (ethereum#729) --------- Signed-off-by: mark-terry <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: protolambda <[email protected]> Co-authored-by: Felix Lange <[email protected]> Co-authored-by: mrod502 <[email protected]> Co-authored-by: Mario Vega <[email protected]> Co-authored-by: Gabriel-Trintinalia <[email protected]> Co-authored-by: mark-terry <[email protected]> Co-authored-by: Bjerg <[email protected]> Co-authored-by: Gabriel Fukushima <[email protected]> Co-authored-by: gfukushima <[email protected]> Co-authored-by: omahs <[email protected]> Co-authored-by: Danno Ferrin <[email protected]> Co-authored-by: Andrew Ashikhmin <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: acolytec3 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes the tests where hive spins up a second instance of besu and sync from the bootnode. Currently node 1 and 2 have the same enode address which stops them from peering with each other
Linked to #4747