Merge pre v2.57 #81
Merged
Merge pre v2.57 #81
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Alex Sharp <[email protected]>
…retire (erigontech#9068) Co-authored-by: Alex Sharp <[email protected]>
Co-authored-by: Alex Sharp <[email protected]>
* Added /validators retrieval to beacon api * fixed race in sentinel * fixed possible UB in gossip management
This adds a simulator object with implements the SentryServer api but takes objects from a pre-existing snapshot file. If the snapshot is not available locally it will download and index the .seg file for the header range being asked for. It is created as follows: ```go sim, err := simulator.NewSentry(ctx, "mumbai", dataDir, 1, logger) ``` Where the arguments are: * ctx - a callable context where cancel will close the simulator torrent and file connections (it also has a Close method) * chain - the name of the chain to take the snapshots from * datadir - a directory potentially containing snapshot .seg files. If not files exist in this directory they will be downloaded * num peers - the number of peers the simulator should create * logger - the loger to log actions to It can be attached to a client as follows: ```go simClient := direct.NewSentryClientDirect(66, sim) ``` At the moment only very basic functionality is implemented: * get headers will return headers by range or hash (hash assumes a pre-downloaded .seg as it needs an index * the header replay semantics need to be confirmed * eth 65 and 66(+) messaging is supported * For details see: `simulator_test.go More advanced peer behavior (e.g. header rewriting) can be added Bodies/Transactions handling can be added
This change introduces additional processes to manage snapshot uploading for E2 snapshots: ## erigon snapshots upload The `snapshots uploader` command starts a version of erigon customized for uploading snapshot files to a remote location. It breaks the stage execution process after the senders stage and then uses the snapshot stage to send uploaded headers, bodies and (in the case of polygon) bor spans and events to snapshot files. Because this process avoids execution in run signifigantly faster than a standard erigon configuration. The uploader uses rclone to send seedable (100K or 500K blocks) to a remote storage location specified in the rclone config file. The **uploader** is configured to minimize disk usage by doing the following: * It removes snapshots once they are loaded * It aggressively prunes the database once entities are transferred to snapshots in addition to this it has the following performance related features: * maximizes the workers allocated to snapshot processing to improve throughput * Can be started from scratch by downloading the latest snapshots from the remote location to seed processing ## snapshots command Is a stand alone command for managing remote snapshots it has the following sub commands * **cmp** - compare snapshots * **copy** - copy snapshots * **verify** - verify snapshots * **manifest** - manage the manifest file in the root of remote snapshot locations * **torrent** - manage snapshot torrent files
implement the `txpool_contentFrom` rpc, used to retrieve the specified address's tx contents. --------- Signed-off-by: jsvisa <[email protected]>
Co-authored-by: Alex Sharp <[email protected]>
…ation (erigontech#9089) Adds unit tests for: - Bor Heimdall Stage - `checkHeaderExtraData` - at end of each sprint verifies that the validators in the header extra data matches the selected proposers from the heimdall span - 1 test for selected proposers length mismatch - 1 test for selected proposers bytes mismatch - BorHeimdall Stage - `persistValidatorSets` - verifies that each header is created by a validator in the validator set - in such situation we set the unwind point
…erigontech#9096) Heimdall prepares the next span a number of sprints before the current span ends. Currently we always fetch the next span regardless of which sprint we are in during the current span. This causes a liveness issue due to how the Heimdall client works (it infinitely retries until it fetches a span - this issue will be fixed in a separate PR). This PR fixes this by matching what bor does - it fetches the next span only in the last sprint of the current span. Changes: - Adds a unit test for the above - Adds a new function BlockInLastSprintOfSpan - Some code reorg and cleanup - moves the span num related functions from the bor package to the span sub package for better logical grouping
follow up on naming as suggested here erigontech#9096 (review)
Corresponds to the client fix in this PR description - erigontech#9096 (comment)
…se (erigontech#9103) Users reported this error ``` [bor.heimdall] an error while trying fetching path=clerk/event-record/list attempt=5 error="unexpected end of JSON input" ``` Which may happen if: 1. Heimdall is behind and not sync-ed - for more info check maticnetwork/heimdall#993 2. Or the header time erigon is sending is far into the future The logs in this PR will help us see which of the 2 is the culprit but most likely it is 1. We will investigate further 2. if it ever happens. Changes: 1. Improves logging upon heimdall client retries - prints out the full url that failed. 2. Fixes a bug where the body was incorrectly checked if it is empty - `len(body) == 0` vs `body == nil` 3. Unit test for the bug regression 4. Adds a log to indicate to users to check their heimdall process if they run into this scenario since that may be the culprit Example output with new logs <img width="1465" alt="Screenshot 2023-12-29 at 20 16 57" src="https://github.com/ledgerwatch/erigon/assets/94537774/1ebfde68-aa93-41d6-889a-27bef5414f25">
…eacon/rewards/sync_committee/{block_id} (erigontech#9102)
* Changed slightly archive format (again)
* Added all of the remaining rewards endpoints
…h#9099) Mdbx now takes a logger - but this has not been pushed to all callers - meaning it had an invalid logger This fixes the log propagation. It also fixed a start-up issue for http.enabled and txpool.disable created by a previous merge
* Fixed mispelling in json fields * Added CORS
* move mocks to the owner packages * squash single file packages * move types to more appropriate files * remove unused mocks
…h#9240) If any DB method is called while Close() is waiting for db.kv.Close() (it waits for ongoing method calls/transactions to finish) a panic: "WaitGroup is reused before previous Wait has returned" might happen. Use context cancellation to ensure that new method calls immediately return during db.kv.Close().
seems we just choose the first 8 trackers in https://github.com/ledgerwatch/trackerslist/blob/master/trackers_best.txt.
This test uses a pre-built database to test Erigon's behaviour on ctrl-c during the block download phase (Caplin sync).
…h#9215) `save-state` and `set-output` commands used in GitHub Actions are deprecated and [GitHub recommends using environment files](https://github.blog/changelog/2023-07-24-github-actions-update-on-save-state-and-set-output-commands/). This PR updates the usage of `set-output` to `$GITHUB_OUTPUT` Instructions for envvar usage from GitHub docs: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-output-parameter
Current fork digest for Görli has changed after the [Dencun upgrade](https://blog.ethereum.org/2024/01/10/goerli-dencun-announcement) on 17 Jan.
Initial support of the upcoming Napoli hard fork on Polygon – see [PIP-33](https://forum.polygon.technology/t/pip-33-napoli-upgrade). Per [PIP-31](https://github.com/maticnetwork/Polygon-Improvement-Proposals/blob/main/PIPs/PIP-31.md), it parallels the [Cancun](https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md) upgrade of Ethereum, but does not include [EIP-4788](https://eips.ethereum.org/EIPS/eip-4788), [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844), [EIP-7516](https://eips.ethereum.org/EIPS/eip-7516). In other words, Napoli includes [EIP-1153](https://eips.ethereum.org/EIPS/eip-1153), [EIP-5656](https://eips.ethereum.org/EIPS/eip-5656), [EIP-6780](https://eips.ethereum.org/EIPS/eip-6780) from Cancun. This PR implements [PIP-31](https://github.com/maticnetwork/Polygon-Improvement-Proposals/blob/main/PIPs/PIP-31.md), [PIP-16: Transaction Dependency Data](https://github.com/maticnetwork/Polygon-Improvement-Proposals/blob/main/PIPs/PIP-16.md) (by merging `ParallelUniverseBlock` into `NapoliBlock`; the bulk of PIP-16 was implemented in PR erigontech#8037), and [PIP-27: Precompiled for secp256r1 Curve Support](https://github.com/maticnetwork/Polygon-Improvement-Proposals/blob/main/PIPs/PIP-27.md) ([EIP-7212](https://eips.ethereum.org/EIPS/eip-7212); see also maticnetwork/bor#1069 & ethereum/go-ethereum#27540). --------- Co-authored-by: Anshal Shukla <[email protected]>
This should help with less frequent lock/unlock. Following from an earlier "TODO"
) In the previous code WaitGroup db.wg.Add(), Wait() and db.closed were not treated in sync. In particular, it was theoretically possible to first check closed, then set closed and Wait, and then call wg.Add() while waiting (leading to WaitGroup panic). In theory it was also possible that db.env.BeginTxn() is called on a closed or nil db.env, because db.wg.Add() was called only after BeginTxn (db.wg.Wait() could already return). WaitGroup is replaced with a Cond variable. Now it is not possible to increase the active transactions count on a closed database. It is also not possible to call BeginTxn on a closed database.
"[EngineBlockDownloader] Downloading PoS headers... height=unknown" might be [confusing](https://discord.com/channels/687972960811745322/1001501454076477450/1197365760125841550) for users.
This is a bit of a hack, and isn't how I'd like to see this change maintained. But, it's also pretty simple/direct and unlikely to create conflicts, so going that way for the moment.
This message is spamming the log even when an unwind is not happening. This is because we forgot to check if we were not in the unwind path before printing.
This simply pulls the 'Optimism' fields into their own section in a few structs. This prevents realigning the rest of the fields and will hopefully reduce merge conflicts.
boyuan-chen
approved these changes
Feb 8, 2024
mmontour1306
approved these changes
Feb 12, 2024
jyellick
pushed a commit
that referenced
this pull request
Feb 26, 2024
Fix `eth_call` RPC relaying state override param
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This brings things up to date just one commit before the v2.57 release.