-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upstream v2.60.2 #106
Upstream v2.60.2 #106
Conversation
Signed-off-by: stayweek <[email protected]>
…es (erigontech#9805) we recently changed field which used to measure progress from `t.BytesCompleted()` to `t.Stats().BytesReadData` - and seems it doesn't include "re-hashing" progress - so progress doesn't grow in logs and looks like it's stuck. argument to change - was: `t.BytesCompleted()` is not recommended to use for "download rate calculation". We don't use this field for rate calc, we using it for progress calc
…rigontech#9857) For better integration with GH Actions added overview in the end of reports and exit with code 1 in case when met some errors.
This pr implements following beacon api https://ethereum.github.io/beacon-APIs/#/Validator/prepareBeaconCommitteeSubnet Steps: 1. once receive committee subscription request, record this subscription info associated with validator index. 2. if `isAggregator` flag is on, prepare internal data structure to collect and aggregate incoming attestation. 3. set expiry and begin to subscribe `beacon_attestation_{subnet}`, so we are capable of receiving attestation data from peer. 4. whenever receive attestation data, try aggregate it if needed.
Fix erigontech#9834 Let's check and validate blob configurations only if the txn is BlobTx Signed-off-by: jsvisa <[email protected]>
Basically, now in order to mark blobs of a block as downloaded, re-check the blocks payload used to download it against the blob store.
# Sync Committee Contribution pooling First of all, a distinction: * Sync Committee Messages are single unaggregated messages from a specific validator * Sync Committee Contributions are aggregated messages from many validators. We get these 2 messages from 2 different gossip sources, and then after validating the Gossip rules, we send everything to the pool which aggregate the `sync committee` or `contribution` into another aggregate. ## Sync Committee subscription: /eth/v1/validator/sync_committee_subscriptions The subscription just starts to subscribe to the `Sync Committee`'s Gossip channel. it is actually really simple, here is the pseudo-code for how the subscription happen for each `ValidatorIndex` requested: ```vb Function PostEthV1ValidatorSyncCommitteeSubscriptions Request: ValidatorIndicies []uint64 Get the head state of the blockchain For each subRequest in ValidatorIndicies do Compute the subnets for the validator index from subRequest using the headState For each subnet in syncnets do Construct a topic name for the sync committee subnet Subscribe to the topic with the sync committee subnet the topic with the calculated expiry time in sentinel End Function ``` ### Extras * /eth/v1/validator/contribution_and_proofs - Submit to the node a contribution to be republished on the Gossip. * /eth/v1/validator/sync_committee_contribution - Retrieve a contribution (from the pool) that we have aggregated from the gossip. * /eth/v1/beacon/pool/sync_committees - Submit to the node a sync sync committee messages to be republished on the Gossip. --------- Co-authored-by: Kewei <[email protected]>
Basically pruning was overpruning useful blobs which were needed for processing of blocks at chain tip, thus leading to degraded performance. Regressions fixed: 1) Make a copy of `GossipData` so that we do not risk republishing gossip bogus/mem corrupted data as published. (`PublishGossip` takes hold of the data bytes for a long time) 2) Blobs download not showing progress 3) Increase request dial timeout to specs 4) Made `Attestations` routine recoverable 5) Fixed inconsistency in the total active balance cache which caused a panic
we did make `downloader` more passive (erigon-driven), but it did break `--seedbox` flag also
- new method `DownloadAndSaveTorrentFile` did download data files instead of .torrent files because `WebSeeds.ByName()` return data urls - not .torrent urls - calling wrong url to download .torrent - save value to disk without `.torrent` suffix - size attack prevention - did return nil err and it stopped webseed scanning
…ts with modified bytecode (erigontech#9438) In this PR I'm introducing so-called `overlays` for doing ad-hoc state simulations of contracts which replay the state with a modified bytecode for any given block range. The new feature adds two new RPC methods: `overlay_callConstructor` and `overlay_getLogs` I've also added a few bench tests and a new postman collection which checks both new methods for the WETH contract in original and modified version. ## Motivation Overlays allow you to add your custom logic to already deployed contracts and simulate on top of them. With overlays you can create new view functions, modify existing ones, change field visibility, emit new events and query the historical data of any contract with your modified source code. Similar commercial products exist in the space and I believe making this feature opensource in erigon will make it easier for everyone to tinker with it and build new cool things on top of it ✨ ## Usage 1) compile your contract to obtain the new deployment bytecode, append the constructor args at the end, if any 2) call `overlay_callConstructor` with the result from 1) to receive the runtime bytecode 3) call `overlay_getLogs` to simulate new logs, passing 2) as state overrides ([example](https://github.com/crebsy/erigon/blob/devel/cmd/rpcdaemon/postman/Overlay_Testing.json)) 4) call `eth_call` to simulate new view function calls passing 2) as state overrides ## Ad-hoc query performance & historical back-filling The current implementation for ad-hoc range queries of `overlay_getLogs` is optimized so that irrelevant blocks are skipped entirely from the simulation context. This can lead to very fast results of a few minutes or even seconds for simulating the entire historical logs of a contract which makes any infra for back-filling redundant. However, such shortcuts are only effectively working for contracts which are not touching too many blocks of the chain's history. So depending on the contract you're trying to patch, you still might need to create a custom back-filling script which simulates the new logs with e.g. 1000 or 10000 block ranges. As a reference for a "busy" contract the simulation rate for the [Uniswap V2 router ](https://etherscan.io/address/0x7a250d5630b4cf539739df2c5dacb4c659f2488d#code) was ~ 280 blocks/s on a host with 24 cores. So while small block ranges returned relatively quickly, the simulation of the entire historical logs for this contract took a few hours on a single node. I'm not collecting any perf metrics yet for this but this could be easily added in the future for tracking better implementations and insights about contract <-> block distribution. ## Configuration 1) Enable the `ots,overlay` api namespaces by adding it to the `--http.api` flag 2) The implementation of `overlay_getLogs` is parallelized so it's good practice to increase the `--db.read.concurrency` to a higher value, e.g. 256 so other erigon goroutines are not getting blocked. ## Supported networks Tested with ethereum and polygon nodes
Fixes problems found during bor-mainnet sync tests: - FetchHeaders: paging-like fetching of headers from a peer - assumption was that a peer response will contain all e.g. 1024 headers since they can fit in < 2 MB however despite that some peers respond with only 16 headers so we need to send further requests until we get all e.g. 1024 headers - It looks like another assumption didn't hold - when we penalize a peer sentry doesn't generate a disconnect even. For that this PR creates a tracking penalizer that updates the peer tracker with a disconnect. - Removes another wrong incorrect - ErrEmptyBody validation because there are blocks which are empty, e.g. first 100,000k blocks - Sentry SentMessageById was disallowing GetBlockBodiesMsg, this PR fixes this
This PR separates the data collection layer from the `Antiquary` object to the `beaconStatesCollector`, which hopefully makes the code cleaner. It is still messy, but now we have a proper separation between the logic and the data collection layer. The reason why we have internal buffers here is that MDBX locks the entire database when writing to it, so we need to minimize the time spent in the write lock. So instead of writing the historical states on write transactions and keeping the lock for hours, we accumulate them in memory and write them in a single write transaction.
Fixes problems found during bor-mainnet sync tests: - sync.Run was missing a for loop for initial sync convergences - temporarily disabling ForkChoiceUpdate execution since we are yet to solve and add missing state sync events and spans - no need to flush storage inside block downloader (better async performance) - logging improvements
…ontech#9882) - fixes an issue where heimdall polling goroutines get terminated upon a heimdall http err which is incorrect - these polling routines should be kept alive upon errors and errors should simply be logged - adds logging - reduces cyclomatic complexity by extracting logic for polling events, spans, checkpoints --------- Co-authored-by: battlmonstr <[email protected]>
- collecting info for "Headers" stage
Implemented a network speed test to run every 180 seconds. This information is crucial for diagnostic tools. It helps us identify if a client is reaching their maximum capacity.
Fixes a whim reported in erigontech#9783
Do not open files which are bigger than 32 bytes but "has no" words in it.
Cherry pick PR erigontech#10636 to Erigon 2
…ion workflows (erigontech#10778) This PR backports improvements that we added to the E3 tests: recording runner name and db version used for testing on MongoDB database.
Merged all the work done from main branch to keep diagnostics up to date.
Cherry pick PR erigontech#10093 into the release branch Co-authored-by: Minhyuk Kim <[email protected]>
Cherry pick PR erigontech#10825 into the release branch Co-authored-by: Minhyuk Kim <[email protected]>
…" (erigontech#10904) This reverts PR erigontech#10499. See erigontech#10495 (comment) and PR erigontech#10901
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM, one comment worth addressing though
@@ -27,6 +27,8 @@ func MarshalReceipt( | |||
if t.Protected() { | |||
chainId = types.DeriveChainId(&t.V).ToBig() | |||
} | |||
case *types.DepositTx: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why was this case removed? I believe that GetChainID
will panic on deposit tx?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a new file. Yes, deposit tx doesn't have a chain Id.
No description provided.