Co-authored-by: Alex Sharp <[email protected]>

…retire (erigontech#9068)

Co-authored-by: Alex Sharp <[email protected]>

)

* Testing Beacon API
* Fixed sentinel code (a little bit)
* Fixed sentinel tests
* Added historical state support
* Fixed state-related endpoints (i was drunk when writing them)

…#9077)

Co-authored-by: Alex Sharp <[email protected]>

* Added /validators retrieval to beacon api
* fixed race in sentinel
* fixed possible UB in gossip management

This adds a simulator object with implements the SentryServer api but
takes objects from a pre-existing snapshot file.

If the snapshot is not available locally it will download and index the
.seg file for the header range being asked for.

It is created as follows: 

```go
sim, err := simulator.NewSentry(ctx, "mumbai", dataDir, 1, logger)
```

Where the arguments are:

* ctx - a callable context where cancel will close the simulator torrent
and file connections (it also has a Close method)
* chain - the name of the chain to take the snapshots from
* datadir - a directory potentially containing snapshot .seg files. If
not files exist in this directory they will be downloaded
 *  num peers - the number of peers the simulator should create
 *  logger - the loger to log actions to

It can be attached to a client as follows:

```go
simClient := direct.NewSentryClientDirect(66, sim)
```

At the moment only very basic functionality is implemented:

* get headers will return headers by range or hash (hash assumes a
pre-downloaded .seg as it needs an index
* the header replay semantics need to be confirmed
* eth 65 and 66(+) messaging is supported
* For details see: `simulator_test.go

More advanced peer behavior (e.g. header rewriting) can be added
Bodies/Transactions handling can be added

This change introduces additional processes to manage snapshot uploading
for E2 snapshots:

## erigon snapshots upload

The `snapshots uploader` command starts a version of erigon customized
for uploading snapshot files to
a remote location.  

It breaks the stage execution process after the senders stage and then
uses the snapshot stage to send
uploaded headers, bodies and (in the case of polygon) bor spans and
events to snapshot files. Because
this process avoids execution in run signifigantly faster than a
standard erigon configuration.

The uploader uses rclone to send seedable (100K or 500K blocks) to a
remote storage location specified
in the rclone config file.

The **uploader** is configured to minimize disk usage by doing the
following:

* It removes snapshots once they are loaded
* It aggressively prunes the database once entities are transferred to
snapshots

in addition to this it has the following performance related features:

* maximizes the workers allocated to snapshot processing to improve
throughput
* Can be started from scratch by downloading the latest snapshots from
the remote location to seed processing

## snapshots command

Is a stand alone command for managing remote snapshots it has the
following sub commands

* **cmp** - compare snapshots
* **copy** - copy snapshots
* **verify** - verify snapshots
* **manifest** - manage the manifest file in the root of remote snapshot
locations
* **torrent** - manage snapshot torrent files

implement the `txpool_contentFrom` rpc, used to retrieve the specified
address's tx contents.

---------

Signed-off-by: jsvisa <[email protected]>

Co-authored-by: Alex Sharp <[email protected]>

…ation (erigontech#9089)

Adds unit tests for:
- Bor Heimdall Stage - `checkHeaderExtraData`
- at end of each sprint verifies that the validators in the header extra
data matches the selected proposers from the heimdall span
   - 1 test for selected proposers length mismatch
   - 1 test for selected proposers bytes mismatch
- BorHeimdall Stage - `persistValidatorSets`
- verifies that each header is created by a validator in the validator
set
   - in such situation we set the unwind point

…erigontech#9096)

Heimdall prepares the next span a number of sprints before the current
span ends. Currently we always fetch the next span regardless of which
sprint we are in during the current span. This causes a liveness issue
due to how the Heimdall client works (it infinitely retries until it
fetches a span - this issue will be fixed in a separate PR). This PR
fixes this by matching what bor does - it fetches the next span only in
the last sprint of the current span.

Changes:

- Adds a unit test for the above
- Adds a new function BlockInLastSprintOfSpan
- Some code reorg and cleanup - moves the span num related functions
from the bor package to the span sub package for better logical grouping

follow up on naming as suggested here
erigontech#9096 (review)

Corresponds to the client fix in this PR description -
erigontech#9096 (comment)

…se (erigontech#9103)

Users reported this error
```
[bor.heimdall] an error while trying fetching path=clerk/event-record/list attempt=5 error="unexpected end of JSON input"
```

Which may happen if:

1. Heimdall is behind and not sync-ed - for more info check
maticnetwork/heimdall#993
2. Or the header time erigon is sending is far into the future

The logs in this PR will help us see which of the 2 is the culprit but
most likely it is 1. We will investigate further 2. if it ever happens.

Changes:
1. Improves logging upon heimdall client retries - prints out the full
url that failed.
2. Fixes a bug where the body was incorrectly checked if it is empty -
`len(body) == 0` vs `body == nil`
3. Unit test for the bug regression
4. Adds a log to indicate to users to check their heimdall process if
they run into this scenario since that may be the culprit


Example output with new logs
<img width="1465" alt="Screenshot 2023-12-29 at 20 16 57"
src="https://github.com/ledgerwatch/erigon/assets/94537774/1ebfde68-aa93-41d6-889a-27bef5414f25">

…eacon/rewards/sync_committee/{block_id} (erigontech#9102)

* Changed slightly archive format (again)
* Added all of the remaining rewards endpoints

…h#9099)

Mdbx now takes a logger - but this has not been pushed to all callers -
meaning it had an invalid logger

This fixes the log propagation.

It also fixed a start-up issue for http.enabled and txpool.disable
created by a previous merge

* Fixed mispelling in json fields
* Added CORS

fix the relative path in cmd/evm/README.md


![image](https://github.com/ledgerwatch/erigon/assets/24953789/058b0486-f8cf-4fa3-ad53-e01b5ea6f0c3)

Hi, I made three suggestions for this section:

- "devenet" should be "devnet" (typo).

- "are currently build" should be "are currently built" (grammatical
error).

- "sptep" should be "step" (typo).

Thanks.

See ethereum/EIPs#8051 &
ethereum/execution-specs#860

Getting an error in one of the bor nodes in devnet when trying to run
the "state-sync" scenario:
```
[EROR] [01-03|16:55:44.179] cli.StartRpcServer error                 err="could not start separate Websocket RPC api at port 8546: listen tcp 127.0.0.1:8546: bind: address already in use"
```

This happens for scenarios with more than 1 node.

Digging further this regressions has happened due to this change:
erigontech#8909

This PR fixes this by updating the devnet `NodeArgs` struct to set the
corresponding `--ws.port` `arg` tag which now exists.

…9121)

A crash on startup happens on --chain=mumbai , because I've confused
chainConfig.Bor (from type chain.Config) and config.Bor (from type
ethconfig.Config) in the setup code.

ethconfig.Config.Bor property contained bogus values, and was used only
to check its type in CreateConsensusEngine(). Its value was never read
(before PR erigontech#9117).

This change removes the property to avoid confusion and fix the crash.

Devnet network.BorStateSyncDelay was implemented using
ethconfig.Config.Bor, but it wasn't taking any effect. It should be
fixed separately in a different way.

…ontech#9146)

While working on fixing the bor mining loop I stumbled across an error
in `ChainReader.BorSpan` - not implemented panic. Also hit a few other
panics due to missed logger in `ChainReaderImpl` struct initialisations.
This PR fixes both.

This moved the log level of retire progress messaging to debug, to avoid
log nose on qa and test runs

this PR adds amoy bootnodes to the configuration

…ontech#9156)

refactored data structure for sync statistics

…0 - fix (erigontech#9163)

* Added handling of AggregateAndProof

…ech#9168)

erigontech#9130

erigontech#9128

erigontech#9164)

…testing on Sepolia) (erigontech#9151)

---------

Co-authored-by: Alex Sharp <[email protected]>

…#9177)

macos-13-xlarge was needed to use M1 CPU,
because previously silkworm-go didn't have macOS Intel support

Currently the mining loop is broken for the polygon chain. This PR fixes
this.

High level changes:

- Introduces new Bor<->Heimdall stage specifically for the needs of the
mining flow
- Extracts out common logic from Bor<->Heimdall sync and mining stages
into shared functions
- Removes `mine` flag for the Bor<->Heimdall sync stage
- Extends the current `StartMining` function to prefetch span zero if
needed before the mining loop is started
- Fixes Bor to read span zero (instead of span 1) from heimdall when the
span is not initially set in the local smart contract that the Spanner
uses

Test with devnet "state-sync" scenario:
![Screenshot 2024-01-05 at 17 41
23](https://github.com/ledgerwatch/erigon/assets/94537774/34ca903a-69b8-416a-900f-a32f2d4417fa)

integration test TestMiningBenchmark is failing the CI on devel - this
PR aims to fix it

…act integration tests (erigontech#9183)

This PR fixes an infinite recursion (stack overflow) error in devnet
integration tests that surfaced in our integration CI a week or more ago
and additionally enables some integration tests that are now fixed -
`TestStateSync` & `TestCallContract`.

![Screenshot 2024-01-09 at 17 10
24](https://github.com/ledgerwatch/erigon/assets/94537774/a5a8c9c9-9f68-4084-9e08-1bf3c1601cab)

logic: 1/42 of ram, but not more than 2Gb for chandata and not more than
256mb for other databases.

---------

Co-authored-by: battlmonstr <[email protected]>

…rigontech#9190)

In PR erigontech#9184 I confused slot with epoch. Refer to
gnosischain/configs#26.

Added publishing of aggregates and proofs

This PR contains 3 fixes for interaction between the Bor mining loop and
the TX pool which where causing the regular creation of blocks with zero
transactions.

* Mining/Tx pool block synchronization
The synchronization of the tx pool between the sync loop and the mining
loop has been changed so that both are triggered by the same event and
synchronized via a sync.Cond rather than a polling loop with a hard
coded loop limit. This means that mining now waits for the pool to be
updated from the previous block before it starts the mining process.
* Txpool Startup consolidated into its MainLoop
Previously the tx pool start process was dynamically triggered at
various points in the code. This has all now been moved to the start of
the main loop. This is necessary to avoid a timing hole which can leave
the mining loop hanging waiting for a previously block broadcast which
it missed due to its delay start.
* Mining listens for block broadcast to avoid duplicate mining
operations
The mining loop for bor has a recommit timer in case blocks re not
produced on time. However in the case of sprint transitions where the
seal publication is delayed this can lead to duplicate block production.
This is suppressed by introducing a `waiting` state which is exited upon
the block being broadcast from the sealing operation.

Integration tests CI is failing due to a port clash in devnet tests. I
believe this is because there are 2 packages of devnet integration tests
and go can run tests from separate packages in parallel (by default it
does package level parallelism). A simple fix would be to just have all
devnet integration tests in 1 package and run all these tests
sequentially within the package (ie not use t.Parallel). This PR moves
all devnet integration tests in 1 package.

`"ContextStart devnet start failed: private api: could not create
listener: listen top 127.0.0.1:10090: bind: address already in use,
addr=localhost:10090"`

![Screenshot 2024-01-10 at 13 38
37](https://github.com/ledgerwatch/erigon/assets/94537774/06bda987-45e5-46ef-9e0b-3876b3f85c01)

This fixes the snapshot uploader start as flags have been moved to
default

…er env.Open() (erigontech#9189)

…#9198)

This PR
- increase the test time
- use the updated python scripts that reports as an issue a long exit
time
- change the test name to states that it is runned in the snapshot
downloading phase
(a later PR will add a similar test on the block downloading phase)

When discarding spamming we need to remove the tx from the subpools as
well as the sender tx list. Otherwise the tx is ignored by other
operations and left in the subpool

As well as the fix here this PR also contains several changes to TX
TRACING and other logging to make it easier to see what is going on with
pool processing

…rigontech#9196)

Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go)
from 0.38.1 to 0.38.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/quic-go/quic-go/releases">github.com/quic-go/quic-go's
releases</a>.</em></p>
<blockquote>
<h2>v0.38.2</h2>
<p>This release contains fixes for a resource exhaustion attack on
QUIC's path validation logic (CVE-2023-49295), see <a
href="https://seemann.io/posts/2023-12-18-exploiting-quics-path-validation">https://seemann.io/posts/2023-12-18-exploiting-quics-path-validation</a>
for details:</p>
<ul>
<li>limit the number of queued PATH_RESPONSE frames to 256 (<a
href="https://redirect.github.com/quic-go/quic-go/pull/4199">quic-go/quic-go#4199</a>)</li>
<li>don't retransmit PATH_CHALLENGE and PATH_RESPONSE frames (<a
href="https://redirect.github.com/quic-go/quic-go/pull/4200">quic-go/quic-go#4200</a>)</li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/quic-go/quic-go/compare/v0.38.1...v0.38.2">https://github.com/quic-go/quic-go/compare/v0.38.1...v0.38.2</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/quic-go/quic-go/commit/9aaefe19fc3dc8c8917cc87e6128bb56d9e9e6cc"><code>9aaefe1</code></a>
don't retransmit PATH_CHALLENGE and PATH_RESPONSE frames (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4200">#4200</a>)</li>
<li><a
href="https://github.com/quic-go/quic-go/commit/17fc98c2d81dbe685c19702dc694a9d606ac56dc"><code>17fc98c</code></a>
limit the number of queued PATH_RESPONSE frames to 256 (<a
href="https://redirect.github.com/quic-go/quic-go/issues/4199">#4199</a>)</li>
<li>See full diff in <a
href="https://github.com/quic-go/quic-go/compare/v0.38.1...v0.38.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/quic-go/quic-go&package-manager=go_modules&previous-version=0.38.1&new-version=0.38.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ledgerwatch/erigon/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ontech#9119) (erigontech#9195)

The handler had race conditions in the candidates processing goroutine.

…on resumable. (erigontech#9211)

Co-authored-by: Bayram Guvanjov <[email protected]>

This fixes a couple of regressions for running the uploader for mumbai

* Now flags have moved to a higher context they need to be set in the
context not the flag values
* Span 0 of mumbai has a header/span mismatch for span zero sprint 0. So
the check here needs to be suppressed

reasons: 
- indexing done in background (or in many workers)
- `recsplit` has 2 etl collectors

can do it, because we don't do read-after-write in same rwtx

…Cut (erigontech#9202)

above go1.18  Index calls are more clearly written using Cut.
golang/go#46336

Pr is ready to review and merge. 

This PR includes implementing and integrating Ethereum Deneb's hard work
with the Caplin Ethereum client.

Changes:

- Full compatibility with Deneb Ethereum hard fork
- Added new EIPs introduced in Deneb. (`EIP-4788`, `EIP-4844`,
`EIP-7044`, `EIP-7045`, `EIP-7514`)
- Tests integration

---------

Co-authored-by: Giulio <[email protected]>

…9021)

Removed the check for milestoneID in the GetVoteOnHash()

maticnetwork/bor#1109

Pos downloader and the engine api code are not necessary for Bor. This
adds code to suppress their start up.

Hi, caught a few mistakes while reading the docs. Hope I can be of any
help.
Have a nice day.

…ontech#9219)

When the sync loop first runs it suppresses block sync events both in
the initial loop and when the blocks being processed are greater than
1000.

This fix removed the first check, because otherwise the first block
received by the process ends up not getting sent to the tx pool. Which
means it won't produce new block for polygon.

As well as this fix - I have also moved the gas initialization to the
txpool start method rather than prompting it with a 'synthetic block
event'

As the txpool start has access to the core & tx DB's it can find the
current block and chain config internally so that it doesn't need to be
externally activated it can just do this itself on start up. This has
the advantage of making the txpool more self contained.