Add Agra HF details for bor mainnet.

…ontech#8683)

we plan step-by-step keep increasing this default
still see users for who it helped to handle more rpc 
tradeoff: increasing of this flag - increasing "historical rpc"
throughput and decreasing "recent data rpc" throughput

…12.1 in /erigon-lib (erigontech#8680)

Bumps
[github.com/consensys/gnark-crypto](https://github.com/consensys/gnark-crypto)
from 0.12.0 to 0.12.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/consensys/gnark-crypto/releases">github.com/consensys/gnark-crypto's
releases</a>.</em></p>
<blockquote>
<h2>v0.12.1</h2>
<h2>What's Changed</h2>
<ul>
<li>fix: GLV exponentiation in GT with large exponents by <a
href="https://github.com/ThomasPiellard"><code>@​ThomasPiellard</code></a>
in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/451">Consensys/gnark-crypto#451</a>
<a
href="https://github.com/Consensys/gnark-crypto/security/advisories/GHSA-pffg-92cg-xf5c">Security
Advisory</a></li>
<li>feat: kzg extended transcript by <a
href="https://github.com/ThomasPiellard"><code>@​ThomasPiellard</code></a>
in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/452">Consensys/gnark-crypto#452</a></li>
<li>perf(fft): introduce cache efficient bit reverse shuffling by <a
href="https://github.com/gbotrel"><code>@​gbotrel</code></a> in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/446">Consensys/gnark-crypto#446</a></li>
<li>perf: Improve performance of point equality checks by <a
href="https://github.com/jsign"><code>@​jsign</code></a> in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/450">Consensys/gnark-crypto#450</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/Consensys/gnark-crypto/compare/v0.12.0...v0.12.1">https://github.com/Consensys/gnark-crypto/compare/v0.12.0...v0.12.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/da0317fd013308db6ce847bc9c3d506a2a3ae0ff"><code>da0317f</code></a>
clean: remove useless snippet in mulWindowed (<a
href="https://redirect.github.com/consensys/gnark-crypto/issues/453">#453</a>)</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/65cdb1d1c214e51454ae6dad4c466324f3ae510d"><code>65cdb1d</code></a>
Feat/kzg extended transcript (<a
href="https://redirect.github.com/consensys/gnark-crypto/issues/452">#452</a>)</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/95e674b3d69279dd0446e34cd8960e2fc4ed8393"><code>95e674b</code></a>
perf(fft): introduce cache efficient bit reverse shuffling (<a
href="https://redirect.github.com/consensys/gnark-crypto/issues/446">#446</a>)</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/ec6be1a037f7c496d595c541a8a8d31c47bcfa3d"><code>ec6be1a</code></a>
Merge pull request <a
href="https://redirect.github.com/consensys/gnark-crypto/issues/451">#451</a>
from Consensys/fix/glv</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/aef1eef48ab992db67631dcbec1c8dc259673e5b"><code>aef1eef</code></a>
fix: add GT ExpGLV fix to BLS24 + BW6</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/feaf161b4d1e53abf68b5679780ab497ef4818d4"><code>feaf161</code></a>
fix: merge and fix tests for expGLV</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/ced3076befee5c056f80ec5301438001ef5e9cdd"><code>ced3076</code></a>
fix: use max(size(s1), size(s2)) for ExpGLV</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/4096ad9913f1bfc9c67cdddade36ba981ade72ac"><code>4096ad9</code></a>
feat: discared useless files</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/ace5318b726c7185249bc9c92e1cadae701c1654"><code>ace5318</code></a>
Merge pull request <a
href="https://redirect.github.com/consensys/gnark-crypto/issues/450">#450</a>
from jsign/jsign-eq-improv</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/dc4e2d4f7799c4c36e213d1278fd3b77c07eb4da"><code>dc4e2d4</code></a>
feat: added test for [-s]G=-[s]G</li>
<li>Additional commits viewable in <a
href="https://github.com/consensys/gnark-crypto/compare/v0.12.0...v0.12.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/consensys/gnark-crypto&package-manager=go_modules&previous-version=0.12.0&new-version=0.12.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ledgerwatch/erigon/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

At the team meeting in Istanbul we decided to remove the auto-stale bot.

This PR adds beacon blocks snapshots for the following chains:

* Mainnet snapshots
* Sepolia snapshots

erigontech#7748

Paramatrizing a hardcoded parameter through CLI flags. 100k is the
default, as it was set previously.

Revert inadvertent change made by PR erigontech#8515

go: no module declaration in go.mod. To specify the module path:
	go mod edit -module=example.com/mod

…ch#8727)

Make it so that erigon can the enable beacon api.

Update the list with nodes collected by observer

```
EROR] [11-16|07:33:02.592] RPC method eth_getLogs crashed: runtime error: invalid memory address or nil pointer dereference
Nov 16 07:33:02 i-0e4d4ce2636f49d8a erigon.sh[1739584]: [service.go:219 panic.go:914 panic.go:261 signal_unix.go:861 block_reader.go:615 block_reader.go:405 eth_receipts.go:228 value.go:596 value.go:380 service.go:224 handler.go:493 handler.go:443 handler.go:391 handler.go:222 handler.go:315 asm_amd64.s:1650]
```

…ss tx (erigontech#8726)

Tests that don't affect each other should run in parallel

This is a similar PR of
ethereum/go-ethereum#28460, support traceCall in
the middle of a block

Signed-off-by: jsvisa <[email protected]>

# Background

Erigon currently uses a combination of Victoria Metrics and Prometheus
client for providing metrics.

We want to rationalize this and use only the Prometheus client library,
but we want to maintain the simplified Victoria Metrics methods for
constructing metrics.

This task is currently partly complete and needs to be finished to a
stage where we can remove the Victoria Metrics module from the Erigon
code base.

## Tests

### Functional
* Make sure that the format change int->float implied by VM to
Prometheus does not impact clients (pay particular attention to block
numbers)
* Check that the prometheus/grafana dashboards defined in cmd/prometheus
are functional after the change
(see docker-compose.yml for details and
https://github.com/ledgerwatch/erigon/tree/devel/cmd/prometheus#readme)
* Confirm that the underlying go metrics are still generated
* Confirm the following flags setting work:
    --metrics, --metrics.addr, --metrics.port with the new code
* Confirm that --metrics and --proff settings and handlers configuration
still allow metrics and pprof to share a port

#### Float counters - scientific notation test case
![Screenshot_2023-11-07_at_15 57
21](https://github.com/ledgerwatch/erigon/assets/94537774/32f0a6f6-968b-477c-8ec8-bb1812f3e848)

![Screenshot 2023-11-15 at 16 26
56](https://github.com/ledgerwatch/erigon/assets/94537774/3f402b2e-e343-4928-9fbb-18fa4d077485)


#### Float counters - NaN test case
![Screenshot_2023-11-07_at_16 04
25](https://github.com/ledgerwatch/erigon/assets/94537774/cbf90d5d-3749-4bd7-971d-e2124e54267c)

![Screenshot 2023-11-15 at 16 28
36](https://github.com/ledgerwatch/erigon/assets/94537774/5924915e-1977-4b7f-8082-23f73d0957d5)

### Performance
* Check the performance of counters created by RPC calls measurements
created by rpc/metrics.go are not impacted by the change.

#### RPC
Performed tests on rpcdaemon & erigon on localhost using
`etc_blockNumber`.
Did tests with 100, 1000, 10000 requests. Got a steady 15 ms response
time.

#### Memory
![Screenshot 2023-11-16 at 09 58
39](https://github.com/ledgerwatch/erigon/assets/94537774/5dd956d7-903f-4bea-a460-d3644da56201)

- changed communication tunnel to web socket in order to connect to
remote nodes
- changed diagnostics.url flag to diagnostics.addr as now user need to
enter only address and support command will connect to it through
websocket
- changed flag debug.urls to debug.addrs in order to have ability to
change connection type between erigon and support to websocket and don't
change user API
- added auto trying to connect to connect to ws if connection with was
failed

…erigontech#8747)

Upload beacon snapshots to R2 every week by default

fixed erigontech#8746

Problem:
Some goroutines are blocked on shutdown:
1. table close <-tab.closed // because table loop pending
1. table loop <-refreshDone // because lookup shutdown blocks doRefresh
1. lookup shutdown <-it.replyCh // because it.queryfunc (findnode -
ensureBond) is blocked, and not returning errClosed (if it returns and
pushes to it.replyCh, then shutdown() will unblock)
1. findnode - ensureBond <-rm.errc // because the related replyMatcher
was added after loop() exited, so there's nothing to push errClosed and
unlock it

If addReplyMatcher channel is buffered, it is possible that
UDPv4.pending() adds a new reply matcher after closeCtx.Done().
Such reply matcher's errc result channel will never be updated, because
the UDPv4.loop() has exited at this point. Subsequent discovery
operations will deadlock.

Solution:
Revert to an unbuffered channel.

…ontech#8757)

Making the addReplyMatcher channel unbuffered makes the loop
going too slow sometimes for serving parallel requests.
This is an alternative fix for keeping the channel buffered.

Added initial proof generation tests for polygon reverse flow for devnet

Blocks tested, receipts need trie proof clarification

Revisit PR erigontech#8750

Added trie findpath to support bor receipt format.

This requires testing against the polygon deployed contracts to confirm
parent path format

Added jsonification to beacon state

We store public keys contiguosly and batch update them on forkchoice

Review for following benches for the sake of clarity:

- debug_traceBlochByNumber
- debug_traceBlochByHsh
- debug_traceTransaction
- debug_traceCall 


Bench name `benchTraceBlockByHash` has been moved to
`benchDebugTraceBlockByHash`
Bench name `benchTraceTransaction` has been moved to
`benchDebugTraceTransaction` (to avoid confusion with future bench for
trace_transaction APIs)

erigontech#8762)

# Background

Erigon currently uses a combination of Victoria Metrics and Prometheus
client for providing metrics.

We want to rationalize this and use only the Prometheus client library,
but we want to maintain the simplified Victoria Metrics methods for
constructing metrics.

This task is currently partly complete and needs to be finished to a
stage where we can remove the Victoria Metrics module from the Erigon
code base.

# Summary of changes

- Adds missing `NewCounter`, `NewSummary`, `NewHistogram`,
`GetOrCreateHistogram` functions to `erigon-lib/metrics` similar to the
interface VictoriaMetrics lib provides
- Minor tidy up for consistency inside `erigon-lib/metrics/set.go`
around return types (panic vs err consistency for funcs inside the
file), error messages, comments
- Replace all remaining usages of `github.com/VictoriaMetrics/metrics`
with `github.com/ledgerwatch/erigon-lib/metrics` - seamless (only import
changes) since interfaces match

…ntech#8742)

* limit remote transactions re-broadcast to 3-6 peers  
  Broadcasting to 100 peers generates too much outgoing traffic.
* limit transactions count/size in PooledTransactions replies

Before it was sending 5.5-6.5 MiB/sec:

![Screenshot 2023-11-17 at 15 50
15](https://github.com/ledgerwatch/erigon/assets/11477595/bd2f51c5-190b-4f3e-aabf-4ff42ab8972d)


With the fixes it stays at 3-3.5 MiB/sec:

![Screenshot 2023-11-17 at 15 39
10](https://github.com/ledgerwatch/erigon/assets/11477595/74b18037-6017-49f1-8c00-9d7f3d1818b3)

P.S. A baseline if everything is disabled (BroadcastPooledTxs,
AnnouncePooledTxs, responses to GetPooledTransactions) is 0.5-1 MiB/sec.

Fixes
https://github.com/ledgerwatch/erigon/actions/runs/6930037092/job/18850944152

Now keep few beacon block headers in mdbx

log `ctx.Err()` - it can be canceled by many reasons: timeout, etc...

…tech#8812)

See Issue erigontech#8789

…eacon node (erigontech#8813)

* Correct naming of hash func in Eth2
* Customizable mode of operation for Caplin

Co-authored-by: cby3149 <[email protected]>

TestUDPv4_smallNetConvergence tests are often timeout on mac - disabling
this tests on mac CI

…gontech#8811)

…case) (erigontech#8801)

I using `https://heimdall-api-testnet.polygon.technology/` and seems
5sec timeout is not enough sometime - even that remote service working
well (node syncing well)

most of timeouts comes from same endpoint: 
```
 [bor.heimdall] request canceled          reason="context deadline exceeded" path=/milestone/lastNoAck attempt=2
```

# Background

Erigon currently uses a combination of Victoria Metrics and Prometheus
client for providing metrics.

We want to rationalize this and use only the Prometheus client library,
but we want to maintain the simplified Victoria Metrics methods for
constructing metrics.

This task is currently partly complete and needs to be finished to a
stage where we can remove the Victoria Metrics module from the Erigon
code base.

# Summary of changes

- Remove `UsePrometheusClient` boolean flag
- Remove `VictoriaMetrics` client lib and related code (simplifies
registry and prometheus http handler initialisation since now we have
only 1 registry and can use default `promhttp.Handler`)

…lock-STM (erigontech#8037)

This PR adds support to store the transaction dependency (generated by
the block producer) in the block header for bor. This transaction
dependency will then be used by the parallel processor
([Block-STM](erigontech#7812)).

I have created another
[PR](ledgerwatch/erigon-lib#1064) in the
erigon-lib repo which adds the `IsParallelUniverse()` function.

Current value: 16 was added by me 1 year ago and didn't mean anything.
Never seen this field holding much data, probably can increase.

Currently I see logs like (and 10x like this): 
[DBUG] [11-24|06:59:38.353] slow peer or too many requests, dropping its old requests name=erigon/v2.54.0-aeec5...

…gontech#8820)

because files with different name can have same hash: BitTorrent is
content-addressable.

…rigontech#8793)

…8808)

…eate new tables (erigontech#8821)

sometimes need tool to apply migrations - without starting erigon (if
erigon is buggy, or to prevent snapshots downloading, etc...)

Changed distribution of httpcfg.HttpCfg to be pointer.
Added new flags:
rpc.slow.log - which is false by default, this flag need to enable
logging slow RPC requests
rpc.slow.log.threshold - which is 100 by default, this flag specify slow
threshold in milliseconds
Updated rpc handler to log slow requests:
- added map[request id] {method, timestamp}
- put every request details to map above
- delete request details from map above
- added time interval check for elements in map and if time difference
is more than given threshold print request id and the method
- app will print slow requests in next cases:
1. As soon as request take more than given threshold
2. Every 20 seconds if request still in process
3. After request finished and it took more than give threshold

---------

Co-authored-by: alex.sharov <[email protected]>

- dont return nil for config :)

From comment
erigontech#8623 (comment)

it didn't cause problems yet. but it seems a good idea in-general.

Co-authored-by: Mark Holt <[email protected]>
Co-authored-by: alex.sharov <[email protected]>

This aligns the tracing behavior with Geth, Besu and Nimbus.

use manifest.txt instead of webseed.toml in public buckets

3TB is not enough for Polygon

see
erigontech/silkworm-go@c5729e3

eventsource is required for the validator api. this implements the
eventsource sink/server handler

the implementation is based off of this document:
https://html.spec.whatwg.org/multipage/server-sent-events.html

note that this is a building block for the full eventsource server.
there still needs to be work done

prysm has their own custom solution based off of protobuf/grpc:
https://hackmd.io/@prysmaticlabs/eventstream-api using that would be not
good

existing eventsource implementations for golang are not good for our
situation. options are:

1. https://github.com/r3labs/sse - has most stars - this is the best
contender, since it uses []byte and not string, but it allocates and
copies extra times in the server (because of use of fprintf) and makes
an incorrect assumption about Last-Event-ID needing to be a number (i
can't find this in the specification).
2. https://github.com/antage/eventsource -requires full buffers, copies
many times, does not provide abstraction for headers. relatively
unmaintained
3. https://github.com/donovanhide/eventsource - missing functionality
around sending ids, requires full buffers, etc
4. https://github.com/bernerdschaefer/eventsource - 10 years old,
unmaintained.

additionally, implemetations other than r3labs/sse are very incorrect
because they do not split up the data field correctly when newlines are
sent. (parsers by specification will fail to encode messages sent by
most of these implementations that have newlines, as i understand it).
the implementation by r3labs/sse is also incorrect because it does not
respect \r

finally, all these implementations have very heavy implementation of the
server, which we do not need since we will use fixed sequence ids.
r3labs/sse for instance hijacks the entire handler and ties that to the
server, losing a lot of flexiblity in how we implement our server
 
for the beacon api, we need to stream: 

```head, block, attestation, voluntary_exit, bls_to_execution_change, finalized_checkpoint, chain_reorg, contribution_and_proof, light_client_finality_update, light_client_optimistic_update, payload_attributes```
 
some of these are rather big json payloads, and the ability to simultaneously stream them from io.Readers instead of making a full copy of the payload every time we wish to rebroadcast it will save a lot of heap size for  both resource constrained environments and serving at scale.  

the protocol itself is relatively simple, there are just a few gotchas

Cherry pick ethereum/go-ethereum#27029

---------

Signed-off-by: jsvisa <[email protected]>
Co-authored-by: Delweng <[email protected]>

…erigontech#8878)

This PR has fixes for a number of instances in the bor heimdall stage
where nil headers are either ignored or inadvertently processed.

It also has a demotion of milestone related logging messages to debug
for missing blocks because the process is not at the head of the chain +
a general reduction in periodic logging to 30 secs rather than 20 to
reduce the log output on long runs.

In addition there is a refactor of persistValidatorSets to perform
validator set initiation in a seperate function. This is intended to
clarify the operation of persistValidatorSets - which is till performing
2 actions, persisting the snapshot and then using it to check the header
against synthesized validator set in the snapshot.

This PR is to add the request rate limiter. 

The solution is to count the request number for each peer for each
minute, if the peer exceeds the limit, block the requests for a
specified time.

Current limits:
- Request limited to `5000` requests per minute for each handler.
- Penalty blockage time `1-minute`

allows the disabling of file logging using the new flag
`--log.dir.disable`



NOTE: diagnostics tool logs WILL NOT function if this is set.

As per the
[comment](erigontech#8869 (comment))
in PR erigontech#8869, 8 TB is actually not enough for Polygon.

Erigon by default must use as minimum resources as possible to build
snapshots. But cli commands - can go opposite and maximize.