This task is mostly implemented to be used in
`erigon/erigon-lib/downloader/mdbx_piece_completion.go` and maybe in
`nodesDB` (where we need many parallel RwTx)

I was agains adding this "trick"/"api" last years, because thought that
we can implement our App to be more 1-big-rwtx-friendly. And we did it
in Erigon - StagedSync. TxPool also did, but with a bit less happy face
- by "map+mutex with periodic flush to db". But `anacrolix/torrent` is
external library and unlikely will survive such big mind-model-change.
Maybe it's time to add `db.Batch()`.

#### Batch Rw transactions

Each `DB.Update()` waits for disk to commit the writes. This overhead
can be minimized by combining multiple updates with the `DB.Batch()`
function:

```go
err := db.Batch(func(tx *bolt.Tx) error {
	...
	return nil
})
```

Concurrent Batch calls are opportunistically combined into larger
transactions. Batch is only useful when there are multiple goroutines
calling it.

The trade-off is that `Batch` can call the given
function multiple times, if parts of the transaction fail. The
function must be idempotent and side effects must take effect only
after a successful return from `DB.Batch()`.

For example: don't display messages from inside the function, instead
set variables in the enclosing scope:

```go
var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
	// Find last key in bucket, decode as bigendian uint64, increment
	// by one, encode back to []byte, and add new key.
	...
	id = newValue
	return nil
})
if err != nil {
	return ...
}
fmt.Println("Allocated ID %d", id)
```


---- 

Implementation mostly taken from
https://github.com/etcd-io/bbolt/?tab=readme-ov-file#batch-read-write-transactions

Maybe in future can push-down it to
https://github.com/erigontech/mdbx-go

- PutUvarint can produce 10 bytes
- re-using buffer - faster and less gc

…or: See comment (erigontech#9972)

* Fixed and simplified unaggregated bits check.
* There are 2 bits on, one for the attester and one for the
End-of-bitlist, needed to account for end of bitlist bit
 * Wrong publishing topic for sync_committee_ messages
* Added more Ignore by receiving specific errors to avoid forwarding
useless data.
 * Replaced `validateAttestation` with full message processing
 * Fixed forwarding of sync committee aggregates
 * Fixed subnet announcements

---------

Co-authored-by: kewei <[email protected]>

…n't check .lock. Otherwise can't create .torrent for existing .seg files. (erigontech#10004)

…ch#10014)

This is for E2.

It implements the backward compatible output field for traces on
ots_traceTransaction: otterscan/execution-apis#1

It'll be consumed by Otterscan in an upcoming release of this feature:
otterscan/otterscan#1530

there was error:
```
prog.go:12:2: missing go.sum entry for module providing package github.com/golang/mock/mockgen/model; to add:
	go mod download github.com/golang/mock
```

To Follow suit with rest of the naming

- voluntary_exit
- bls_to_execution_change
- proposer_slashing
- expirable lru

---------

Co-authored-by: Giulio <[email protected]>

Added command which prints to console diagnostics data. In this initial
version it is possible to print stages list and snapshot download
progress. Erigon should be running with --metrics flag
There are two available commands:
- "downloader"
- "stages" "current"
There are two possible options for output: text and json
Run command - ./build/bin/diag [command] [text | json]

---------

Co-authored-by: Mark Holt <[email protected]>

Co-authored-by: awskii <[email protected]>

Add missing body close method when webseed roundtrip is retried

Set baseFeePerGas value in graphql resolver for block

The `StateTransition` property `gas` actually tracks the remaining gas
in the current context. This PR is to improve code readability.
Geth also uses similar naming.

Signed-off-by: fuyangpengqi <[email protected]>

- attempt to integrate sonar with test coverage by following 
-
https://sonarcloud.io/project/configuration/GitHubActions?id=ledgerwatch_erigon
-
https://docs.sonarsource.com/sonarcloud/advanced-setup/ci-based-analysis/github-actions-for-sonarcloud/
- adds sonar properties file to specify code coverage output
- also properties file can be used to filter out generated code from
sonar scan
    - protobuf
    - graphql
    - ignore pedersen hash bindings code
- ... there will be more ignores coming in later PRs (e.g. some c/c++
code we dont need to scan, some js code, some contract gen code, etc.)

Fixes error in Sonar GitHub action:
<img width="1645" alt="Screenshot 2024-04-23 at 17 46 01"
src="https://github.com/ledgerwatch/erigon/assets/94537774/3833db1c-6a8a-4db2-8bb7-5de58b57e638">

…tech#10009)

This PR allows the computation for the computation of the
`SyncAggregate` field in block production:
https://sepolia.beaconcha.in/slot/4832922 proof of the code working is
that now Caplin validators can include sync aggregates in their blocks.

Things modified:
* We do not aggregate pre-aggregated `SyncContributionAndProof`s,
instead we just listen to the network and pick the most profitable ones
for each sub sync committee (4 sync subcommittee on mainnet).
profitability == most bits set in `AggregationBits` field.
* Separate aggregates set for contribution to be included in a block
from the ones constructed from `SyncCommitteeMessage`s, combining the
two causes some contributions to be marked as invalid and not
aggregable.
* Remove SyncContributionMock in favor of gomock

…ontech#10032)

Observed the following issue in a long running Astrid sync on
bor-mainnet:
```
[DBUG] [04-17|14:25:43.504] [p2p.peerEventObserver] received new peer event id=Disconnect peerId=51935aa1eeabdb73b70d36c7d5953a3bfdf5c84e88241c44a7d16d508b281d397bdd8504c934bfb45af146b86eb5899ccea85e590774f9823d056a424080b763
[DBUG] [04-17|14:25:43.504] [p2p.peerEventObserver] received new peer event id=Connect peerId=51935aa1eeabdb73b70d36c7d5953a3bfdf5c84e88241c44a7d16d508b281d397bdd8504c934bfb45af146b86eb5899ccea85e590774f9823d056a424080b763
```

Note the timestamps are the same on the millisecond level, however the
disconnect was processed before the connect which is wrong (connect
should always be first).

This then got the `PeerTracker` in a bad state - it kept on returning
peer
`51935aa1eeabdb73b70d36c7d5953a3bfdf5c84e88241c44a7d16d508b281d397bdd8504c934bfb45af146b86eb5899ccea85e590774f9823d056a424080b763`
as a valid peer to download from, which caused repeated `peer not found`
errors when sending messages to it.

Fix is to have the message listener wait for all observers to finish
processing peer event 1 before proceeding to notifying them about peer
event 2.

Fixes Sonar warnings:
<img width="550" alt="Screenshot 2024-04-23 at 19 37 53"
src="https://github.com/ledgerwatch/erigon/assets/94537774/b85c9607-3800-408d-8a1b-c5bf80da38b2">

- Excludes go mock generated files from analysis
- Excludes broken js files (valid as they are used for tracers and test
data) to fix below warnings
<img width="1658" alt="Screenshot 2024-04-24 at 11 12 04"
src="https://github.com/ledgerwatch/erigon/assets/94537774/7925d07f-37f3-43c9-b34a-9a5361e48a8a">

Accept a slice of block numbers that represents the final block number
that will be available to the client of the simulator.Any data after the
iteration stage end is not accessible to the client.

The iteration moves to the next stage under certain conditions:
- requesting the latest span via `FetchSpan`
- requesting state sync events beyond current last iteration block's
timestamp

This fixes this issue:

erigontech#9499

which is caused by restarting erigon during the bor-heimdall stage.

Previously after the initial call to bor-heimdall (header 0), forward
downloading was disabled, but backward
downloading recursively collects headers - holding results in memory
until it can roll them forward. This should
only be called for a limited number of headers, otherwise it leads to a
large amount of memory >45GB for bor
main net if the process is stopped at block 1.

The downloader is not complete until all of its requested files have
been downloaded.

This changes adds a request count to the downloader stats to be checked
for completeness, otherwise the downloader
may appear complete before all required torrents have been added.

We reverted support of this flag in `updateForkChoice` because
implementation was too complex and fragile:
erigontech#9900

But it's good-enough if StageSenders will preserve this flag - then next
stages (exec) will also follow (because they look at prev stage
progress).

It's good-enough - because users just want to save some partial progress
after restoring node from backup (long downtime). And enforce "all
stages progress together" invariant

**Summary**
Fixes prune point for log (+index)
- Unnecessary to use ETL again for deleting `kv.Log` entries, can just
introduce `RwCursor` in the initial loop
- Put the last `pruneTo` block number in the `PruneState` - this will
begin pruning from that point. Earlier the `pruneFrom` point being
passed in was buggy as it used some other assumption for this value

…10038)

This is for E2.

The block rewards returned by Otterscan API is incorrect since the
merge. It replaces very old code with the same calculation used for
trace_block.

this code probably won't work with Aura consensus, but that's ok since
the current one doesn't work as well. It would actually require exposing
more code from block execution and I don't want to handle it for now,
let's fix only the post-merge calc for now.

Co-authored-by: sealer3 <[email protected]>

This PR introduces support for customising Silkworm RpcDaemon settings
in Erigon++.

Common RPC settings between Erigon and Silkworm are simply translated
from the existing Erigon command-line options. They include:
- `--http.addr`
- `--http.port`
- `--http.compression`
- `--http.corsdomain`
- `--http.api`
- `--ws`
- `--ws.compression`

Moreover, the following Silkworm-specific command-line options are
added:
- `--silkworm.verbosity`
- `--silkworm.contexts`
- `--silkworm.rpc.log`
- `--silkworm.rpc.log.maxsize`
- `--silkworm.rpc.log.maxfiles`
- `--silkworm.rpc.log.response`
- `--silkworm.rpc.workers`
- `--silkworm.rpc.compatibility`

Default values cover the common usages of Erigon++ experimental
features, yet such options can be useful for testing some corner cases
or collecting information.

Finally, this PR adds a new `LogDirPath` function to `logging` module
just to determine the log dir path used by Erigon and put there also
Silkworm RPC interface logs, when enabled.

* Decrease memory footprint on chain tip
* Fix a race
* Better times on `Attestation` processing. 1 sec -> 54 ms

…" (erigontech#10055)

This reverts commit f4aefdc.

See PR erigontech#9738

Fix some comments

Reverts erigontech#10036

drop go 1.20 support 

use ` github.com/erigontech/torrent v1.54.2-alpha` - to simplify future
support and features backport

…10060)

Reverts erigontech#9982

After switching to more durable db mode
erigontech#10010 - we don't need manual
fsync anymore.

…h#10065)

This PR fixes 2 things:
* Superset handling (should ignore)
* SSZ offset not set for custom ssz in attestation encoding after json
unmarshalling

feat: add `fullTx` params to `NewPendingTransactions`

Closes erigontech#9203

In PR: 
- new .lock format introduced by
erigontech#9766 is not backward
compatible. In the past “empty .lock” did mean “all prohibited” and it
was changed to “all allowed”.
- commit

Not in PR: I have idea to make .lock also forward compatible - by making
it whitelist instead of blacklist: after adding new snap type it will
not be downloaded by accident. Will do it in next PR.

But I need one more confirmation - why do we need exceptions from .lock?
Why we breaking "download once" invariant for some type of files? Can we
avoid it?

This PR makes the channel that is used to send logs to subscriptions
configurable so logs are not dropped when the channel gets filled. See
issue 9699.
This is just an initial version since I wanted to gather some feedback
and was unsure if this is the correct approach to solve this.

…ntech#10066)

…erigontech#10070)

For E2: fix incorrect type + overflow in certain blocks

Corresponding otterscan issue:
otterscan/otterscan#1658

…gontech#10039)

- Added method `tx.Context()` - because Tx already bounded to context by
`db.BeginRo(ctx)`
- Removed ctx parameter from `BlockWithSenders` method in interfaces
- Added `dbg.ToContext()` and `dbg.Enabled(ctx)` methods to set/get
debugging tag to `ctx`.

Added way to debug single http request: 
To print more detailed logs for 1 request - add `--http.dbg.single=true`
flag. Then can send HTTP header `"dbg: true"`:

```
curl -X POST -H "dbg: true" -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "method": "eth_blockNumber", "params": [], "id":1}' localhost:8545
```

---------

Co-authored-by: battlmonstr <[email protected]>

In the current go 1.21 version used in the project, slices are no longer
an experimental feature and have entered the standard library

Co-authored-by: alex.sharov <[email protected]>

As the other fields are json marshaled into lowerUpper case, we should
use the same style.

---------

Signed-off-by: jsvisa <[email protected]>

TL;DR: on a reorg, the common ancestor block is not being published to
subscribers of newHeads

#### Expected behavior

if the reorg's common ancestor is 2, I expect 2 to be republished

1, 2, **2**, **3**, **4**

#### Actual behavior

2 is not republished, and 3's parentHash points to a 2 header that was
never received

1, 2, **3**, **4**

This PR is the same thing as
erigontech#9738 except with a test.

Note... the test passes, but **this does not actually work in
production** (for Ethereum mainnet with prysm as external CL).

Why? Because in production, `h.sync.PrevUnwindPoint()` is always nil:
https://github.com/ledgerwatch/erigon/blob/a5270bccf5e69a6beaaab9a0663bdad80e989505/turbo/stages/stageloop.go#L291
which means the initial "if block" is never entered, and thus we have
**no control** of increment/decrement `notifyFrom` during reorgs
https://github.com/ledgerwatch/erigon/blob/a5270bccf5e69a6beaaab9a0663bdad80e989505/eth/stagedsync/stage_finish.go#L137-L146

I don't know why `h.sync.PrevUnwindPoint()` is seemingly always nil, or
how the test can pass if it fails in prod. I'm hoping to pass the baton
to someone who might. Thank you @indanielo for original fix.

If we can figure this bug out, it closes erigontech#8848 and closes erigontech#9568 and
closes erigontech#10056

---------

Co-authored-by: Daniel Gimenez <[email protected]>

use https://github.com/Abirdcfly/dupword to check repetitive words

Reverts erigontech#10072

…ch#10079)

erigontech#10074)

Pros:
- it allows to not pre-alloc files:
erigontech#8688
- it allows to not "sig-bus" when no space left on disk (return
user-friendly error). see:
erigontech#8500 - but DB will be MMAP
anyway and may get "sig-bus"

FYI:
- seems no perf difference (but i tested only on cloud drives)
- erigon will anyway open it as mmap 
 
Cons: 
- i did implemented `fsync` for mmap (
anacrolix/torrent#755 ) - probably will need
implement it for bufio: anacrolix/torrent#937
- no zero-copy: more `alloc` memory will be holded by APP (PageCache
starvation). I see 2x mem usage (at `--torrent.download.slots=500` 20gb
vs 40gb)
- i see "10K threads exchaused" error earlier (on
`--torrent.download.slots=500`).
- what else?

fix issue erigontech#10089

For period where there are not many sync events (mostly testnets) sync
event fecthing can be slow becuase sync events are fetched at the end of
every sprint.

Fetching the next and looking at its block number optimizes this because
fetches can be skipped until the next known block with sync events.

Cherry pick ethereum/go-ethereum#29441

---------

Co-authored-by: Marius van der Wijden <[email protected]>
Co-authored-by: Martin Holst Swende <[email protected]>

fixes erigontech#7593 

it introduced a regression: `"fmt"` and `"reflect"` imports were added
for all files generated by `abigen` assuming that they will be used in
all cases, however that assumption was wrong for some cases resulting in
invalid code being generated (in this case after running `make
devtools`):
<img width="982" alt="Screenshot 2024-04-27 at 10 50 37"
src="https://github.com/ledgerwatch/erigon/assets/94537774/9a1b93a5-2141-40d9-8c9e-01a1ff6c031c">

…Slashing`s, `BlsExecutionToChange`s and `Attestation`s into block production (erigontech#10071)

This PR add operations inclusion.
## Normal operations

* BlsExecutionChange
* VoluntaryExit
* Slashings

Each of these operations blacklist the index they work on so we do not
have repeating indices for the same operations twice. we assume all
signatures are pre-validated and just see if it is a good time to
produce a block with them (by looking at their slot)
## Aggregated Attestations

There is a lot of trash attestations on the network so we separate our
algorithm in 3 steps:

### Eligibility

We iterate over the entire pool of accumulated attestations and filter
out all attestations who cannot be included at the current slot, and
compute their expected reward. (filter out if 0).

### Ranking

We rank the `Attestation`s by their expected reward (we just sort the
array of candidates) by expected reward in ascending order.

### Filtering by superset

We may have some supersets left-over, filter attestation which ends up
being supersets of other. this process is done from highest reward down
to lowest reward.

`HasNext` will return true even with existing error and the application
will expect a next entry. The `Next` function can get into an internal
error (such as a `panic()`) while fetching the next cursor item and thus
fail to return the error.

---------

Co-authored-by: alex.sharov <[email protected]>

- replaces usages of `moq` in `erigon-lib` with `mockgen` (gomock)
- adds a `make mocks` and `mocks-clean` command for `erigon`
- updates existing `make mocks` command and adds a `mocks-clean` common
for `erigon-lib`

Use `mockgen -typed=true` to generate mocks with type-safe `Return`,
`Do`, `DoAndReturn` function -
https://github.com/uber-go/mock?tab=readme-ov-file#flags

adds:
- `make gen`
- `make solc`
- `make abigen`
- `make codecgen`
- `make gencodec`
- `make graphql`

tidies up `make devtools`

Added command to print databases tables basic info. There are two
options :
- print all info: ./build/bin/diag dbs all
- print only populated tables and dbs: ./build/bin/diag dbs pop   

Here is example output:
![Screenshot 2024-04-28 at 21 38
18](https://github.com/ledgerwatch/erigon/assets/29065143/f0a04931-8d87-4c45-b71a-71d75404f3fc)


@taratorio if you want I can add flag which will print specific DB.

…0109)

Tweaks I did:
1) Decreased attestation expiry down to 30 minutes
2) Removed slot check in committeeSubAggregation
3) More reliable algorithm for the dependent root

Results:
* Better aggregates
* Less strain on the node
* No blocks/attestations missed

Problem: if --pageSize parameter not set - we using `default pagesize`
instead of `real pagesize of db`. And it causing different `dirtySpace`
size (because it's accounted in "pages")

for erigontech#10099
for things like `eth_getTransactionReceipt`,
`ots_searchTransactionsAfter`, etc...

Also moved:
- moved `api.chainConfig()` inside `api.getReceipts()`
- switched `ots` to use blocks/receipts lru
- switched price oracle to use blocks/receipts

- use sonar badge for code coverage
- remove unnecessary "Coverage" GitHub action and unnecessary duplicate
test run on "devel" CI for it
- the existing coverage job + badge didn't seem to be accurate (wasn't
taking into account `erigon-lib` sub-module)

<img width="982" alt="Screenshot 2024-04-29 at 12 06 46"
src="https://github.com/ledgerwatch/erigon/assets/94537774/e47367ed-340d-42b5-ad00-2f59edce100c">

Implemented limit for saving peers in an Erigon node memory to be able
to turn on diagnostics data collection by default.

Signed-off-by: luchenhan <[email protected]>

…y Giulio (erigontech#10077)

Reverts erigontech#10006 and add a proper migration routine

Enabled diagnostics by default to collect data. It will allow to connect
to node and get stored data. It includes three new flags:
- "diagnostics.disabled" - it's set to "false" by default. Set to "true"
if you want to disable diagnostics.
- "diagnostics.endpoint.addr" - address of HTTP endpoint to get
diagnostics data
- "diagnostics.endpoint.port" - port of HTTP endpoint to get diagnostics
data

[DO NOT MERGE] as it depend on: 
- erigontech#10069
- update support command
- update diagnostics UI

Reverts erigontech#10113

Implementation of db and snapshot storage for additional synced hiemdall
waypoint types

* Checkpoint
* Milestones

This is targeted at the Astrid downloader which uses waypoints to verify
headers during syncing and fork choice selection.

Post milestones for heimdall these types are currently downloaded by
erigon but not persisted locally. This change adds persistence for these
types.

In addition to the pure persistence changes this PR also contains a
refactor step which is part of the process of extracting polygon related
types from erigon core into a seperate package which may eventually be
extracted to a separate module and possibly repo.

The aim is rather than the core `turbo\snapshotsync\freezeblocks` having
to know about types it manages and how to exaract and index their
contents this can concern it self with a set of macro shard management
actions.

This process is partially completed by this PR, a final step will be to
remove BorSnapshots and to simplify the places in the code which has to
remeber to deal with them. This requires further testing so has been
left out of this PR to avoid delays in delivering the base types.

# Status

* Waypont types and storage are complete and integrated in to the
BorHeimdall stage, The code has been tested to check that types are
inserted into mdbx, extracted and merged correctly
* I have verified that when produced from block 0 the new snapshot
correctly follow the merging strategy of existing snapshots
* The functionality is enables by a **--bor.waypoints=true** this is
false by default.

# Testing

This has been tested as follows:

* Run a Mumbai instance to the tip and check current processing for
milestones and checkpoints

# Post merge steps

* Produce and release snapshots for mumbai and bor mainnet
* Check existing node upgrades
* Remove --bor.waypoints flags

When adding bor waypont types I have removed snaptype.AllTypes because
it causes package cross-dependencies.

This fixes the places where all types have been used post the merge
changes.

… occasiona Reorg (erigontech#10085)

This PR
- avoids installing Golang on every test run, 
- clean up the testbed datadir at the end of the test

- upgrade docker 
- remove tendermint

…ntech#10105)

running `go generate ./...` fails with:
```
codecgen error: error running 'go run codecgen-main-2.generated.go': exit status 1, console: panic: encoding alphabet includes duplicate symbols

goroutine 1 [running]:
encoding/base64.NewEncoding(...)
    /usr/local/go/src/encoding/base64/base64.go:82
github.com/ugorji/go/codec.init()
    /Users/milen/go/pkg/mod/github.com/ugorji/go/[email protected]/gen.go:168 +0xf1c
exit status 2
```

this is a problem when using go1.22 and it has been fixed here:
-
ugorji/go@8286c2d
- issue: ugorji/go#407

relates to erigontech#10139

…rigontech#10147)

This resolves erigontech#10135

All enums are constrained by their owning type which forces package
includsion and hence type registration.

Added tests for each type to check the construction cycle.

Fix a timing hole where torrents that get created before webseeds have
been downloaded don't get webseeds set.

…ontech#10158)

Cherry-pick PR erigontech#10119 into the release

Co-authored-by: Marcello Ardizzone <[email protected]>

…erigontech#10166)

Fixes
erigontech#10157 (comment)

Problem was:
```
grep -r -l --exclude-dir=erigon-lib "^// Code generated by MockGen. DO NOT EDIT.$" . | xargs rm -r
```

was deleting the `mockgen` binary after it was built 🙃