Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge pre v2.56 #74

Merged
merged 194 commits into from
Jan 24, 2024
Merged

Merge pre v2.56 #74

merged 194 commits into from
Jan 24, 2024

Conversation

jyellick
Copy link

This brings our develop branch just up to before the v2.56 release was tagged. We'll want to consider how we do our branching strategy, but I think it makes sense to pick 2.56 as our RC to be maintained with fixes as needed, while we continue to track the upstream separately.

dlscjf151 and others added 30 commits November 9, 2023 07:40
Add Agra HF details for bor mainnet.
…ontech#8683)

we plan step-by-step keep increasing this default
still see users for who it helped to handle more rpc 
tradeoff: increasing of this flag - increasing "historical rpc"
throughput and decreasing "recent data rpc" throughput
…12.1 in /erigon-lib (erigontech#8680)

Bumps
[github.com/consensys/gnark-crypto](https://github.com/consensys/gnark-crypto)
from 0.12.0 to 0.12.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/consensys/gnark-crypto/releases">github.com/consensys/gnark-crypto's
releases</a>.</em></p>
<blockquote>
<h2>v0.12.1</h2>
<h2>What's Changed</h2>
<ul>
<li>fix: GLV exponentiation in GT with large exponents by <a
href="https://github.com/ThomasPiellard"><code>@​ThomasPiellard</code></a>
in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/451">Consensys/gnark-crypto#451</a>
<a
href="https://github.com/Consensys/gnark-crypto/security/advisories/GHSA-pffg-92cg-xf5c">Security
Advisory</a></li>
<li>feat: kzg extended transcript by <a
href="https://github.com/ThomasPiellard"><code>@​ThomasPiellard</code></a>
in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/452">Consensys/gnark-crypto#452</a></li>
<li>perf(fft): introduce cache efficient bit reverse shuffling by <a
href="https://github.com/gbotrel"><code>@​gbotrel</code></a> in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/446">Consensys/gnark-crypto#446</a></li>
<li>perf: Improve performance of point equality checks by <a
href="https://github.com/jsign"><code>@​jsign</code></a> in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/450">Consensys/gnark-crypto#450</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/Consensys/gnark-crypto/compare/v0.12.0...v0.12.1">https://github.com/Consensys/gnark-crypto/compare/v0.12.0...v0.12.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/da0317fd013308db6ce847bc9c3d506a2a3ae0ff"><code>da0317f</code></a>
clean: remove useless snippet in mulWindowed (<a
href="https://redirect.github.com/consensys/gnark-crypto/issues/453">#453</a>)</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/65cdb1d1c214e51454ae6dad4c466324f3ae510d"><code>65cdb1d</code></a>
Feat/kzg extended transcript (<a
href="https://redirect.github.com/consensys/gnark-crypto/issues/452">#452</a>)</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/95e674b3d69279dd0446e34cd8960e2fc4ed8393"><code>95e674b</code></a>
perf(fft): introduce cache efficient bit reverse shuffling (<a
href="https://redirect.github.com/consensys/gnark-crypto/issues/446">#446</a>)</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/ec6be1a037f7c496d595c541a8a8d31c47bcfa3d"><code>ec6be1a</code></a>
Merge pull request <a
href="https://redirect.github.com/consensys/gnark-crypto/issues/451">#451</a>
from Consensys/fix/glv</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/aef1eef48ab992db67631dcbec1c8dc259673e5b"><code>aef1eef</code></a>
fix: add GT ExpGLV fix to BLS24 + BW6</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/feaf161b4d1e53abf68b5679780ab497ef4818d4"><code>feaf161</code></a>
fix: merge and fix tests for expGLV</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/ced3076befee5c056f80ec5301438001ef5e9cdd"><code>ced3076</code></a>
fix: use max(size(s1), size(s2)) for ExpGLV</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/4096ad9913f1bfc9c67cdddade36ba981ade72ac"><code>4096ad9</code></a>
feat: discared useless files</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/ace5318b726c7185249bc9c92e1cadae701c1654"><code>ace5318</code></a>
Merge pull request <a
href="https://redirect.github.com/consensys/gnark-crypto/issues/450">#450</a>
from jsign/jsign-eq-improv</li>
<li><a
href="https://github.com/Consensys/gnark-crypto/commit/dc4e2d4f7799c4c36e213d1278fd3b77c07eb4da"><code>dc4e2d4</code></a>
feat: added test for [-s]G=-[s]G</li>
<li>Additional commits viewable in <a
href="https://github.com/consensys/gnark-crypto/compare/v0.12.0...v0.12.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/consensys/gnark-crypto&package-manager=go_modules&previous-version=0.12.0&new-version=0.12.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ledgerwatch/erigon/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
At the team meeting in Istanbul we decided to remove the auto-stale bot.
This PR adds beacon blocks snapshots for the following chains:

* Mainnet snapshots
* Sepolia snapshots
erigontech#7748

Paramatrizing a hardcoded parameter through CLI flags. 100k is the
default, as it was set previously.
go: no module declaration in go.mod. To specify the module path:
	go mod edit -module=example.com/mod
Update the list with nodes collected by observer
```
EROR] [11-16|07:33:02.592] RPC method eth_getLogs crashed: runtime error: invalid memory address or nil pointer dereference
Nov 16 07:33:02 i-0e4d4ce2636f49d8a erigon.sh[1739584]: [service.go:219 panic.go:914 panic.go:261 signal_unix.go:861 block_reader.go:615 block_reader.go:405 eth_receipts.go:228 value.go:596 value.go:380 service.go:224 handler.go:493 handler.go:443 handler.go:391 handler.go:222 handler.go:315 asm_amd64.s:1650]
```
Tests that don't affect each other should run in parallel
This is a similar PR of
ethereum/go-ethereum#28460, support traceCall in
the middle of a block

Signed-off-by: jsvisa <[email protected]>
# Background

Erigon currently uses a combination of Victoria Metrics and Prometheus
client for providing metrics.

We want to rationalize this and use only the Prometheus client library,
but we want to maintain the simplified Victoria Metrics methods for
constructing metrics.

This task is currently partly complete and needs to be finished to a
stage where we can remove the Victoria Metrics module from the Erigon
code base.

## Tests

### Functional
* Make sure that the format change int->float implied by VM to
Prometheus does not impact clients (pay particular attention to block
numbers)
* Check that the prometheus/grafana dashboards defined in cmd/prometheus
are functional after the change
(see docker-compose.yml for details and
https://github.com/ledgerwatch/erigon/tree/devel/cmd/prometheus#readme)
* Confirm that the underlying go metrics are still generated
* Confirm the following flags setting work:
    --metrics, --metrics.addr, --metrics.port with the new code
* Confirm that --metrics and --proff settings and handlers configuration
still allow metrics and pprof to share a port

#### Float counters - scientific notation test case
![Screenshot_2023-11-07_at_15 57
21](https://github.com/ledgerwatch/erigon/assets/94537774/32f0a6f6-968b-477c-8ec8-bb1812f3e848)

![Screenshot 2023-11-15 at 16 26
56](https://github.com/ledgerwatch/erigon/assets/94537774/3f402b2e-e343-4928-9fbb-18fa4d077485)


#### Float counters - NaN test case
![Screenshot_2023-11-07_at_16 04
25](https://github.com/ledgerwatch/erigon/assets/94537774/cbf90d5d-3749-4bd7-971d-e2124e54267c)

![Screenshot 2023-11-15 at 16 28
36](https://github.com/ledgerwatch/erigon/assets/94537774/5924915e-1977-4b7f-8082-23f73d0957d5)

### Performance
* Check the performance of counters created by RPC calls measurements
created by rpc/metrics.go are not impacted by the change.

#### RPC
Performed tests on rpcdaemon & erigon on localhost using
`etc_blockNumber`.
Did tests with 100, 1000, 10000 requests. Got a steady 15 ms response
time.

#### Memory
![Screenshot 2023-11-16 at 09 58
39](https://github.com/ledgerwatch/erigon/assets/94537774/5dd956d7-903f-4bea-a460-d3644da56201)
- changed communication tunnel to web socket in order to connect to
remote nodes
- changed diagnostics.url flag to diagnostics.addr as now user need to
enter only address and support command will connect to it through
websocket
- changed flag debug.urls to debug.addrs in order to have ability to
change connection type between erigon and support to websocket and don't
change user API
- added auto trying to connect to connect to ws if connection with was
failed
Problem:
Some goroutines are blocked on shutdown:
1. table close <-tab.closed // because table loop pending
1. table loop <-refreshDone // because lookup shutdown blocks doRefresh
1. lookup shutdown <-it.replyCh // because it.queryfunc (findnode -
ensureBond) is blocked, and not returning errClosed (if it returns and
pushes to it.replyCh, then shutdown() will unblock)
1. findnode - ensureBond <-rm.errc // because the related replyMatcher
was added after loop() exited, so there's nothing to push errClosed and
unlock it

If addReplyMatcher channel is buffered, it is possible that
UDPv4.pending() adds a new reply matcher after closeCtx.Done().
Such reply matcher's errc result channel will never be updated, because
the UDPv4.loop() has exited at this point. Subsequent discovery
operations will deadlock.

Solution:
Revert to an unbuffered channel.
AlexeyAkhunov and others added 24 commits December 21, 2023 12:42
First "smoke test", to check Erigon clean exiting on ctrl-c
erigon didn't download form webseeds if StageSnapshots progress > 0. but
sometime we want "just delete files and re-download them"
…ntech#9054)

Found the different log style for announced and broadcasted tx:

```
[INFO] [12-22|05:18:01.363] Local tx broadcasted                     txHash=ec6b1c87aafd7f8ead5794477be50bda696f2ce17271ad4f6022a756722fa0be to peer=10
[INFO] [12-22|05:18:01.363] local tx announced                       tx_hash=ec6b1c87aafd7f8ead5794477be50bda696f2ce17271ad4f6022a756722fa0be to peer=40 baseFee=1
```

adjust them to the same style

Signed-off-by: jsvisa <[email protected]>
`STAGES_ONLY_BLOCKS=true` may help to produce BlockSnaps by Erigon2 on
weak machines - it disabling all stages after StageSenders.
Add paths to the hiemdall config URL when creating calls so that extra
paths needs by, for example proxy servers are not stripped from the flag
value passed into the process.
This PR is ready to review. 

PR introduces `blocksByRange` and `blocksByRoot` P2P RPC methods
- `blocksByRange` - allows peers to request a range of blocks
- `blocksByRoot` - enables block requests using their root hashes(list
format)

Reference:
https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#beaconblocksbyrange
* Chunked format -> blinded
* LZ4 -> ZSTD
* Implemented parent block root support for history download
* Rationale: Allows to optimize GC collection easily on state
reconstruction and it allows to read fast attestations in historical
states reader
We upstreamed the txpool disable gossip changes, but they're slightly
different from what we have in our code.  The merge messed things up a
bit, so this commit fixes them up.
The job simply fails with a "This Job Failed" message, enabling debug
logs does nothing, and it seems likely/possible that this is a quota
issue of some sort.
@jyellick jyellick merged commit d87c004 into boba-develop Jan 24, 2024
7 checks passed
jyellick pushed a commit that referenced this pull request Feb 26, 2024
…-tx-gossip

Update README for Transaction gossiping
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.