Erigon 0.40.1 mainnet + embeded CL stopped syncing

[zekchan]

System information

Erigon version: ./erigon --version
erigon version 2.40.1-stable-94b4cee5
OS & Version: Windows/Linux/OSX
ubuntu 22.04 Docker
Commit hash:

Erigon Command (with flags/config):

"--txpool.disable",
                "--v5disc",
                "--lightclient.discovery.addr=***",
                "--log.dir.path=/data/logs",
                "--chain",
                "mainnet",
                "--metrics",
                "--rpc.returndata.limit=999999999",
                "--rpc.batch.limit=2000",
                "--metrics.addr",
                "***",
                "--metrics.port",
                "6060",
                "--datadir",
                "/data",
                "--torrent.download.rate=1000mb",
                "--http.api=eth,erigon,web3,net,debug,trace,txpool",
                "--http.addr",
                "***",
                "--http.port",
                "***:,
                "--http.vhosts",
                "*",
                "--ws",
                "--http"

Concensus Layer:
Embeded
Concensus Layer Command (with flags/config):

Chain/Network:
Mainnet

Expected behaviour

Syncing

Actual behaviour

I have two erigons in docker with same setups. Both stoppped 16850951-16850951 and started do somehing only in several minutes with long lags

Logs

Nothing just that in both nodes:

[WARN] [03-18|00:44:35.849] Could not validate update                err="invalid sync committee"
[WARN] [03-18|00:44:36.252] Could not validate update                err="invalid sync committee"
[WARN] [03-18|00:44:38.259] Could not send beacon block to ETH1      err="missing attestation data"
[WARN] [03-18|00:44:41.850] Could not validate update                err="invalid sync committee"
[WARN] [03-18|00:44:44.781] Could not validate update                err="invalid sync committee"

Sync stages graph:

[tirumerla]

I'm seeing similar issues on one of the node with 2.40.1

[fernandezpaco]

Same here but different logs, i created an issue but i have closed it in favor of this one just in case it is the same problem

#7131

[egorFiNE]

2.40.1, mainnet.

Same issue. Node stopped at 16850994.

[whidrasl]

same. block 16854519

  |   | [WARN] [03-18\|16:32:32.195] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:32.833] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:33.033] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:33.434] Could not validate update                err="invalid sync committee"
  |   | [INFO] [03-18\|16:32:33.963] [p2p] GoodPeers                          eth68=102 eth66=9 eth67=91
  |   | [WARN] [03-18\|16:32:34.035] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:58.668] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:59.268] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:59.473] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:59.673] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:32:59.873] Could not validate update                err="invalid sync committee"

...

  |   | [WARN] [03-18\|16:33:00.274] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:33:01.075] Could not validate update                err="invalid sync committee"
  |   | [INFO] [03-18\|16:33:01.727] [LightClient] P2P                        peers=180
  |   | [INFO] [03-18\|16:33:01.741] [txpool] stat                            block=16854519 pending=10000 baseFee=3044 queued=30000 alloc=3.9GB sys=8.3GB
  |   | [WARN] [03-18\|16:33:02.327] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:33:02.527] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:33:03.401] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:33:04.201] Could not validate update                err="invalid sync committee"
  |   | [WARN] [03-18\|16:33:05.403] Could not validate update                err="invalid sync committee"

[whidrasl]

same on Goerli

[whidrasl]

same for v2.38.1 (with embedded Consensus layer)

[egorFiNE]

Sometimes it syncs a few blocks then continues.

[wainach]

Seeing same issue.

[winchell]

Same. Goerli. I have another Goerli node using Lighthouse CL and it's fine. v2.40.1 for both.

[Giulio2002]

#7134 fixed this

[whidrasl]

there is no more such logs since 19:50 GMT. Even without fix/without restarting nodes/with v2.40.1/v2.38.1

[egorFiNE]

Same here, it stopped without fixes and happily runs for at least 10h now.

[egorFiNE]

...and we're back.

@Giulio2002 could you please release a stable version with this fix included?

[jisunglim]

mainnet v2.40.1. Same here. It's back.

[jisunglim]

After I downgraded to v2.40.0, It's working.

[ybstaked]

I'm still seeing same issue even with git_branch=HEAD git_tag=v2.42.0-dirty git_commit=beb97784d43ece5acde365a74efe8763692ebdba

[INFO] [03-29|14:23:52.550] [2/15 Headers] Waiting for Consensus Layer...
[INFO] [03-29|14:24:01.019] [2/15 Headers] Handling new payload      height=16933519 hash=0xc3e8476632bbb842e1e781e2c5823ca41ba18ec31ac977fb5f5c60368459bd92
[INFO] [03-29|14:24:01.766] [2/15 Headers] Waiting for Consensus Layer...
[INFO] [03-29|14:24:02.266] [2/15 Headers] Fork choice update: flushing in-memory state (built by previous newPayload)
[INFO] [03-29|14:24:02.921] Commit cycle                             in=56.565277ms
[INFO] [03-29|14:24:02.921] Timings (slower than 50ms)               Headers=566ms CallTraces=121ms AccountHistoryIndex=86ms StorageHistoryIndex=142ms LogIndex=84ms TxLookup=97ms
[INFO] [03-29|14:24:02.921] Tables                                   PlainState=88.8GB AccountChangeSet=185.6GB StorageChangeSet=283.3GB BlockTransaction=10.8GB TransactionLog=551.7GB FreeList=143.1MB ReclaimableSpace=143.1GB
[INFO] [03-29|14:24:02.921] RPC Daemon notified of new headers       from=16933518 to=16933519 hash=0xc3e8476632bbb842e1e781e2c5823ca41ba18ec31ac977fb5f5c60368459bd92 header sending=12.998µs log sending=211ns
[INFO] [03-29|14:24:04.893] [2/15 Headers] Waiting for Consensus Layer...

[begetan]

The same issue with erigon v2.48.0 and prysm v4.0.5
I've updated to the v2.48.1 and prysm v.4.0.8 now and want to open a new GitHub issue with the same name