Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Erigon stuck on shutdown #9119

Closed
wmitsuda opened this issue Jan 2, 2024 · 4 comments
Closed

Erigon stuck on shutdown #9119

wmitsuda opened this issue Jan 2, 2024 · 4 comments
Assignees
@battlmonstr

Comments

@wmitsuda
Copy link
Member

wmitsuda commented Jan 2, 2024

The following log is from e35 branch from a few days ago commit, but I've noticed something similar on releases branches (although not sure if it is the same scenario).

anyway, posting stack trace from what I found out since this one is short:

INFO[01-02|17:54:14.157] [p2p] GoodPeers                          eth68=96 eth67=89
INFO[01-02|17:54:14.165] [txpool] stat                            pending=10000 baseFee=0 queued=30000 alloc=1.8GB sys
=3.0GB
^CINFO[01-02|17:55:03.993] Got interrupt, shutting down...
INFO[01-02|17:55:03.993] Exiting Engine...
INFO[01-02|17:55:03.993] RPC server shutting down
INFO[01-02|17:55:03.993] RPC server shutting down
INFO[01-02|17:55:03.993] Exiting...
INFO[01-02|17:55:03.993] Engine HTTP endpoint close               url=127.0.0.1:8551
INFO[01-02|17:55:03.993] HTTP endpoint closed                     url=[::]:8545
INFO[01-02|17:55:03.993] HTTP endpoint closed                     url=[::]:8546
INFO[01-02|17:55:03.993] RPC server shutting down









^CWARN[01-02|17:57:57.798] Already shutting down, interrupt more to panic. times=9
^CWARN[01-02|17:57:58.696] Already shutting down, interrupt more to panic. times=8
^CWARN[01-02|17:57:59.022] Already shutting down, interrupt more to panic. times=7
^CWARN[01-02|17:57:59.178] Already shutting down, interrupt more to panic. times=6
^CWARN[01-02|17:57:59.314] Already shutting down, interrupt more to panic. times=5
^CWARN[01-02|17:57:59.450] Already shutting down, interrupt more to panic. times=4
^CWARN[01-02|17:57:59.598] Already shutting down, interrupt more to panic. times=3
^CWARN[01-02|17:57:59.757] Already shutting down, interrupt more to panic. times=2
^CWARN[01-02|17:57:59.930] Already shutting down, interrupt more to panic. times=1
^Cpanic: boom

goroutine 1985 [running]:
github.com/ledgerwatch/erigon/turbo/debug.LoudPanic(...)
        github.com/ledgerwatch/erigon/turbo/debug/loudpanic.go:24
github.com/ledgerwatch/erigon/turbo/debug.ListenSignals({0x310c8c0?, 0xc00095a000}, {0x313cc40, 0xc000a75b20})
        github.com/ledgerwatch/erigon/turbo/debug/signal.go:38 +0x35c
created by github.com/ledgerwatch/erigon/node.StartNode in goroutine 1
        github.com/ledgerwatch/erigon/node/node.go:414 +0xe5

goroutine 1 [chan receive, 6394 minutes]:
github.com/ledgerwatch/erigon/node.(*Node).Wait(...)
        github.com/ledgerwatch/erigon/node/node.go:268
github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve(0xc034f90ef0)
        github.com/ledgerwatch/erigon/turbo/node/node.go:34 +0x6e
main.runErigon(0xc00146ca00)
        github.com/ledgerwatch/erigon/cmd/erigon/main.go:75 +0x405
github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1(0xc00146ca00)
        github.com/ledgerwatch/erigon/turbo/app/make_app.go:52 +0x19f
github.com/urfave/cli/v2.(*Command).Run(0xc00148a160, 0xc00146ca00, {0xc0000520e0, 0xe, 0xe})
        github.com/urfave/cli/[email protected]/command.go:274 +0x998
github.com/urfave/cli/v2.(*App).RunContext(0xc0000394a0, {0x3128af0?, 0x52c1c40}, {0xc0000520e0, 0xe, 0xe})
        github.com/urfave/cli/[email protected]/app.go:332 +0x5b7
github.com/urfave/cli/v2.(*App).Run(...)
        github.com/urfave/cli/[email protected]/app.go:309
main.main()
        github.com/ledgerwatch/erigon/cmd/erigon/main.go:34 +0x85
                                                                                                            [212/1876]
goroutine 12 [chan receive, 6394 minutes]:
gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0xc001275920)
        gopkg.in/natefinch/[email protected]/lumberjack.go:379 +0x3f
created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1 in goroutine 1
        gopkg.in/natefinch/[email protected]/lumberjack.go:390 +0x96

goroutine 1797 [chan receive, 6394 minutes]:
github.com/ledgerwatch/erigon-lib/direct.(*SubscribeLogsStreamS).Recv(0xc034fa2640?)
        github.com/ledgerwatch/[email protected]/direct/eth_backend_client.go:158 +0x25
github.com/ledgerwatch/erigon/ethdb/privateapi.(*LogsFilterAggregator).subscribeLogs(0xc034fa2640, {0x31376b8, 0xc03d0
10810})
        github.com/ledgerwatch/erigon/ethdb/privateapi/logsfilter.go:133 +0xab
github.com/ledgerwatch/erigon/ethdb/privateapi.(*EthBackendServer).SubscribeLogs(0x0?, {0x31376b8?, 0xc03d010810?})
        github.com/ledgerwatch/erigon/ethdb/privateapi/ethbackend.go:256 +0x2a
github.com/ledgerwatch/erigon-lib/direct.(*EthBackendClientDirect).SubscribeLogs.func1()
        github.com/ledgerwatch/[email protected]/direct/eth_backend_client.go:125 +0x92
created by github.com/ledgerwatch/erigon-lib/direct.(*EthBackendClientDirect).SubscribeLogs in goroutine 1875
        github.com/ledgerwatch/[email protected]/direct/eth_backend_client.go:122 +0x131

goroutine 1812 [chan receive]:
github.com/ledgerwatch/erigon/consensus/ethash.(*meterArbiter).tick(0x5282920)
        github.com/ledgerwatch/erigon/consensus/ethash/meter.go:143 +0x7c
created by github.com/ledgerwatch/erigon/consensus/ethash.newHashRateMeter in goroutine 1
        github.com/ledgerwatch/erigon/consensus/ethash/meter.go:19 +0xbf

goroutine 1875 [chan receive, 6394 minutes]:
github.com/ledgerwatch/erigon-lib/direct.(*SubscribeLogsStreamC).Recv(0xc03e7565e8?)
        github.com/ledgerwatch/[email protected]/direct/eth_backend_client.go:185 +0x25
github.com/ledgerwatch/erigon/cmd/rpcdaemon/rpcservices.(*RemoteBackend).SubscribeLogs(0xc034fa3130, {0x3128f88, 0xc00
17961e0}, 0xc03c472910, 0x0?)
        github.com/ledgerwatch/erigon/cmd/rpcdaemon/rpcservices/eth_backend.go:219 +0x1bf
github.com/ledgerwatch/erigon/turbo/rpchelper.New.func2()
        github.com/ledgerwatch/erigon/turbo/rpchelper/filters.go:101 +0xe6
created by github.com/ledgerwatch/erigon/turbo/rpchelper.New in goroutine 1
        github.com/ledgerwatch/erigon/turbo/rpchelper/filters.go:91 +0x5fe

goroutine 1979 [chan receive, 4 minutes]:
github.com/ledgerwatch/erigon/p2p.(*dialScheduler).loop(0xc024cff380, {0x3123100, 0xc03e754ba0})
        github.com/ledgerwatch/erigon/p2p/dial.go:324 +0x105b
created by github.com/ledgerwatch/erigon/p2p.newDialScheduler in goroutine 1888
        github.com/ledgerwatch/erigon/p2p/dial.go:186 +0x4db

goroutine 6443904 [chan receive, 4 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown(...)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance(0xc047eae2d0)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88 +0x2a5
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run(0xc047eae2d0)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64 +0x25
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey(0xc0392d4620, 0x1e842d9f0a169b8f?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:321 +0xef
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func5()
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:684 +0x99
created by github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop in goroutine 1793
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:682 +0x539

goroutine 6443905 [chan receive, 6157 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode(0xc0392d4620, {0x5f, 0x96, 0xda, 0x19, 0x20, 0x6d, 0xb3,
0x25, 0xd2, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:403 +0x305
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1(0x25b36d2019da965f?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:353 +0x9a
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query(0xc047eae2d0, 0xc04dd60c60, 0xc000616d80?)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:157 +0xaf
created by github.com/ledgerwatch/erigon/p2p/discover.(*lookup).startQueries in goroutine 6443904
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:128 +0x225

goroutine 1837 [syscall]:
os/signal.signal_recv()
        runtime/sigqueue.go:152 +0x29
os/signal.loop()
        os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1985
        os/signal/signal.go:151 +0x1f

goroutine 51292881 [chan receive, 4464 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode(0xc03e70b340, {0x29, 0xd5, 0x45, 0x87, 0x4c, 0x38, 0x61,
0xac, 0x13, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:403 +0x305
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1(0xac61384c8745d529?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:353 +0x9a
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query(0xc0617e6360, 0xc0355d43f0, 0xc000616d80?)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:157 +0xaf
created by github.com/ledgerwatch/erigon/p2p/discover.(*lookup).startQueries in goroutine 51292880
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:128 +0x225

goroutine 1935 [semacquire, 4 minutes]:
sync.runtime_Semacquire(0x45ead90?)
        runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc032c7a068?)
        sync/waitgroup.go:116 +0x48
github.com/ledgerwatch/erigon/p2p.(*dialScheduler).stop(0xc024cff380)
        github.com/ledgerwatch/erigon/p2p/dial.go:193 +0x2f
github.com/ledgerwatch/erigon/p2p.(*Server).run(0xc0320bfb00)
        github.com/ledgerwatch/erigon/p2p/server.go:901 +0x162d
created by github.com/ledgerwatch/erigon/p2p.(*Server).Start in goroutine 1888
        github.com/ledgerwatch/erigon/p2p/server.go:543 +0x56d

goroutine 111837324 [chan receive, 2042 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode(0xc03e70b340, {0x5b, 0x2c, 0x64, 0x6f, 0xa8, 0xdd, 0xc3,
0x28, 0x47, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:403 +0x305
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1(0x28c3dda86f642c5b?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:353 +0x9a
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query(0xc06ae1dc20, 0xc037ef8240, 0xc067dd5b00?)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:157 +0xaf
created by github.com/ledgerwatch/erigon/p2p/discover.(*lookup).startQueries in goroutine 111835630
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:128 +0x225

goroutine 14251 [chan receive, 4 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown(...)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance(0xc0424374d0)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88 +0x2a5
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run(0xc0424374d0)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64 +0x25
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey(0xc03e70b340, 0x8a791530ecb6e94b?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:321 +0xef
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func5()
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:684 +0x99
created by github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop in goroutine 1975
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:682 +0x539

goroutine 14252 [chan receive, 6393 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).ensureBond(0xc03e70b340, {0x86, 0x60, 0x56, 0xf3, 0x6e, 0xb2, 0x97
, 0x5b, 0x4a, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:815 +0xc5
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode(0xc03e70b340, {0x86, 0x60, 0x56, 0xf3, 0x6e, 0xb2, 0x97,
0x5b, 0x4a, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:367 +0x48
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1(0x5b97b26ef3566086?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:353 +0x9a
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query(0xc0424374d0, 0xc034bb39e0, 0x2dc5630?)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:157 +0xaf
created by github.com/ledgerwatch/erigon/p2p/discover.(*lookup).startQueries in goroutine 14251
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:128 +0x225

goroutine 13885843 [chan receive, 5889 minutes]:                                                             [74/1876]
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).ensureBond(0xc03e70b340, {0xaa, 0x5e, 0x5a, 0x75, 0x8d, 0x8e, 0xfc
, 0xa3, 0x95, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:815 +0xc5
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode(0xc03e70b340, {0xaa, 0x5e, 0x5a, 0x75, 0x8d, 0x8e, 0xfc,
0xa3, 0x95, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:367 +0x48
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1(0xa3fc8e8d755a5eaa?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:353 +0x9a
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query(0xc037977200, 0xc0447dd9e0, 0x0?)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:157 +0xaf
created by github.com/ledgerwatch/erigon/p2p/discover.(*lookup).startQueries in goroutine 13885842
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:128 +0x225

goroutine 162678293 [semacquire, 4 minutes]:                                                                 [60/1876]
sync.runtime_Semacquire(0xc034fa4ae0?)
        runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc03e76a780?)
        sync/waitgroup.go:116 +0x48
github.com/ledgerwatch/erigon/p2p.(*Server).Stop(0xc0320bfb00)
        github.com/ledgerwatch/erigon/p2p/server.go:454 +0x119
github.com/ledgerwatch/erigon/p2p/sentry.(*GrpcServer).Close(...)
        github.com/ledgerwatch/erigon/p2p/sentry/sentry_grpc_server.go:1209
github.com/ledgerwatch/erigon/eth.(*Ethereum).Stop(0xc0001e7080)
        github.com/ledgerwatch/erigon/eth/backend.go:1386 +0x225
github.com/ledgerwatch/erigon/node.(*Node).stopServices(0xc04d504748?, {0xc034f90ee0, 0x1, 0xc07bc443b0?})
        github.com/ledgerwatch/erigon/node/node.go:216 +0xb2
github.com/ledgerwatch/erigon/node.(*Node).Close(0xc00095a000)
        github.com/ledgerwatch/erigon/node/node.go:159 +0x185
created by github.com/ledgerwatch/erigon/turbo/debug.ListenSignals in goroutine 1985
        github.com/ledgerwatch/erigon/turbo/debug/signal.go:29 +0x266

goroutine 69563334 [chan receive, 4 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown(...)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance(0xc063bd4480)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88 +0x2a5
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run(0xc063bd4480)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64 +0x25
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey(0xc03e70b340, 0xec080d6b43343aeb?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:321 +0xef
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func5()
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:684 +0x99
created by github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop in goroutine 1975
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:682 +0x539

goroutine 69567128 [chan receive, 3741 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode(0xc03e70b340, {0xd9, 0x71, 0x39, 0x15, 0xe9, 0x4, 0xde, 0
x8e, 0xf3, ...}, ...)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:403 +0x305
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1(0x8ede04e9153971d9?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:353 +0x9a
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query(0xc063bd4480, 0xc065bd63f0, 0xc07b3c0500?)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:157 +0xaf
created by github.com/ledgerwatch/erigon/p2p/discover.(*lookup).startQueries in goroutine 69563334
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:128 +0x225

goroutine 111835630 [chan receive, 4 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown(...)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance(0xc06ae1dc20)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88 +0x2a5
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run(0xc06ae1dc20)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64 +0x25
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey(0xc03e70b340, 0x310dbe0?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:321 +0xef
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).Resolve(0xc03e70b340, 0xc0008ec000)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:247 +0x12f
github.com/ledgerwatch/erigon/p2p.(*dialTask).resolve(0xc03b3e30b0, 0xc024cff380)
        github.com/ledgerwatch/erigon/p2p/dial.go:532 +0xad
github.com/ledgerwatch/erigon/p2p.(*dialTask).run(0xc03b3e30b0, 0xbe2a4c?)
        github.com/ledgerwatch/erigon/p2p/dial.go:505 +0x8e
github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial.func1()
        github.com/ledgerwatch/erigon/p2p/dial.go:472 +0x45
created by github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial in goroutine 1979
        github.com/ledgerwatch/erigon/p2p/dial.go:470 +0x3b4

goroutine 13885842 [chan receive, 4 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown(...)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance(0xc037977200)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88 +0x2a5
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run(0xc037977200)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64 +0x25
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey(0xc03e70b340, 0x99e8dba17e9bddf2?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:321 +0xef
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func5()
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:684 +0x99
created by github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop in goroutine 1975
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:682 +0x539

goroutine 51292880 [chan receive, 4 minutes]:
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown(...)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance(0xc0617e6360)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88 +0x2a5
github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run(0xc0617e6360)
        github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64 +0x25
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey(0xc03e70b340, 0x241e0e616562cee2?)
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:321 +0xef
github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func5()
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:684 +0x99
created by github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop in goroutine 1975
        github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:682 +0x539
@0x1un
Copy link

0x1un commented Jan 3, 2024

I had the same problem on v2.55.1, but the developer replied that it was solved (hopelessly) and finally I was able to reach a normal shutdown using kill -HUP <PID>!

@toolchainx
Copy link

I had the same problem on v2.55.1

@battlmonstr battlmonstr self-assigned this Jan 10, 2024
@battlmonstr
Copy link
Contributor

This seems to be another regression from #8661 that went unnoticed.

battlmonstr added a commit that referenced this issue Jan 10, 2024
@battlmonstr
p2p/discv4: revert gotreply handler change from #8661 (#9119)
cab2386
mh0lt pushed a commit that referenced this issue Jan 11, 2024
@battlmonstr
p2p/discv4: revert gotreply handler change from #8661 (#9119) (#9195)
0449818 
The handler had race conditions in the candidates processing goroutine.
battlmonstr added a commit that referenced this issue Jan 11, 2024
@battlmonstr
p2p/discv4: revert gotreply handler change from #8661 (#9119) (#9195)
cb5b75a 
The handler had race conditions in the candidates processing goroutine.
yperbasis added a commit that referenced this issue Jan 11, 2024
@battlmonstr @yperbasis
p2p/discv4: revert gotreply handler change from #8661 (#9119) (#9195) (…
c9216ce 
…#9210)

Co-authored-by: yperbasis <[email protected]>
@darinvhs darinvhs mentioned this issue Jan 12, 2024
Merged
racytech pushed a commit to racytech/erigon that referenced this issue Jan 12, 2024
@battlmonstr @racytech
p2p/discv4: revert gotreply handler change from erigontech#8661 (erig…
bc78f59 
…ontech#9119) (erigontech#9195)

The handler had race conditions in the candidates processing goroutine.
wmitsuda added a commit that referenced this issue Jan 19, 2024
@wmitsuda
Squashed commit of the following:
68a6e0f 
commit 0c7300e
Author: Willian Mitsuda <[email protected]>
Date:   Fri Jan 12 20:05:43 2024 -0300

    Squashed work from ots2-alpha4

commit c9216ce
Author: battlmonstr <[email protected]>
Date:   Thu Jan 11 17:13:46 2024 +0100

    p2p/discv4: revert gotreply handler change from #8661 (#9119) (#9195) (#9210)

    Co-authored-by: yperbasis <[email protected]>

commit d079008
Author: Andrew Ashikhmin <[email protected]>
Date:   Mon Jan 8 17:44:30 2024 +0100

    release: Amoy bootnodes (#9166)

    Cherry pick #9158

    ---------

    Co-authored-by: Arpit Temani <[email protected]>

commit a49fcb8
Author: Andrew Ashikhmin <[email protected]>
Date:   Fri Jan 5 13:03:28 2024 +0100

    release params: remove dev from version (#9143)
@yperbasis
Copy link
Member

Should be fixed in v2.57.0 thanks to @battlmonstr. Please re-open if that's not the case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@battlmonstr battlmonstr

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@wmitsuda @toolchainx @battlmonstr @0x1un @yperbasis