Sync & connection hangs

[juligasa]

Describe the bug
When doing a peer sync (force sync in frontend) while connecting to a peer, the connection/syncing loop hangs

To Reproduce Steps to reproduce the behavior:

1. Run daemon
2. Connect to a peer
3. While still connecting, quickly fire a force sync
4. Try to connect to another peer
5. The syncyng loop should be hung by now
6. Reset the daemon
7. Get panick when init libp2p

Expected behavior
queuing sync until the connection in step 2 is resolved.

Screenshots If applicable, add screenshots to help explain your problem.

2022-09-19T12:52:24.492+0200    DEBUG    mintter/network    mttnet/connect.go:39    ConnectStarted    {"device": "bafzaajaiaejca7ykuk2cqklyqzztqladwxk2374fsibtinazu5ndilobrrcpsxja"}
2022-09-19T12:52:46.471+0200    INFO    mintter/syncing    syncing/syncing.go:176    SyncLoopStarted    {"traceID": 1663584766}
2022-09-19T12:52:46.471+0200    DEBUG    mintter/network    mttnet/connect.go:39    ConnectStarted    {"device": "bafzaajaiaejcaccp3hzuju6w54mmiitzdlavtpwumzxy2pobqrwo6ju4izayjygh"}
2022-09-19T12:53:03.968+0200    DEBUG    mintter/network    mttnet/connect.go:41    ConnectFinished    {"device": "bafzaajaiaejcaccp3hzuju6w54mmiitzdlavtpwumzxy2pobqrwo6ju4izayjygh", "error": "failed to connect to peer 12D3KooWANp3vz8emDZb1P6rHAnwnn9SyxeJse4sZNpAbLxey8ML: routing: not found", "isMintterPeer": false}
2022-09-19T12:54:31.909+0200    DEBUG    mintter/network    mttnet/connect.go:39    ConnectStarted    {"device": "bafzaajaiaejca7ykuk2cqklyqzztqladwxk2374fsibtinazu5ndilobrrcpsxja"}
2022-09-19T13:06:51.555+0200	DEBUG	autorelay	autorelay/relay_finder.go:637	refreshed relay slot reservation	{"relay": "12D3KooWNmjM4sMbSkDEA6ShvjTgkrJHjMya46fhZ9PjKZ4KVZYq"}

note the time lapse between the two last traces.

Every time daemon restarts after that event, it panics:

2022-09-19T13:10:23.655+0200    DEBUG    mintter/network    mttnet/connect.go:41    ConnectFinished    {"device": "bafzaajaiaejcb472txdxwhtfvk74fekdimwcytjsdjtgnnovggpc352ugtitsh2o", "isMintterPeer": true}
panic: proof not found

goroutine 6321 [running]:
mintter/backend/mttnet.(*Node).handshakeInfo.func1()
    /home/julio/Documents/mintter/plz-out/tmp/backend/mintterd._build/backend/mttnet/connect.go:178 +0x206
sync.(*Once).doSlow(0xc0004f8000?, 0x19b1a90?)
    /nix/store/dlpbgh2hqzk83gs4xhz3r1b0x24hfcpw-go-1.18.2/share/go/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
    /nix/store/dlpbgh2hqzk83gs4xhz3r1b0x24hfcpw-go-1.18.2/share/go/src/sync/once.go:59
mintter/backend/mttnet.(*Node).handshakeInfo(0xc001356640?, {0x19b1a90?, 0xc0028bb860?})
    /home/julio/Documents/mintter/plz-out/tmp/backend/mintterd._build/backend/mttnet/connect.go:158 +0x6e
mintter/backend/mttnet.(*Node).Connect(0xc001356640, {0x19b1a90, 0xc0028bb860}, {{0xc0027ea330, 0x26}, {0xc002ccb6b0, 0x1, 0x1}})
    /home/julio/Documents/mintter/plz-out/tmp/backend/mintterd._build/backend/mttnet/connect.go:67 +0x51d
mintter/backend/mttnet.(*Node).Client(0xc001356640, {0x19b1a90, 0xc0028bb860}, {{0xc0027ea270?, 0x28?}})
    /home/julio/Documents/mintter/plz-out/tmp/backend/mintterd._build/backend/mttnet/mttnet.go:174 +0x11a
mintter/backend/syncing.(*Service).syncWithPeer(0xc00122a320, {0x19b1a90, 0xc0028bb860}, {{0xc0027ea270?, 0xc000f5ef84?}})
    /home/julio/Documents/mintter/plz-out/tmp/backend/mintterd._build/backend/syncing/syncing.go:371 +0x63
mintter/backend/syncing.(*Service).Sync.func1(0x1, {{0xc0028c6030, 0x26, 0x26}, 0x2, 0x2})
    /home/julio/Documents/mintter/plz-out/tmp/backend/mintterd._build/backend/syncing/syncing.go:254 +0x225
created by mintter/backend/syncing.(*Service).Sync
    /home/julio/Documents/mintter/plz-out/tmp/backend/mintterd._build/backend/syncing/syncing.go:240 +0x61c

Desktop (please complete the following information):

• OS: Linux

Additional context Add any other context about the problem here.

[burdiyan]

@juligasa I tried testing this, and indeed there's something wrong in there. In my tests, the process hangs right inside the libp2p.Connect code, but I don't know why.

I see that libp2p is finally made a monorepo, and I wanted to update it, but I don't know if we still must use our fork, or were those relay issues fixed in the upstream.

https://github.com/libp2p/go-libp2p/releases/tag/v0.22.0

[juligasa]

#736 Maybe a related issue. since the crash after a network restart seems to be the same reason panic: proof not found The only difference is that in #736 we are working with new libp2p libraries and lastest master

[juligasa]

Im not able to reproduce it anymore after we upgraded libp2p version to latest.