force close on peer with "internal error" no further explanations

[heyambob]

Issue and Steps to Reproduce

It's been the 4th time this happened. This time I was able to coordinate with my LND peer and I have more detail to share. The force close happened when I was doing maintenace on the machine, a couple restart on c-lightning in the low traffic time.

In my log:

2022-03-14T08:52:50.319Z UNUSUAL 03xxxchan#10423: Peer permanent failure in CHANNELD_NORMAL: channeld: received ERROR error channel f8xxx: internal error
2022-03-14T08:52:50.321Z INFO    03xxxchan#10423: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL

on his log:

Mar 14 08:42:23 lx lnd[1265]: 2022-03-14 08:42:23.510 [INF] DISC: Creating new GossipSyncer for peer=02xxx Mar 14 08:52:59 lx lnd[1265]: 2022-03-14 08:52:59.969 [INF] PEER: unable to read message from 02xxx EOF
Mar 14 08:52:59 lx lnd[1265]: 2022-03-14 08:52:59.969 [INF] PEER: disconnecting 02xxx, reason: read handler closed
Mar 14 08:53:00 lx lnd[1265]: 2022-03-14 08:53:00.171 [INF] DISC: Removing GossipSyncer for peer=020xxx

Nothing really stands out in his log. There are several "unable to read message" and "read handler closed", before and after the incidence

getinfo output

When it happened I was still running with

lightningd v0.10.2-262-g1087f9a-modded

Now I just upgraded to

lightningd v0.10.2-456-ga2a6b8c

In the hope that the issue has been fixed

[niftynei]

The channel closed because the peer returned an error message. The received ERROR is the hint here, everything after this is provided by the peer's implementation.

2022-03-14T08:52:50.319Z UNUSUAL 03xxxchan#10423: Peer permanent failure in CHANNELD_NORMAL: channeld: received ERROR error channel f8xxx: internal error

'Unable to read message' seems like a good hint. Typically EOF is the signal that you've reached the end of a stream to read etc. Maybe worth asking the other impl for help figuring out what cases this might happen in.

Edited.

[heyambob]

I ran into a few more like this and I caught a good one with c-otto which he described it here: lightningnetwork/lnd#6485

[heyambob]

I ran into this one, it's my 2nd time, lightningnetwork/lnd#6564 . This one is interesting though as LND cannot read something that we sent and eventually the "internal error" was fired. Devs on both side should work together

[zerofeerouting]

Just an update. I ran into something similar with @ZoltanAB lightningnetwork/lnd#6593

It feels like LND is sending ERROR messages for something that's not worth to force-close a channel over, but CLN behaving according to the specs does.

[whitslack]

It feels like LND is sending ERROR messages for something that's not worth to force-close a channel over, but CLN behaving according to the specs does.

This wouldn't be the first time. LND hands out ERROR messages like candy on Halloween.

[rustyrussell]

#5326 should fix this?

[zerofeerouting]

I think we can close this one.

Running the fix right now. Will open a new issue in case something like this happens again.