[bug]: Unable to construct funding tx/derived public key does not match bip32 derivation info public key

[data-retriever]

Background

We are using two lnd instances (remote signer feature) and trying to open a channel:

lncli openchannel --node_key 02015cb10a74914fe6c126e591b1a4bda66662ca807e97a94cf973ffe408259f74 --local_amt 3000000 --sat_per_vbyte 1
[lncli] rpc error: code = Unknown desc = unable to construct funding tx: error signing with remoteinstance: remote signer returned invalid key spend signature: signature too short with 0 bytes

The relevant logs on the watch only instance are:

2022-10-11 20:43:02.050 [INF] SWPR: Manual fee rate input of 250 sat/kw is too low, using 253 sat/kw instead
2022-10-11 20:43:02.050 [INF] FNDG: Initiating fundingRequest(local_amt=0.03 BTC (subtract_fees=false), push_amt=0 mSAT, chain_hash=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f, peer=02015cb10a74914fe6c126e591b1a4bda66662ca807e97a94cf973ffe408259f74, min_confs=1)
2022-10-11 20:43:02.050 [WRN] LNWL: Web API does not have a fee rate for target=3, using the fee rate for target=2 instead
2022-10-11 20:43:02.050 [INF] CHFD: Performing funding tx coin selection using 253 sat/kw as fee rate
2022-10-11 20:43:02.065 [WRN] CHFD: Unable to find funding output for shim intent: unable to create witness script, no funding keys
2022-10-11 20:43:02.082 [INF] FNDG: Target commit tx sat/kw for pendingID(c2c839c49b0cc1ea2f1c8aa67c7e979c29f4cc3c219af6c48e50f9cf828f9699): 1275
2022-10-11 20:43:02.082 [INF] FNDG: Dust limit for pendingID(c2c839c49b0cc1ea2f1c8aa67c7e979c29f4cc3c219af6c48e50f9cf828f9699): 0.00000354 BTC
2022-10-11 20:43:02.082 [INF] FNDG: Starting funding workflow with lrttbpewpyf4w4ecprhgfih4ymhhssbnqmrcyjzbksjmwhxhc5ra63yd.onion:9736 for pending_id(c2c839c49b0cc1ea2f1c8aa67c7e979c29f4cc3c219af6c48e50f9cf828f9699), committype=tweakless
2022-10-11 20:43:03.311 [INF] FNDG: Recv'd fundingResponse for pending_id(c2c839c49b0cc1ea2f1c8aa67c7e979c29f4cc3c219af6c48e50f9cf828f9699)
2022-10-11 20:43:03.313 [ERR] FNDG: Unable to process contribution from &{{[2465652 16775426 43306903 33684062 40264394 6893417 40786203 20945668 17462417 22316]} {[59556220 3875047 1227694 23039521 940911 37078236 64646312 30028152 54437282 974180]}}: unable to construct funding tx: error signing with remoteinstance: remote signer returned invalid key spend signature: signature too short with 0 bytes
2022-10-11 20:43:03.313 [INF] FNDG: Cancelling funding reservation for node_key=02015cb10a74914fe6c126e591b1a4bda66662ca807e97a94cf973ffe408259f74, chan_id=c2c839c49b0cc1ea2f1c8aa67c7e979c29f4cc3c219af6c48e50f9cf828f9699
2022-10-11 20:43:03.313 [ERR] RPCS: unable to open channel to NodeKey(02015cb10a74914fe6c126e591b1a4bda66662ca807e97a94cf973ffe408259f74): unable to construct funding tx: error signing with remoteinstance: remote signer returned invalid key spend signature: signature too short with 0 bytes
2022-10-11 20:43:03.313 [ERR] RPCS: [/lnrpc.Lightning/OpenChannel]: unable to construct funding tx: error signing with remoteinstance: remote signer returned invalid key spend signature: signature too short with 0 bytes

If we look at the remote signer logs, the only relevant line is:

[WRN] BTWL: SignPsbt: Skipping input 0, derived public key 02464e51e1eede9721d0c97bc15cbfea5c38a6f6b820c69ac14623c971932464d3 does not match bip32 derivation info public key 02cd0efa576d888e03623a311e3192de0f2df06a88a0493ee64463c91954b3a075

The node is "identity_pubkey": "030e7d77f4012c809e4bcd0701e7d50b3a0869679f328e325e222ae56acabc74b2" which has dozens of channels. The only recent change was the update from 0.15.1 to 0.15.2 and some channels that are being force-closed.

Environment

Both instances are using 0.15.2-beta
Backend is a full bitcoin core 22 in neutrino mode (-txindex=1 -blockfilterindex=1 -peerblockfilters=1)

[data-retriever]

So I tried --reset-wallet-transactions and it did nothing to change the error.

Then I tried to open a smaller channel and got an err=Synchronizing blockchain
(While always getting synced_to_chain=true and synced_to_graph=true on lncli getinfo and active=true and synced=true on lncli neutrino status)

Then I switched to bitcoin core rpc and now everything is working. So this must be an issue with neutrino.

What's weird is that this setup ran flawlessly for many months and nothing has changed recently besides the upgrade to 0.15.2-beta

[sputn1ck]

Just for clarity, you were running bitcoin core yourself serving block filters and using them from your watch-only lnd instance and switching over to using bitcoind directly as the backend fixed it for you?

[data-retriever]

Just for clarity, you were running bitcoin core yourself serving block filters and using them from your watch-only lnd instance and switching over to using bitcoind directly as the backend fixed it for you?

Exactly. This is a kubernetes cluster, all the instances are of different pods communicating through the internal network. I control everything.

[guggero]

I took a quick look as well to make sure there's no new issue with remote signing setups.
This sounds very weird to me. I cannot really see how the chain backend would have an influence on the derivation paths. Is it possible the above error was just a temporary problem in general? Are you using some sort of load balancing and perhaps the watch-only node contacted/connected to the wrong remote signing instance?

[data-retriever]

I took a quick look as well to make sure there's no new issue with remote signing setups. This sounds very weird to me. I cannot really see how the chain backend would have an influence on the derivation paths. Is it possible the above error was just a temporary problem in general? Are you using some sort of load balancing and perhaps the watch-only node contacted/connected to the wrong remote signing instance?

There are no other signing instances on this cluster and the error really seemed to be persistent.

I tried to change back again to neutrino and the only error I'm able to get is the err=Synchronizing blockchain that happened before while trying to open a small channel. I don't have much on chain balance so I may try later with more funds to open a big channel.

So right now neutrino seems to be really broken. At least for channel opening purposes.

[guggero]

Could it be that you got the err=Synchronizing blockchain from the remote peer? Or did you try both times with the same remote peer?

[data-retriever]

Could it be that you got the err=Synchronizing blockchain from the remote peer? Or did you try both times with the same remote peer?

Both times the err=Synchronizing blockchain come while trying to open a channel to 1ML. Looking at the comments on

lnd/lnwire/error.go

Line 18 in b0e9442

// ErrSynchronizingChain is returned by a remote peer that receives a

it seems that yes, it could have come from the remote peer.

[nikicat]

Could it be that you got the err=Synchronizing blockchain from the remote peer? Or did you try both times with the same remote peer?

Both times the err=Synchronizing blockchain come while trying to open a channel to 1ML. Looking at the comments on

lnd/lnwire/error.go

Line 18 in b0e9442

// ErrSynchronizingChain is returned by a remote peer that receives a

it seems that yes, it could have come from the remote peer.

I can confirm, I've faced the same error trying to open channel to 1ML and debug logs says

2022-10-17 20:55:52.252 [ERR] RPCS: unable to open channel to NodeKey(0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266): received funding error from 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266: chan_id=1ecd05ec14ec1462f77665239fb53341f58a7c82f9b6e03f9856ec86ddef16f0, err=Synchronizing blockchain

[data-retriever]

It's happening again but now using the RPC:

Watch Only:

2022-10-27 16:57:10.864 [INF] FNDG: Recv'd fundingResponse for pending_id(1299ae4a6fb9d4c2d20b6c5f79b8513f245d63eb3456926fcfc8d44dc2c57c83)
2022-10-27 16:57:10.867 [ERR] FNDG: Unable to process contribution from &{{[42673605 2107050 10768122 29318847 38745622 56734640 50472538 59755664 26394517 2918186]} {[4150870 10742057 66321416 1250398 32948464 14909017 31841400 22939989 60274405 1463876]}}: unable to construct funding tx: error signing with remoteinstance: remote signer returned invalid key spend signature: signature too short with 0 bytes
2022-10-27 16:57:10.867 [INF] FNDG: Cancelling funding reservation for node_key=02b21ca992bf95e3f324302265ad86cec24f36166fd7afca44efa0809aaa8b25c5, chan_id=1299ae4a6fb9d4c2d20b6c5f79b8513f245d63eb3456926fcfc8d44dc2c57c83
2022-10-27 16:57:10.867 [ERR] RPCS: unable to open channel to NodeKey(02b21ca992bf95e3f324302265ad86cec24f36166fd7afca44efa0809aaa8b25c5): unable to construct funding tx: error signing with remoteinstance: remote signer returned invalid key spend signature: signature too short with 0 bytes
2022-10-27 16:57:10.867 [ERR] RPCS: [/lnrpc.Lightning/OpenChannel]: unable to construct funding tx: error signing with remoteinstance: remote signer returned invalid key spend signature: signature too short with 0 bytes

Signer:

2022-10-27 16:57:10.867 [WRN] BTWL: SignPsbt: Skipping input 0, derived public key 02663bf07bc2f47108f0c9e9aa63f0ef4d31fb3b44141b3d21321049133379d6f7 does not match bip32 derivation info public key 039b4c16e5f41a8ff887a7ab6ac6cb50667a1380a299a6e0fa55053f96a3af4c6a

Any ideas for a workaround?

[data-retriever]

Also I found out I can't move these coins. Same error

[guggero]

Can you compare the output of lncli wallet accounts list of both the watch-only and the signer node? The only difference should be the watch_only flag and the name of some accounts. But there should be the same number of accounts with exactly the same xpubs.

[data-retriever]

Yes, the only differences are the name, watch_only, external_key_count and internal_key_count (these counts are always zero on signer). The quantity and values of xpubs and derivation_paths are the same.

[data-retriever]

Also I found out I can't move these coins. Same error

I'm unable to use the total but I was able to transfer a fraction. My hunch is that the error happens when it tries to include a p2tr output. If it only uses a p2wpkh address it seems to work.

[guggero]

Interesting... Thanks for the info. I'll see if I can reproduce this.

[data-retriever]

I'm unable to use the total but I was able to transfer a fraction. My hunch is that the error happens when it tries to include a p2tr output. If it only uses a p2wpkh address it seems to work.

Another hunch is the following:
I've read somewhere that LND won't let you spend all your funds because it requires some reserved funds. So instead of telling you that you can't spend most of funds, because of this watch-only/signer segregation, it is hitting a bug on this corner case and the result is this weird error instead of the correct message.

[guggero]

Gave this another attempt at reproduction. Your hunch sounded very plausible but it seems like that is not the case. I tried with lnd v0.15.4-beta and I got a proper error ([lncli] rpc error: code = Unknown desc = not enough witness outputs to create funding transaction, need 0.00198425 BTC only have 0.001923 BTC available) when attempting to open a channel with a size that would leave insufficient balance for the anchor reserve.

Can you reproduce the issue reliably or does it only happen sometimes? Would you be willing to share a bit more info about your setup and your UTXOs (perhaps on Slack for more privacy)?

[data-retriever]

Gave this another attempt at reproduction. Your hunch sounded very plausible but it seems like that is not the case. I tried with lnd v0.15.4-beta and I got a proper error ([lncli] rpc error: code = Unknown desc = not enough witness outputs to create funding transaction, need 0.00198425 BTC only have 0.001923 BTC available) when attempting to open a channel with a size that would leave insufficient balance for the anchor reserve.

Can you reproduce the issue reliably or does it only happen sometimes? Would you be willing to share a bit more info about your setup and your UTXOs (perhaps on Slack for more privacy)?

Yes, it's very reproducible. I've joined the slack