[bug]: Not all transfer proofs generated

[youngzhenhao]

Background

Only a portion of proofs is generated locally after SendAsset to 12 addresses, this is also true after 6 blocks have been generated in bitcoin mainnet.

Bitcoin of asset's this UTXO has actually been spent, but asset's this UTXO is still existing, can not be used, and no output index 0's asset UTXO available.

Your environment

litd: 0.13.3-alpha
lnd: v0.18.2-beta
tapd: 0.4.1-alpha
network: mainnet
system: Linux 4.19.157
lnd use neutrino

• any other relevant environment details

Steps to reproduce

1. Create a new instance of the lightning-terminal daemon and run it.
2. Unlock lnd and wait it synced to chain.
3. Call SendAsset of taprpc to send assets to 12 addresses are from different tapd.
4. Wait several new block generates.
5. Check proof files of transfer in directory.

Expected behavior

All proof file with output index 0 to 12 are generated in the {tapddir}\data\mainnet\proofs\{asset_id}\.

Actual behavior

Only proof file with output index 0 was generated in the {tapddir}\data\mainnet\proofs\{asset_id}.

Logs here

Proof file with index 0 here

The same thing happened two weeks ago, in one of the previous transfers, until now, only proof files with output indexes 0,1,2 were generated after SendAsset to other 12 addresses.

Expectation

1. What would cause only a portion of proof files be generated?
2. How to make asset's this invalid UTXO usable?
3. How to use the existing data to regenerate all the proof files?

[guggero]

I see this:

2024-10-31 09:12:07.415 [ERR] FRTR: Error evaluating state (SendStateStoreProofs): error verifying proof: error verifying proofs: invalid exclusion proof: invalid taproot proof: derived_keys=([]asset.SerializedKey) (len=2 cap=2) {
 (asset.SerializedKey) (len=33 cap=33) {
  00000000  02 0a f8 5f f8 96 d8 2a  0e ea 2f 33 1a 57 ed cc  |..._...*../3.W..|
  00000010  2f 76 08 36 01 1f 65 81  70 71 01 44 ef a6 67 ba  |/v.6..e.pq.D..g.|
  00000020  9a                                                |.|
 },
 (asset.SerializedKey) (len=33 cap=33) {
  00000000  02 3f 8a 95 bc 22 76 e5  39 ff a8 bb 38 f4 9d 47  |.?..."v.9...8..G|
  00000010  1d 02 c2 c1 ad 19 0b 55  6e fc 3e 6f ae e3 e8 e9  |.......Un.>o....|
  00000020  8e                                                |.|
 }
}
, expected_key=1082ad76c8eab2bcd3c515d87b3d3d2fc83eb8395664a91d1cb852d52182d267

So some proof wasn't created correctly... I'll investigate.

[guggero]

@youngzhenhao I did take a closer look and it seems like the one proof (for index 0) that was generated is valid. So something must be up with index 1 and later.
But those proofs aren't on the disk yet, only in the SQLite database.
Do you think it would be possible for you to send me a copy of your database?
You can either reach me by e-mail (see my GitHub profile), keybase or our Slack server.

If that's problematic, I can also send you a query you can run to just extract some of the invalid proofs from the DB. Let me know what works best for you.

[youngzhenhao]

@guggero Sure, db file here

[guggero]

Thank you for the DB, that was really helpful.

So I think you ran into an edge case which should be disallowed (but apparently the check for this hasn't been implemented). The problem is that you have multiple transfers with different anchor transaction output indexes that use the same asset ID and same script key (probably by using the same Taproot Asset address multiple times):

sqlite> select t.output_id, t.amount, hex(s.tweaked_script_key) from asset_transfer_outputs t left join script_keys s on s.script_key_id = t.script_key;
output_id|amount|hex(s.tweaked_script_key)

1|4150000|021A79C9B213E0EB1ADDABA079AE9119501EBE017D59ADE215D783E1F66CC0C68E
2|100000|02CC7BD3D287ED641E816C38B016DB09C77A922FBC2FFFD65E607B3DAC9C3C8D15
3|100000|02CC7BD3D287ED641E816C38B016DB09C77A922FBC2FFFD65E607B3DAC9C3C8D15
4|100000|028288030EB0C0EA9DBAFDD13D196EF592F7E302F780AF1B2E10BD4E18D2D2439E
5|100000|02DDB258B82B35AD69A5DD55069564CE2A0DA4D93D38ACE2749F190FA7DA4B9CB1
6|100000|021DD642C5A2BF8A5F70190968A66FE5C526A784F24699968F71B020026B8351A2
7|50000|0234E9EC3BE713648DFE8AA58ADBB9FAB48AE3385A4446E475903D9EF49E4A9FD0
8|50000|029EAE70BB07B2346AE2BCFFEC6450EFE9E837D91F3F49297D36C101E2E75F7A27
9|50000|0228569DD56DA580900234924A9FA8C8A6AB8D48F2C9A854D713C19E4BFABAF025
10|50000|02481253F97E4A2DC3CB120139F788C21D6DAB63E66DC6B8495F023F89CBB0D0E1
11|50000|02FBAAC2F9CC6BBE44A8DB1284AE3C4F1DB488D518718403BE5FC4DF683951F3C7
12|50000|02FBAAC2F9CC6BBE44A8DB1284AE3C4F1DB488D518718403BE5FC4DF683951F3C7
13|50000|0276CFF40ACA9FD9A6C2CEF28294DDBB237E3815155C5F03D6DC3DD21890107232

This breaks the exclusion proofs... So we'll need to add a check for that.

Are these new assets? Can you re-issue them? Because I'm not sure these proofs can be corrected, since they break a fundamental property of the protocol.

[youngzhenhao]

I did not check the same asset address, that seems to be the problem.
Proof for index 0 that was generated is valid, does it mean that this portion of asset can be restored to valid asset?

[guggero]

Yes, I think some of assets can be restored, while others might be lost due to the error... I created a fix here: #1181, but unfortunately that will just make sure it doesn't happen again.

Do you want me to look into manually fixing parts of the proofs? Or can you somehow re-issue the assets? It's not clear to me whether you received them or minted them yourself.

[youngzhenhao]

It is received assets, and here's a proof of receipt, but their UTXO has been spent. Is it possible to restore these assets?

[guggero]

I'll try my best to restore some of the assets. I'll get back to you soon.

[guggero]

Okay, I was able to create valid proofs from the template for all but 4 of the outputs. So that means 300'000 of the units are effectively burned (2*100000 + 2*50000 where the same script keys were used)...
Really sorry about that, but I don't think there's anything that can be done about those. But that means the rest of the units (4'700'000) units can be rescued by doing the following:

1. Make sure you stop tapd (whole litd if you're running everything in one process).
2. Create a backup of your tapd database file (/data/user/0/io.bitlong/files/.tapd/data/mainnet/tapd.db).
3. Execute the following queries in SQLite (assuming you have sqlite3 installed, you run sqlite /data/user/0/io.bitlong/files/.tapd/data/mainnet/tapd.db then paste the following commands in there):

-- delete the four incorrect outputs.
delete from asset_transfer_outputs where script_key = (select script_key_id from script_keys where upper(hex(tweaked_script_key)) = '02CC7BD3D287ED641E816C38B016DB09C77A922FBC2FFFD65E607B3DAC9C3C8D15');
delete from asset_transfer_outputs where script_key = (select script_key_id from script_keys where upper(hex(tweaked_script_key)) = '02FBAAC2F9CC6BBE44A8DB1284AE3C4F1DB488D518718403BE5FC4DF683951F3C7');

4. Exit the sqlite3 console by typing .quit and pressing <enter>.
5. Start tapd (litd) again.
6. Observe the log. The assets should now be correctly sent/created.

If you have questions or get an error during any of the steps, please let me know!

[youngzhenhao]

It does work. Thanks for all your help!