Add a electrum syncing example

[rajarshimaitra]

This PR adds an electrum syncing example.. Fixes #72

The electrum sync is done in two steps.

• wallet_txid_scan returns a candidate KeychainScan with TxGraph data.
• The candidate is compared with tracker's SparseChain, and list of new tx data to fetch is derived from the ChangeSet.
• New transactions are fetched in batch and added into the KeychainScan update.
• Apply the final KeychainScan on tracker and DB.

[codecov-commenter]

Codecov Report

Merging #77 (6051596) into master (fbc88da) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #77   +/-   ##
=======================================
  Coverage   59.39%   59.39%           
=======================================
  Files           8        8           
  Lines         266      266           
=======================================
  Hits          158      158           
  Misses        108      108           

Impacted Files	Coverage Δ
bdk_chain/src/chain_graph.rs	0.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[rajarshimaitra]

Restructured the commit to handle #72 requirements.

• Make Chaingraph consistent, by adding few assertions at the apply_update() API.
• Make the electrum sync in two steps. Get all the txids, find new ones, fetch tx data, apply update..

[rajarshimaitra]

Me and @evanlinjin tend to disagree on the approach of the consistency assertion.. While to me it seems the best place to apply it is the update API, Evan thinks there could be a better approach.. Opening this up for discussion and reviews..

[rajarshimaitra]

Rebased on current master, and updated the example syncing with recent invariants around ChainGraph. Similar to RPC syncing, the first stage returns a SparseChain. The nex to_fetch_tx id set is derived using determine_changeset between the returned sparsechain and existing sparaechain within the tracker..

[evanlinjin]

Good work so far!

[rajarshimaitra]

Updated with the above suggestions.

• No default reorg depth mechanism. Takes in a ref of existing chain, and inserts checkpoint until the last common check point is found. And then push the current tip into update. In case of reorg, all checkpoints after the last common block is added into the update structure, and while SparseChain::apply_update new checkpoints will replace old checkpoints. No error is thrown.

• No default stop gap.

• A new function added in the main to collect all the common code between Scan and Sync.

• Transaction fetching is done opportunistically, for only txids that we haven't seen before. For transactions that only changed in Index tx data is fetched from existing TxGraph of the tracker.

Q: What to do when we only have TxNode::Partial in graph, and index changes of that transaction? 2 possible options.

• We can fetch such transactions again from the blockchain before applying update.
• Have mechanism to insert_tx() with TXNode::Partial

[LLFourn]

Q: What to do when we only have TxNode::Partial in graph, and index changes of that transaction? 2 possible options.

* We can fetch such transactions again from the blockchain before applying update.

* Have mechanism to `insert_tx()` with `TXNode::Partial`

Haven't reviewed just comment: we don't keep the index for TxNode::Partial.

[rajarshimaitra]

Haven't reviewed just comment: we don't keep the index for TxNode::Partial.

Done..

[LLFourn]

Thanks @rajarshimaitra. Looks great. A few minor requests to finish this off. The main thing that's missing is to protect against re-org during sync.

[rajarshimaitra]

It seems like there is still a bit of duplicate code. Isn't it the case that everything after getting the sparse chain and applying the last active indexes will be the same?

Yes.. I tried to add this into a separate function but that's making the code more complicated, because many trait bounds needs to be satisfied for the KeychainStore::append_changeset(), which are handled in bdk_cli.. It still could be done, but after doing it it made the code more complex than it needs to be, and thought would be detriment for readers reading this as example, and its only removing 3 lines. So kept it as it is for now..

[LLFourn]

I just took another pass at this. The list of things I've proposed to fix here is final now I think.

[rajarshimaitra]

Thanks @LLFourn for the comments.. Updated as suggested and used the new API.. modulo #77 (comment).. Code looks much simpler..

[LLFourn]

Look great @rajarshimaitra. Main things:

• use batch calls to electrum
• Intead of matching every error and then throwing a new anyhow error just use anyhow::Context::context
• Remove function complete_update in favor of just getting the new_sparsechain from the match arms and doing the update from there like what @evanlinjin does here: https://github.com/LLFourn/bdk_core_staging/pull/107/files#diff-6f57072cac1464050942f5bf972a9a8f76cb82a151f93f5b5d16cab1515f925aR74

[rajarshimaitra]

@LLFourn updated with the suggestions, module #77 (comment).

Let me know if this looks good and you want the the commit history squashed..

[rajarshimaitra]

Sqaushed and rebased on master..

[LLFourn]

I just did some work to get this working:

• break at the right time in the loop for stop gap
• add signet electrum server
• print out scnning progress
• Make client configurable
• Stop returning the sparse chain from inflate_changeset because it
  makes it harder to ? the error and you can just clone it if you still
  need it on failure.

Check it out @rajarshimaitra.

This is working and probably ready to merge with final review from @evanlinjin.

[rajarshimaitra]

ACK 3531907

Thanks, @LLFourn for the fixes.. Code looks much better..

Added a nit fix..

Testing out the full thing..

[evanlinjin]

@rajarshimaitra You forced-pushed away the changes...

[rajarshimaitra]

Holy shit..

[evanlinjin]

@rajarshimaitra Fixed.

[rajarshimaitra]

Sorry dude.. Intended to just push a nit.. Fat fingered..

[evanlinjin]

ACK 3531907

Just some nits