Bitcoin backend generalization #3488

darosior · 2020-02-06T16:36:59Z

This standardizes our requests for Bitcoin data as a set of commands plugins can register, and makes our bitcoind interaction through bitcoin-cli the default plugin, which registers all of these commands.

This makes it possible to not rely on a bitcoin-core backend, which can help for instance for mobile integration (cf #3484).

See #3354 for some design decisions, but here is a summary:

Separated plugins can register different calls, which allows plugins specialization (credits to @cdecker)
Uses JSON-RPC commands instead of hooks (see Bitcoin backend plugins planning #3354).
Needs 5 commands to be registered by our plugin(s):
- getchaininfo - called at startup to get infos about our network, and about the block chain.
- getrawblockatheight - get the block at that height in hex format, along with its id.
- sendrawtransaction - self explanatory
- gettxout - get the amount and the scriptPubkey of this txo.
- getfeerate - get the feerate in btc/kVB, because we love vbytes for fee-sensitive Bitcoin applications. Params are bitcoind-style.

This is based on #3480 as it uses its higher-level JSONRPC helpers for plugins.

TODO

bcli-specific functional tests
~~interface documentation, for helping Bitcoin L1 developers to develop a plugin to serve as a C-lightning Bitcoin backend :)~~

diff --git a/lightningd/jsonrpc.c b/lightningd/jsonrpc.c
index d2f896904..15a2df66a 100644
--- a/lightningd/jsonrpc.c
+++ b/lightningd/jsonrpc.c
@@ -984,6 +984,13 @@ static struct io_plan *jcon_connected(struct io_conn *conn,
 static struct io_plan *incoming_jcon_connected(struct io_conn *conn,
                                               struct lightningd *ld)
 {
+       /* The topology is required to be ready by most of the JSONRPC calls,
+        * even the simplest ones (`getinfo` queries the tip). However we
+        * want to serve the RPC socket before we setup the topology (mostly
+        * for plugins), so accept incoming requests but don't treat them yet. */
+       if (!ld->topology->ready)
+               return io_wait(conn, ld->jsonrpc, incoming_jcon_connected, ld);
+
        /* Lifetime of JSON conn is limited to fd connect time. */
        return jcon_connected(notleak(conn), ld);
 }
diff --git a/lightningd/lightningd.c b/lightningd/lightningd.c
index 0874bee54..594383fff 100644
--- a/lightningd/lightningd.c
+++ b/lightningd/lightningd.c
@@ -855,6 +855,11 @@ int main(int argc, char *argv[])
        setup_topology(ld->topology, ld->timers,
                       min_blockheight, max_blockheight);
 
+       /* Now that the topology is set up, we can treat incoming JSONRPC
+        * requests ! */
+       assert(ld->topology->ready);
+       io_wake(ld->jsonrpc);
+

But we need to start plugins in order to setup the topology.... And plugins use libplugin which requires the JSONRPC to answer a request (listconfigs in order to set the allow_deprecated_apis global) before responding to init.

So either we get rid of this call in handle_init, which I believe is not acceptable, or we setup the plugins needed to setup the topology before the RPC socket (and other current plugins), or we special-case the bcli plugin to not require this call to listconfigs, and synchronously wait for it to be configured in setup_topology().. Which is about the same situation as now but in a less explicit way.

I think the right thing would be to make a distinction between plugins which rely upon lightningd (and thus need the RPC socket, are setup after the topology, etc..), and plugins which lightningd relies upon.
That said, with this PR the only plugin(s) lightningd relies on are bitcoind-ones, so it's special-cased into lightningd/bitcoind(and this is cleaner than all my attempts to create the socket before starting these plugins).

darosior · 2020-02-10T15:24:41Z

Rebased, renamed gettxout to getutxout, removed the requirement for errmsg to be present if sendrawtransaction succeeds, and generalized find_plugin_for_command().

The doc is ready but I wanted @cdecker feedback before as he had a reserve about documenting too early.

rustyrussell · 2020-02-11T04:23:27Z

Yeah, OK, let's do this for now then. Nasty stuff though :(

rustyrussell · 2020-02-11T06:10:50Z

Trivial rebase. Still needs a Changelog entry though!

We don't take the callback result into account, so it can better be void. Having a general callback parameter is handy, because for bcli we want to pass it the struct bcli.

Most is taken from lightningd/bitcoind and adapted. This currently exposes 5 commands: - `getchaininfo`, currently called at startup to check the network and whether we are on IBD. - `getrawblockbyheight`, which basically does the `getblockhash` + `getblock` trick. - `getfeerate` - `sendrawtransaction` - `getutxout`, used to gather infos about an output and currently used by `getfilteredblock` in `lightningd/bitcoind`.

This is also taken and adapted from lightningd/bitcoind. The call to 'getblockchaininfo' is replaced by 'echo' as we don't make use of the result and the former can sometimes be slow (e.g. on IBD).

We are going to initialize a plugin before its creation, so log as UNUSUAL instead. Also, `pay` and `fundchannel` inits are using rpc_delve(), so we need to io_new_conn() (which sets the socket as non blocking) after calling the plugin's init.

Exit early if we won't be able to fully communicate with our Bitcoin backend.

We need our Bitcoin backend to be initialized, but the plugins have not yet been started at this point.

The Bitcoin backend is generalized through the Bitcoin plugin and this was specific to core.

This adds `getchaininfo` and `getrawblockbyheight` handling lightningd-side, and use them in setup_topology(). We then remove legacy bitcoind_getblockcount() (we already get the count in `getchaininfo`), bitcoind_getblockchaininfo() (it was only used in setup_topology()), and wait_for_bitcoind() (this was specific to bitcoin-core and we assume our Bitcoin backend to be functional if the plugin responds to `init`).

This restrains the informations we get about how the sending went to an errmsg as we cant rely on bitcoin-cli specific output nor its exit code.

This avoids the getblockhash+getblock, and more importantly that was the last functionality making use of bitcoind_getrawblock() and bitcoin_getblockhash(), so we can also get rid of them.

And remove bitcoin-cli interaction code, now unused.

Changelog-Added: pluggable backends for Bitcoin data queries, default still bitcoind (using bitcoin-cli).

We need our Bitcoin backend to be ready to get infos about some utxos

For bitcoind_fail_first: We only ever send `getblock` if we got a successful block hash from `getblockhash`, and if we can't get the block in that case it means our Bitcoin backend is faulty and we shouldnt continue. So, mock `getblockhash` instead, which is authorized to spuriously fail. For both bitcoind_fail_first and bitcoind_failure: Adapt the logs.

darosior · 2020-02-11T10:57:15Z

Rebased and changeloged

cdecker

Still running some more tests (especially against TEST_DB_PROVIDER=postgres and TEST_NETWORK=liquid-regtest) to make sure this is stable in all configurations. I'll add fixups if necessary.

plugins/libplugin.h

plugins/bcli.c

cdecker · 2020-02-11T15:46:45Z

lightningd/bitcoind.c

+		db_begin_transaction(call->bitcoind->ld->wallet->db);
+		call->cb(call->bitcoind, NULL, NULL, call->cb_arg);
+		db_commit_transaction(call->bitcoind->ld->wallet->db);


Shouldn't this already be wrapped in a DB transaction by virtue of being part of the io_loop?

I don't see why it should: lightningd/io_loop_with_timers only begins a db_transaction is a timer expires (out of the io_loop).

Before this bcli_finished would make a db transaction for each callback

lightning/lightningd/bitcoind.c

Lines 224 to 226 in 86c28b2

db_begin_transaction(bitcoind->ld->wallet->db);

ok = bcli->process(bcli);

db_commit_transaction(bitcoind->ld->wallet->db);

So process_getblock was also inside a db_transaction

cdecker · 2020-02-11T15:46:56Z

lightningd/bitcoind.c

+	db_begin_transaction(call->bitcoind->ld->wallet->db);
+	call->cb(call->bitcoind, &blkid, blk, call->cb_arg);
+	db_commit_transaction(call->bitcoind->ld->wallet->db);


lightningd/chaintopology.c

cdecker · 2020-02-11T16:16:17Z

lightningd/bitcoind.c

+	struct jsonrpc_request *req;
+
+	req = jsonrpc_request_start(bitcoind, "getfeerate",
+				    bitcoind->log, getfeerate_callback,
+				    call);
+	json_add_num(req->stream, "blocks", call->blocks[call->i]);
+	json_add_string(req->stream, "mode", call->estmode[call->i]);
+	jsonrpc_request_end(req);
+	plugin_request_send(strmap_get(&bitcoind->pluginsmap,
+				       "getfeerate"), req);


Would be nice if we could bundle these and send all the block/mode combinations to the plugin at once. That'd allow us to have a single all-or-nothing handler and not having to loop in lightningd.

Indeed. As a cleanup or before the release ?

Post. Too late for release now.

Opened #3508 to track this.

lightningd/bitcoind.c

tests/test_plugin.py

darosior · 2020-02-11T19:22:15Z

Spurious Travis error

cdecker · 2020-02-11T20:35:11Z

Let's see if @bitcoin-bot can recogniZe clean squashes yet 😉

ACK 93596e3

rustyrussell · 2020-02-11T23:03:34Z

Apparently not :( Mind you, GH doesn't give anything for that 'force-push' link either...

rustyrussell · 2020-02-12T00:56:29Z

Squashed fixup:

Ack f7b48ed

cdecker · 2020-02-12T01:14:14Z

Interestingly the web frontend seems to recognize them as identical (green background for the comparison between the last two versions). I'll look into it

darosior force-pushed the bitcoin_backend_plugin branch 3 times, most recently from 1eb5490 to 3cf1571 Compare February 6, 2020 22:54

darosior force-pushed the bitcoin_backend_plugin branch from 3cf1571 to 083d123 Compare February 7, 2020 12:30

darosior force-pushed the bitcoin_backend_plugin branch from 083d123 to 8fe0233 Compare February 8, 2020 20:22

darosior marked this pull request as ready for review February 8, 2020 21:11

darosior requested a review from cdecker as a code owner February 8, 2020 21:11

rustyrussell added this to the 0.8.1 milestone Feb 10, 2020

rustyrussell reviewed Feb 10, 2020

View reviewed changes

darosior force-pushed the bitcoin_backend_plugin branch from 8fe0233 to 07dc09f Compare February 10, 2020 15:22

darosior force-pushed the bitcoin_backend_plugin branch 2 times, most recently from 35e7bc8 to 3b463f9 Compare February 10, 2020 19:10

rustyrussell force-pushed the bitcoin_backend_plugin branch from 3b463f9 to 8d8fdee Compare February 11, 2020 06:10

darosior force-pushed the bitcoin_backend_plugin branch from 8d8fdee to f1d4322 Compare February 11, 2020 10:53

darosior added 7 commits February 11, 2020 11:56

libplugin: add a 'still_pending' helper

fd6172a

libplugin: generalize the plugin_timer callback type

22780f8

We don't take the callback result into account, so it can better be void. Having a general callback parameter is handy, because for bcli we want to pass it the struct bcli.

plugins/bcli: wait for bitcoind to be warmed up at init

44d408c

This is also taken and adapted from lightningd/bitcoind. The call to 'getblockchaininfo' is replaced by 'echo' as we don't make use of the result and the former can sometimes be slow (e.g. on IBD).

plugins/bcli: register Bitcoin-related options

8261ff1

chaintopology: check bitcoin plugin commands at startup

fbc92b0

Exit early if we won't be able to fully communicate with our Bitcoin backend.

darosior added 12 commits February 11, 2020 11:56

bitcoind: initialize Bitcoin-backend plugin early

c21d902

We need our Bitcoin backend to be initialized, but the plugins have not yet been started at this point.

chaintopology: dont check bitcoin-core version at startup

ed2a714

The Bitcoin backend is generalized through the Bitcoin plugin and this was specific to core.

lightningd/chaintopology: use plugin for tip polling

0ff12ce

lightningd/bitcoind: use the Bitcoin plugin to send transactions

6aa9233

This restrains the informations we get about how the sending went to an errmsg as we cant rely on bitcoin-cli specific output nor its exit code.

lightningd/bitcoind: use getrawblockatheight for getfilteredblock

900af43

This avoids the getblockhash+getblock, and more importantly that was the last functionality making use of bitcoind_getrawblock() and bitcoin_getblockhash(), so we can also get rid of them.

lightningd/bitcoind: use the Bitcoin plugin for getutxout

fa7de2c

lightningd/bitcoind: use the Bitcoin plugin for fee estimates

a9239df

And remove bitcoin-cli interaction code, now unused.

lightningd/bitcoind: remove all bitcoin-cli specific code

6ee5fe4

Changelog-Added: pluggable backends for Bitcoin data queries, default still bitcoind (using bitcoin-cli).

lightningd: clean utxos after the topology is setup

c3b175d

We need our Bitcoin backend to be ready to get infos about some utxos

pytest: fixup flaky test_closing_specified_destination

765033d

darosior force-pushed the bitcoin_backend_plugin branch from f1d4322 to 95563a7 Compare February 11, 2020 10:57

rustyrussell added bitcoind Optech Make Me Famous! Look! Look! Look! COOL NEW FEATURE! labels Feb 11, 2020

cdecker mentioned this pull request Feb 11, 2020

cleanup: Use typesafe_cb in plugin_timer #3504

Closed

cdecker reviewed Feb 11, 2020

View reviewed changes

pytest: test Bitcoin plugin registration and the bcli plugin

f7b48ed

rustyrussell force-pushed the bitcoin_backend_plugin branch from 93596e3 to f7b48ed Compare February 11, 2020 23:02

rustyrussell merged commit a47fd8c into ElementsProject:master Feb 12, 2020

darosior deleted the bitcoin_backend_plugin branch February 12, 2020 09:09

darosior mentioned this pull request Feb 12, 2020

Batching feerate requests for our plugin backend #3508

Closed

andrerfneves mentioned this pull request Mar 3, 2020

[WIP] Add post for socratic 102 BitDevsNYC/BitDevsNYC.github.io#50

Merged

kirelagin mentioned this pull request Mar 30, 2020

bad response to getutxout (bad sats amount) #3610

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bitcoin backend generalization #3488

Bitcoin backend generalization #3488

darosior commented Feb 6, 2020 •

edited

Loading

darosior commented Feb 6, 2020

ZmnSCPxj commented Feb 7, 2020

darosior commented Feb 7, 2020 •

edited

Loading

darosior commented Feb 7, 2020

darosior commented Feb 8, 2020

rustyrussell left a comment

darosior commented Feb 10, 2020 •

edited

Loading

darosior commented Feb 10, 2020

rustyrussell commented Feb 11, 2020

rustyrussell commented Feb 11, 2020

darosior commented Feb 11, 2020

cdecker left a comment

cdecker Feb 11, 2020

darosior Feb 11, 2020 •

edited

Loading

cdecker Feb 11, 2020

cdecker Feb 11, 2020

darosior Feb 11, 2020

rustyrussell Feb 11, 2020

darosior Feb 12, 2020

darosior commented Feb 11, 2020

cdecker commented Feb 11, 2020 •

edited

Loading

rustyrussell commented Feb 11, 2020

rustyrussell commented Feb 12, 2020

cdecker commented Feb 12, 2020

	db_begin_transaction(bitcoind->ld->wallet->db);
	ok = bcli->process(bcli);
	db_commit_transaction(bitcoind->ld->wallet->db);

Bitcoin backend generalization #3488

Bitcoin backend generalization #3488

Conversation

darosior commented Feb 6, 2020 • edited Loading

TODO

Next

darosior commented Feb 6, 2020

ZmnSCPxj commented Feb 7, 2020

darosior commented Feb 7, 2020 • edited Loading

darosior commented Feb 7, 2020

darosior commented Feb 8, 2020

rustyrussell left a comment

Choose a reason for hiding this comment

darosior commented Feb 10, 2020 • edited Loading

darosior commented Feb 10, 2020

rustyrussell commented Feb 11, 2020

rustyrussell commented Feb 11, 2020

darosior commented Feb 11, 2020

cdecker left a comment

Choose a reason for hiding this comment

cdecker Feb 11, 2020

Choose a reason for hiding this comment

darosior Feb 11, 2020 • edited Loading

Choose a reason for hiding this comment

cdecker Feb 11, 2020

Choose a reason for hiding this comment

cdecker Feb 11, 2020

Choose a reason for hiding this comment

darosior Feb 11, 2020

Choose a reason for hiding this comment

rustyrussell Feb 11, 2020

Choose a reason for hiding this comment

darosior Feb 12, 2020

Choose a reason for hiding this comment

darosior commented Feb 11, 2020

cdecker commented Feb 11, 2020 • edited Loading

rustyrussell commented Feb 11, 2020

rustyrussell commented Feb 12, 2020

cdecker commented Feb 12, 2020

darosior commented Feb 6, 2020 •

edited

Loading

darosior commented Feb 7, 2020 •

edited

Loading

darosior commented Feb 10, 2020 •

edited

Loading

darosior Feb 11, 2020 •

edited

Loading

cdecker commented Feb 11, 2020 •

edited

Loading