lnd v0.9.0-beta
This marks the first major release in the v0.9.x series, as well as the first lnd release of the new decade! This release contains several new features, optimizations, new protocol-level features surfaced on the RPC layer, cross-implementation protocol compatibility fixes, and as usual a ton of bug fixes. Shouts out to all the contributors (over 30+ for this release!) as well as all those that reported bugs and tested, that helped make this new release happen.
Database Migrations
This release contains a single migration which modifies the on-disk representation of the invoices to be more flexible by utilizing the TLV serialization defined in the BOLT specifications. This change allows us to easily add more information to the invoices, which will be especially useful as we finalize our support for AMP within the daemon. The migration should look something like this upon initial start up:
2020-01-09 16:35:39.586 [INF] LTND: Version: 0.9.0-beta commit=v0.9.0-beta, build=development, logging=default
2020-01-09 16:35:39.587 [INF] LTND: Active chain: Bitcoin (network=simnet)
2020-01-09 16:35:39.589 [INF] CHDB: Checking for schema update: latest_version=12, db_version=11
2020-01-09 16:35:39.589 [INF] CHDB: Performing database schema migration
2020-01-09 16:35:39.589 [INF] CHDB: Applying migration #12
2020-01-09 16:35:39.589 [INF] CHDB: Migrating invoice bodies to TLV, adding payment addresses and feature vectors.
2020-01-09 16:35:39.589 [INF] CHDB: Migration to TLV invoice bodies, payment address, and features complete!
Verifying the Release
In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:
curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import
Once you have the required PGP keys, you can verify the release (assuming manifest-v0.9.0-beta.txt and manifest-v0.9.0-beta.txt.sig are in the current directory) with:
gpg --verify manifest-v0.9.0-beta.txt.sig
You should see the following if the verification was successful:
gpg: assuming signed data in 'manifest-v0.9.0-beta.txt'
gpg: Signature made Wed Jan 22 09:02:56 2020 PST
gpg: using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <[email protected]>" [ultimate]
That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.
Verifying the Release Binaries
Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.6, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.
Finally, you can also verify the tag itself with the following command:
git verify-tag v0.9.0-beta
Building the Contained Release
Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.9.0-beta.tar.gz are in the current directory, follow these steps:
tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.9.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta" ./cmd/lncli
The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.
Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:
LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh
⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️
Release Notes
Macaroon Bakery 🍪
A new custom macaroon bakery is now available through gRPC (BakeMacaroon) or the command line (lncli bakemacaroon)! The bakery allows users to mint macaroons with their own custom set of read/write permissions for use cases where the three pre-defined default macaroons (admin, readonly and invoice) are not enough. The documentation has also been updated to describe the process.
Note: The bakery requires a new permission that the existing admin.macaroon does not contain. Follow the steps described in the documentation to re-generate the admin macaroon.
New Payment Features
Multi-Path Payments
This release includes full support for receiving Multi-Path Payments as defined in BOLT 04, which is the first flavor of AMP rolling out on the network. MPP allows an invoice to be settled by multiple concurrent HTLCs, each of which carries a portion of the total payment.
With receiving support, MPP-aware senders can make better use of the payee's total inbound capacity in case there is no single channel that can support the payment amount. From a UX perspective, this allows wallet UIs to begin exploring a true, unified receiving capacity since the sender is able to utilize all of the receiver's channels for a single payment. As more MPP-aware senders come on to the network, 0.9 nodes will be able to accept payments without upgrading their software. We are currently targeting support for sending MPP in 0.10.
At a high-level, MPP is implemented via an additional TLV record in the final hop's onion blob, containing a random payment_secret and total_amt_msat being sent. Each HTLC carries this additional record in its onion blob, allowing the receiver to identify and settle an invoice with multiple HTLCs. Invoices will track all HTLCs in the set that are used to settle an invoices, as can be seen in this stripped down invoice settled by two HTLCs:
{
"memo": "multi path payment",
"r_preimage": "73d8d926c13579b0b1decf6b808a2bbf7db59073999c5d0a14d4dcc9c4225a49",
"r_hash": "a3309015130da7cc28f529191a3d0aeb06edfedfd58dac1dfde8b1fbd4eadc91",
"value_msat": "1000000",
"amt_paid_msat": "1000000",
"state": "SETTLED",
"htlcs": [
{
"chan_id": "571746046509056",
"htlc_index": "4",
"amt_msat": "500000",
"accept_height": 525,
"accept_time": "1579112858",
"resolve_time": "1579112860",
"expiry_height": 565,
"state": "SETTLED",
"custom_records": {
},
"mpp_total_amt_msat": "1000000"
},
{
"chan_id": "571746046509056",
"htlc_index": "5",
"amt_msat": "500000",
"accept_height": 525,
"accept_time": "1579112654",
"resolve_time": "1579112654",
"expiry_height": 565,
"state": "SETTLED",
"custom_records": {
},
"mpp_total_amt_msat": "1000000"
}
],
}
Note that amt_msat paid by each HTLC sums both to the mpp_total_amt_msat value included by the sender, as well as amt_paid_msat displaying the total amount received.
The rpc responses for ListPayments, routerrpc.SendPayment, and routerrpc.TrackPayment have also been extended with an htlcs field that exposes additional data about individual HTLC attempts. For now these will only show the last attempted HTLC, but will be populated with a full log of all HTLC attempts sent once MPP sending is fully integrated. This will allow us to display multiple successful HTLCs, as well as failed attempts. The latter will useful in allowing senders to inspect failed routes, and glean insights about why certain attempts failed.
A stripped down example of the new payment responses:
{
"payment_hash": "8eedb4bcd024f7f46fdf60b57e54dba60e19cb2c8934ec5b6fbb7f5efb58b29b",
"payment_preimage": "cf6b808a2bbf7db59073973d8d926c13579b0b1de99c5d0a14d4dcc9c4225a49",
"creation_time_ns": "1578620428000000000",
"path": [
"03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f",
"0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5"
],
"value_msat": "1000",
"fee_msat": "0",
"status": "SUCCEEDED",
"htlcs": [
{
"status": "SUCCEEDED",
"route": {
"total_time_lock": 612315,
"total_fees_msat": "0",
"total_amt_msat": "1000",
"hops": [
{
"chan_id": "612270746636810592",
"chan_capacity": "5000000",
"fee_msat": "0",
"amt_to_forward_msat": "1000",
"expiry": 612171,
"pub_key": "03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f",
"tlv_payload": true,
"mpp_record": null,
"custom_records": {
}
},
{
"chan_id": "612270746636810592",
"chan_capacity": "5000000",
"expiry": 612171,
"fee_msat": "0",
"amt_to_forward_msat": "1000",
"pub_key": "0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5",
"tlv_payload": true,
"mpp_record": {
"payment_addr": "f91731b1e2227cc26137d2a2c0029ceef41ac8c481c8d3cc61510bf76ff27cbe",
"total_amt_msat": "1000"
},
"custom_records": {
}
}
]
},
}
]
}
Here we can see that each HTLC attempt displays it's full route, including the fees paid and timelocks incurred at each hop. MPP payments can be identified by the presences of an mpp_record attached to the final hop, which containing the new fields described above. If this field is not present, it indicates that payment was made using the legacy payment type.
Note: The path field has been deprecated in favor of extracting the pubkeys from the route directly. creation_date has also been deprecated and will replaced by creation_time_ns. As seen above, legacy creation_dates are converted into nanoseconds for display, but still only have one second resolution.
Weakness Addressed by MPP
MPP is more robust against active probing of the receiver, since the sender must know the invoice's payment_secret. This value serves as a one-time authentication token, further restricting the set of individuals who can settle a particular invoice. In particular, invoices can now only be settled by those who have seen it, rather than only needing to know the correct payment hash which is transmitted in clear over the network.
MPP also offers increased safety to situations involving over payment of invoices, most notably zero-value invoices used for tipping or donations. With the legacy protocol, it is possible to snipe the difference between the invoice amount and total amount carried by the HTLC, allowing intermediaries to take the difference as profit in addition to fees earned. MPP payments resolve this by privately communicating the sender's intended payment amount in the onion blob (via total_amt_msat) to the receiver. Since intermediaries will not know the payment_secret, sniping is prevented even if the intermediary can construct a valid HTLC with a smaller total_amt_msat.
Single-Shot MPP Payments by Default
Although 0.9 does not support sending true MPP payments, it does use the new MPP protocol as the default mechanism for single-shot payments (payments with only one HTLC). Doing so addresses the known weaknesses in the legacy payment mechanism discussed above, and seamlessly upgrades the overall privacy and security of the basic payment style used today.
No action needs to be taken in order to benefit from this upgrade. Senders will automatically detect the receiver's support via feature bits in BOLT 11 invoices, falling back to the legacy protocol for older nodes. As such, the new mechanism will be used when paying from 0.9 to another 0.9 node, or a newer release of C-Lightning or Eclair. Further, these improved payments are secure and can be used even if intermediaries are outdated, as it only requires support from the endpoints.
Custom Onion-Tunneled TLV Payment Metadata Records
It is now possible to attach additional data to a payment using custom records. The send calls on the RPC interface have a field dest_custom_records that takes the custom records to send. This field is also present on QueryRoutes. Custom records take up space in the fixed size onion packet, leaving fewer bytes to describe the route. The pathfinding algorithm needs to know how many bytes are left and limit the route length to that. Checks for length and size have been added.
On the receiving end, the custom records are stored in the invoice database and exposed through the existing invoice query calls (LookupInvoice, ListInvoices, etc). This new feature allows any payment on the network to attach additional meta-data along with the payment. As an example, a user could send a payment to an exchange to deposit funds, and attach their account ID along-side the payment, which the exchange then checks+verifies upon receipt before crediting their account with the deposit. This is a highly anticipated feature as it allows far a large class of new applications to be built on top of Lightning. For further ideas on how to leverage this new feature, check out Laolu's talk on Advanced Lightning Applications from last last year.
Sending a payment with custom records attached will look something like the following on the command line:
{
"payment_error": "",
"payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
"payment_route": {
"total_time_lock": 1657728,
"total_fees": "0",
"total_amt": "1000",
"hops": [
{
"chan_id": "1589156041461923840",
"chan_capacity": "16777215",
"amt_to_forward": "1000",
"fee": "0",
"expiry": 1657728,
"amt_to_forward_msat": "1000000",
"fee_msat": "0",
"pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
"tlv_payload": true,
"mpp_record": null,
"custom_records": {
"5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
}
}
],
"total_fees_msat": "0",
"total_amt_msat": "1000000"
},
"payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634"
}
Notice the new custom_records field, as well as the tlv_payload boolean which specifies that this route used the new modern onion paylaoad format.
On the receiver, end, the new output of ListInvoices looks something like:
{
"memo": "",
"r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
"r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634",
<snip>
"htlcs": [
{
"chan_id": "1589156041461923840",
"htlc_index": "8",
"amt_msat": "1000000",
"accept_height": 1657685,
"accept_time": "1578617436",
"resolve_time": "1578617436",
"expiry_height": 1657728,
"state": "SETTLED",
"custom_records": {
"5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
},
"mpp_total_amt_msat": "0"
}
],
<snip>
}
Once again the custom_records will will expose a map from the integer type of the record to the raw bytes of the value. Note that custom records are only permitted with a type greater than 65536, any records below this value are reversed for official protocol usage. For further details with respect to the new RPC-level changes to the payment sending/invoicing calls, see the lnd API documentation which is now up-to-date as of lnd v0.9.0.
With the introduction of custom record sending and receiving, it became possible to attach arbitrary data to a payment. One obvious use case is attaching a human-readable message to a payment. Especially in the case of spontaneous keysend payments, this can give the receiver some context on the payment.
For example: tipping. Usually people sending a tip would want to include some information on who they are or what the tip is for. For Lightning this may be even more desired than for other payment methods, because payments are anonymous by default.
Custom records can also be set using the lncli command like so:
lncli sendpayment -d 0374e7fb33eafd74fe1acb6db7680bb4aa78e1c839a6f954e38abfad680f645ef7 -a 100 --keysend --data 323442=00,3234556=ffff080812
To specify a string value, the standard command line tool xxd can be used (the example record id here is the 3-byte ascii string 'tip' converted to an integer):
--data 7629168=$(echo -n "Thank you!" | xxd -pu -c 10000)
(The -c parameter is to prevent xxd from inserting line breaks)
Note: The available onion blob space of 1300 bytes is used for routing info and custom records. The bigger the size of the custom records, the fewer bytes remain for routing info and the shorter the maximum route length will be.
New Payment Type: keysend
One application of custom records is a spontaneous payment, also known as keysend. In key send, a custom record is used to encode the payment preimage in the onion payload for the recipient of the payment. This allows them to pull the payment without prior knowledge of the preimage.
Note that spontaneous payment is not yet defined in the Lightning spec. Therefore the current implementation should be considered experimental and is subjected to change.
In order to send a keysend payment on the command line, a new flag --keysend needs to be specified, along with the destination (--dest) and the amount to send (--amt). Notice that we don't specify a payment hash anywhere! Instead, the sender will actually encrypt the payment pre-image to the receiver, who will then decrypt the payload along with the rest of the normal per-hop routing information.
A sample keysend executed at the command line resembles something liek the following:
🏔 tlncli sendpayment --keysend --dest=0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b --amt=1000 --final_cltv_delta=40
{
"payment_error": "",
"payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
"payment_route": {
"total_time_lock": 1657728,
"total_fees": "0",
"total_amt": "1000",
"hops": [
{
"chan_id": "1589156041461923840",
"chan_capacity": "16777215",
"amt_to_forward": "1000",
"fee": "0",
"expiry": 1657728,
"amt_to_forward_msat": "1000000",
"fee_msat": "0",
"pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
"tlv_payload": true,
"mpp_record": null,
"custom_records": {
"5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
}
}
],
"total_fees_msat": "0",
"total_amt_msat": "1000000"
},
"payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634"
}
The key area to examine is the following:
"custom_records": {
"5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
}
Here we use lnd's keysend record record to package the pre-image to the receiver. You can verify that the payment_preimage field and the payment_hash field correspond. If one wishes to perform a keysend programmatically via the RPC API, then the DestCustomRecords field will need to be set accordingly.
On the receiver side, by default all nodes will reject any incoming keysend payments. In order to enable receiving such payments, nodes will need to opt-in by starting lnd with a new flag: --accept-keysend. When examining an invoice, it's easy to determine if it was payed using a keysend or not. When we receive a new incoming keysend payment that's to be accepted, we'll insert a new invoice into the database which looks something like the following:
{
"memo": "",
"r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
"r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634",
"value": "1000",
"value_msat": "1000000",
"settled": true,
"creation_date": "1578617436",
"settle_date": "1578617436",
"payment_request": "",
"description_hash": null,
<snip>
{
"chan_id": "1589156041461923840",
"htlc_index": "8",
"amt_msat": "1000000",
"accept_height": 1657685,
"accept_time": "1578617436",
"resolve_time": "1578617436",
"expiry_height": 1657728,
"state": "SETTLED",
"custom_records": {
"5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
},
"mpp_total_amt_msat": "0"
}
],
"features": {
"9": {
"name": "tlv-onion",
"is_required": false,
"is_known": true
}
},
"is_keysend": true
}
Notice that there's no payment_request value, and the set of new fields such as custom_records and information detailing the set of feature bits used to complete the payment (described in further detail below).
First-Class Rebalancing via Circular Payments to Self
We have added support to send a payment to yourself. This allows rebalancing of channels without the need for external scripts. To control the incoming channel, a new last hop routing restriction was added.
This new feature is a valuable tool in one's toolkit for node management as it allows you control the incoming and outgoing channels used to send a payment. Let's say I have a channel with ID 12345 and I sent to send funds from that channel (rebalancing to be able to receive more into the channel) to another channel that I have with peer 02e1dfa (rebalancing to be able to send more out of the channel). First, I'll create a new invoice on the command line for the amount I want to use for rebalancing purposes:
🏔 lncli --network=testnet addinvoice --amt=1000
{
"r_hash": "ac7c8d8f72b34d68b1ed53b1401908b4fa3a2b0c41a21206686289f437bd02fa",
"payment_request": "lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv",
"add_index": "167"
}
Now lnd is ready to execute the rebalancing attempt. To do this, we'll execute a new modified call to our trusty payinvoice command. Note that it's possible to do this programmatically+ as well using the RPC interface API. In any case, here's our command to complete this rebalancing attempt:
lncli --network=testnet payinvoice --allow_self_payment --outgoing_chan_id=12345 --last_hop=02e1dfa --pay_req=lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv
Local balance check
Failure reporting in case none of the local channels has enough balance to execute the payment has been improved. We now return a specific insufficient balance payment result instead of only indicating that no route could be found.
mSAT support
Support for mSAT payments has been added on the RPC level for payments and invoices.
Privacy Enhancement
This new release of lnd will properly initialize the starting mix-header Sphinx packets with random bytes in order to patch a recently discovered privacy leak that could potentially allow the receiver of a payment to ascertain the number of hops the payment traveled. See this email by roasbeef for further details.
BOLT Compatibility
Invalid Onion Payload Failure
0.9 now returns the new invalid_onion_payload failure upon receiving a malformed onion TLV payload.
Flat Features
The legacy local and global feature bit namespaces have been merged into a single namespace, simplifying the way feature bits are assigned and used within the protocol. Features previously advertised as global in init messages will remain for backwards compatibility, while new features will be appended to the existing local features field. The global features field will be deprecated entirely once this is widespread within the network.
Validate Sorted Uncompressed Short Channel IDs
0.9 now verifies that short channel IDs in QueryShortChanIDs and ChannelRangeReply are sent in ascending order for the uncompressed format. This had already implemented for the zlib encoding, but is now checked across both encodings.
Add payment_secret to BOLT 11 Payment Requests
Payment requests containing a payment_secret are properly encoded and decoded in 0.9. In addition, every newly created payment requests will encode a random payment_secret so that senders may use the improved MPP payment mechanism.
Cross-Implementation Protocol Compatibility Fixes
It was discovered that slight deviations existed between the various implementations w.r.t the way they implemented the gossip queries protocol feature. This version of lnd has been patched to ensure that we'll properly send responses as dictated by the spec to ensure better compatibility with c-lightning and various other implementations.
Channel Funding+Management Extensions
Funding Flow Abstraction
This new release of lnd contains a dramatically refactored funding workflow internally. This new refactoring allowed us to abstract away aspects such as how the funding transaction is constructed. We have many new features planned around this cut out such as generic external channel funding driven by PSBT, internal merging of several funding transaction into a single, or specifying a cold commitment key (where all your funds are sent to on force close, only need for sweeping and not channel updates).
With this release, we're shipping a low-level call that allow external crafting of a funding transaction, given extra-protocol negotiation. In other words, a new RPC call that allows two parties to agree on the structure of a funding transaction outside of the protocol, obtain the outpoint, then use that as part of the normal funding workflow. Such a flow is useful for a number of advanced features such as the creation of channel factories.
The new FundingShim parameter allows a caller to initiate a new funding flow, with apriori knowledge of information typically generated dynamically during the funding workflow such as the outpoint of the final funding transaction and the keys used within the multi-sig output. A responder will then use the new FundingStateStep method to register a matching funding intent to dispatch automatically based on an expected pending channel ID.
As mentioned above, we have a number of interesting features planned for this new API call, so stay tuned!
Decoupled Min HTLC Settings
The default minimum htlc amount channel open parameter is lowered from 1000 msat to 1 msat. This parameter is immutable during the life of the channel. By changing it to the lowest possible value, maximum flexibility is achieved. The minimum htlc amount can still be constrained to a higher value through the forwarding policy. Updating the minimum forward amount is now possible via the RPC and lncli.
Option Upfront Shutdown Support
This release adds support for option_upfront_shutdown which enforces payout on cooperative close to an address that is set on channel open. lnd will disconnect from peers that attempt to close out to a script other than the one that they specified on startup. Further information about feature bit can be found in the BOLT2 specification. Use of this feature provides a partial protection against node compromise because channels can be set to close out to addresses which are not controlled by lnd.
An upfront shutdown address can be set with the close_address field in the lnrpc/lncli OpenChannel call. Currently, only seed controlled addresses can be set as the upfront shutdown address when we do not initiate a channel open because this step is non-interactive. lnd can be started with --enable-upfront-shutdown to set seed addresses for all channel opens and closes by default.
Sweep Small Outputs
Sweeper (responsible for sweeping close transaction outputs back into the wallet) has been enhanced so that it also sweeps small outputs that on their own do not reach the sweep transaction dust limit. It does so by attaching additional utxos from the wallet, if available. This change is a preparation for the upcoming anchor commitment format.
This change also makes it easier to use the BumpFee RPC (or lncli wallet bumpfee) call for generic CPFP usage.
Close Address
Channels can be closed out to a custom address using the delivery_address field in lncli/lnrpc's CloseChannel call. Note that this feature cannot be used if option_upfront_shutdown, described above, was pre-set for the channel on opening.
This feature allows users to close out a channel and make a payment within the same transaction. This can also be used with Lightning Loop to initiate a Loop In from a channel close, or even to create a new channel by sending to a P2WSH multi-sig script using the new funding APIs described above.
Autopilot External Score Trigger
Autopilot will now check whether it should open more channels after the external scores list has been updates.
Channel Fitness Tracking
A channelfitness subsytem for tracking the health of the node's existing set of channels has been added. This subsystem tracks the online status of the remote peers that we have channels open with. These results are not currently persisted, so uptime is tracked from node restart rather than from channel open time. A lifetime value which indicates how long a peer has been monitored for is provided to allow calculation of uptime percentages. Uptime and lifetime are exposed as experimental fields on the lnrpc/lncli ListChannels call; both are expressed in seconds.
Note: these fields are experimental and may be moved or changed in future versions on LND.
Routing
Mission control
Several updates have been applied to mission control, the sub-system in lnd that is responsible for tracking the performance of nodes involved in past payments.
The estimation of success probabilities for channels has been improved. Channels are no longer evaluated in isolation. A new factor was added that represents the overall performance of a node. This prevents pathfinding from getting stuck in trying every channel of a badly performing node.
Probability estimation for local channels is more accurate now. This prevents unnecessary detours to reach a direct peer.
Mission control now remembers more about what happened in the past. Previously it only kept the last payment attempt for a channel. With this release, this is extended to tracking both the last failure and the last success. The goal is to improve payment success rates especially when small amount probes are performed frequently.
Finally, mission control state reporting has changed slightly. Probabilities are now amount-dependent and are therefore no longer reported through QueryMissionControl. A new call QueryProbability takes over this function.
Pathfinding Improvements
The routing algorithm acknowledges that a pair of nodes may have multiple channels between them with distinct forwarding policies. This is not a recommended practice, but needs to be dealt with nevertheless. lnd will craft a route that satisfies the most demanding (fee and time lock) channel of the channel set.
Routing decisions are based on cost and reliability. We applied a small touch up in this area to tie break on reliability when costs are equal.
RPC Enhancements
Deeper Feature Bit Inspection
The tried and true lncli getinfo call has been upgraded to show additional detail including the set of feature bits that the node is configured to advertise on the network. A sample of the new format of the call looks something like:
{
"version": "0.9.0-beta commit=queue/v1.0.2-222-g6c8c99dae99e741d2817d444f8b11945ddd15e2e",
"identity_pubkey": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
<snip>
"features": {
"0": {
"name": "data-loss-protect",
"is_required": true,
"is_known": true
},
"5": {
"name": "upfront-shutdown-script",
"is_required": false,
"is_known": true
},
"7": {
"name": "gossip-queries",
"is_required": false,
"is_known": true
},
"9": {
"name": "tlv-onion",
"is_required": false,
"is_known": true
},
"13": {
"name": "static-remote-key",
"is_required": false,
"is_known": true
},
"15": {
"name": "payment-addr",
"is_required": false,
"is_known": true
},
"17": {
"name": "multi-path-payments",
"is_required": false,
"is_known": true
}
}
}
The same feature vectors format is also now exposed on a variety of other rpc responses, each with slightly difference semantics:
GetNodeInfoandDescribeGraph: displays the current feature vector advertised on anode_announcementwithin our routing table.DecodePayReq: displays the feature vector decoded from an arbitrary payment request.ListInvoices: displays the feature vectors advertised on our own invoices.ListPeers: displays the set of features advertised by the remote party while exchanginginitmessages.
Updates to Default gRPC Settings
The receive buffer of the gRPC client in lncli and the REST proxy has been increased from 50MiB to 200MiB to avoid errors on very large response data (for example, thousands of transactions in listchaintxns).
Uniform lncli Hex-Encoding
The gRPC gateway library that is used to expose lnd's interface as a REST/JSON API on top of the normal gRPC API did not support request parameters of the type []byte (or bytes in the gRPC proto language). The update to version 1.8.6 of this library fixes this and should now allow REST users to use all API calls that lnd offers. Fields declared as bytes in the proto file must be encoded as base64 when used over REST. Some string fields in rpc.proto that were only added as a workaround in earlier versions have been deprecated and will be removed in the next version.
With this change, all raw byte fields exposed on the lncli command responses will now all be hex-encoded, with no more usage of base64. This change was made to clear up confusion that users ran into when information was encoded in multiple ways across distinct calls.
Updates to QueryRoutes
The QueryRoutes RPC call is now able to pin which outgoing channel is used when path finding, pin a pubkey to target for the final hop, and also accept custom unrolled hop hints. All of these changes make the RPC even more generally useable as it can be used for things like rebalancing pre-processing, advanced more advanced payment types, and fine-grained channel selection.
New RPC Calls
With the experimental signer RPC subserver you can now sign and verify messages with custom keys of the wallet (limited to the m/1017'/ special purpose derivation branch).
Note: This RPC requires a new permission, re-generate your admin.macaron and/or signer.macaroon to get the new permission.
A new RPC to derive shared Diffie-Hellman (ECDH) keys against a node's identity public key has been added to the experimental signer subserver.
A new RPC endpoint has been added to provide a subscription for peer online and offline events.
Mobile
Building
The mobile build is now created using the github.com/lightninglabs/falafel tool.
API changes
The Start library method has gotten an additional callback. Now lnd will notify both when the wallet is ready to be unlocked, and when lnd is ready to receive commands after the wallet has been unlocked.
Macaroons and TLS support
With falafel v0.7 the proper macaroon and TLS certificate is needed also when using the in-memory gRPC connections (which is done under the hood in the mobile build). This has no external facing changes, so it should work using the same APIs as before. This paves the way for using custom macaroons also on mobile, and moves the mobile API closer to the regular gRPC API.
Docker
The docker quick start files located in the docker directory have been made more user friendly by adding a persistent volume to lnd. This allows the lnd container to be restarted without losing its user data. The build instructions in docker/lnd/Dockerfile now also allow for quicker rebuilds by making use of the docker build cache.
The main Dockerfile in the root directory takes a branch, label or commit as a build argument to allow for custom builds.
The use of the docker-compose file is now easier and you can spin a btcd and lnd pair with just one command.
Note: If you've used the docker-compose.yml file before, make sure you run docker-compose build --no-cache the first time you use to new version to make sure the changes are taken into account.
Config Changes
Default unsafe-disconnect Setting and Deprecation
This option is deprecated and now defaults to true, allowing users to disconnect peers with active channels via the disconnectpeer rpc. This option was originally added long ago for purposes of testing and off by default to prevent unintended consistency bugs. This extra caution has been unneeded for several releases and no longer required. Once unsafe-disconnect is fully removed, this will remain the default behavior.
Staged Travis Builds
The travis build matrix now does preliminary sanity checks before entering a second phase that executes long-running tests. This results in less wasteful usage of travis instances, and tightens the feedback loop for developers contributing to the project.
Bug fixes
-
Accidental modification of database migration code has caused issues in the past. In this release, all migration code was isolated to prevent this from happening again.
-
Several fixes have been applied to the exchange of commitment signatures. Empty commit signatures are no longer sent.
-
Forwarding in
lndis "non-strict", meaning that any channel that is capable of carrying the payment may be utilized. The channel specified in the payload merely serves as a short-hand notation for the next node pub key. A bug was fixed where an incorrect failure message was returned if none of the channels qualified. -
Expired invoices are now canceled correctly. This fixes the "Invoices can be paid after expiration" bug.
-
A bug in the chain notifier has been fixed that led to a panic when paying a subscribed script hash multiple times in different blocks.
Changelog
The full list of changes since v0.8.2-beta can be found here:
Contributors (Alphabetical Order)
Alex Bosworth
Andras Banki-Horvath
Andreas M. Antonopoulos
Anton Kovalenko
Arik Sosman
bitromortac
Bjarne Magnussen
bluetegu
Carla Kirk-Cohen
Carson Mullins
Conner Fromknecht
Daniel McNally
Dennis Reimann
Elle Mouton
Johan T. Halseth
Joost Jager
Juan Pablo Civile
kiwiidb
Lars Lehtonen
Matheus Degiovani
Eugene Siegel
Olaoluwa Osuntokun
Oliver Gugger
Otto Suess
Philipp Gillé
Roei Erez
Steven Roose
Tomas Carnecky
Vignesh Karthikeyan
Wilmer Paulino
Yancy Tibbens
Yan Pritzker