Excess data in channel/node announcements/updates may not be verified/relayed #1814
Comments
Roasbeef
added
bug
dannypaz
added a commit
to sparkswap/lnd
that referenced
this issue
Sep 18, 2018
* contractcourt/chanarb test: expand TestChannelArbitratorCommitFailure to coop and local force close * macaroons: ensure path for macaroon DB exists Co-Authored-By: Karl Ranna <[email protected]> * config+cmd/lncli: make cleanAndExpandPath return empty string on empty path Co-Authored-By: Karl Ranna <[email protected]> * config+lnd: create macaroons files under the network dir Co-Authored-By: Karl Ranna <[email protected]> * config: remove default macaroon path Co-Authored-By: Karl Ranna <[email protected]> * cmd/lncli: add flags to determine the macaroon for the active network Co-Authored-By: Karl Ranna <[email protected]> * cmd/lncli: retrieve the macaroon for the current chain and network Co-Authored-By: Karl Ranna <[email protected]> * docs: update path to where macaroons are stored Co-Authored-By: Karl Ranna <[email protected]> * config: update comment for network directory * docker: update docker README to account for recent lncli/macaroon changes * cmd/lncli: properly parse cert and macaroon paths for all config variants In this commit, we ensure that we're able to properly parse the cert and macaroon paths for all relevant config variants. Before this commit, it would be the case that the macaroon path ended up empty if a user wasn't running the default (mainnet, lnddir) settings. In this commit, we remedy this by parsing each of the two (cert+macaroon) paths independently. * cmd/lncli: rename parseArgs to extractPathArgs, return error * autopilot/agent: add OnNodeUpdates signal Adds a new external signal alerting autopilot that new nodes have been added to the channel graph or an existing node has modified its channel announcment. This allows autopilot to examine its current state, and attempt to open channels if our target state is not yet met. * pilot: signal OnNodeUpdates to autopilot * autopilot/agent_test: test that agent recovers from initial empty graph * lnwire: recognize required query sync feature bit * lnwallet: fix logging message for fee floor * config: properly show list of sub-systems for show debuglevel Fixes lightningnetwork#1772. * chainntnfs: ensure proper fallback to scanning tx manually In this commit, we address a bug where it's possible that we still attempt to manually scan for a transaction to determine whether it's been included in the chain even after successfully checking the txindex and not finding it there. Now, we'll short-circuit this process by exiting early if the txindex lookup was successful but the transaction in question was not found. Otherwise, we'll fall back to the manual scan. * chainntnfs: extract helper test vars and functions In this commit, we extract some of the helper test variables and functions into their own file and guard them under a build flag. This is needed as some unit tests will be introduced in a future commit where most of the same functions within the interface tests are reused. In order to prevent these variables and functions from being exportable, we guard them by the "debug" build tag. * chainntnfs: guard test chain notifier under debug flag * multi: enable txindex on miner harness * build: update btcd+btcwallet+neutrino to latest versions * chainntnfs/neutrinonotify: update to latest API changes * routing/chainview: update to latest API changes * chainntnfs/bitcoind: initialize with height hint cache * chainntnfs/btcdnotify: initialize with height hint cache * chainntnfs/test_util: set 10ms trickleInterval for miner * chainntnfs/bitcoind tests: check found mempool case * chainntnfs/btcd: remove unnecessary check for tx==nil * chainntnfs/bitcoind: remove unnecessary check for tx==nil * chainntnfs: define common TxConfStatus type * chainntnfs/btcd+bitcoind: use common TxConfStatus * chainntnfs/btcd+bitcoind tests: use common TxConfStatus * chainntnfs/btcd: fallback to scan manually only in case of err != nil * chainntnfs/bitcoind: fallback to scan manually only in case of err != nil * chainntnfs/interface: fix TxConfStatus godocs for linter * htlcswitch/link_test: increase fwdpkg gc interval In this commit, we increase the fwdpkg gc interval to avoid having it conflict with switch tests that inspect forwarding packages. The current timeout is a little too short on travis, and sporadically fails TestChannelLinkCleanupSpuriousResponses, which was added recently. * peer: raise readHandler wait group done to defer statement In this commit, we raise the readHandler wait group done into a defer statement at the top of the method. This fixes an existing but that would cause the readHandler to declare it had exited, yet possibly still be waiting on the chan message stream below to exit. * peer: stop chanMsg streams in defer right after creation * peer: move atomic var increment in msgConsumer goroutine to defer In this commit we move the atomic var increment that signals the consumer goourtine has exited to the top of the method in a defer statement. This cleans up some duplicate code and also adheres to the pattern of using defers to signal cleaning up any dependent goroutine state on exit. * funding: add caller quit channel to waitUntilChannelOpen In this commit, we add a caller quit channel to waitUntilChannelOpen. This ensures that the caller won't block forever if it needs to exit before the funding manager exits, or the channel barrier is actually closed. * peer: allow msgStream goroutines to quit within the apply function In this commit, we thread through the quit of the peer to the execution of the apply function for a msgStream. This change ensures that if the target is still processing the message, then the peer is able to exit cleanly and not block insensately. * peer: allow msgConsumers to also exit on peer quit send * server: don't attempt to disable private channels * peer: don't attempt to re-enable any private channels * funding+peer: don't attempt to deliver messages if the peer is shutting down * build: update dep for golang.org/x/crypto to compile under golang 1.11 * lnpeer: extend Peer interface with new QuitSignal method In this commit, we extend the Peer interface with a new QuitSignal method. This method is meant to expose a read-only quit channel which will allow callers to cancel any actions based on the lifetime of the underlying peer. * peer: add QuitSignal to ensure peer struct adheres to lnpeer.Peer interface * htlcswitch+funding+discovery: update mock peers to add new QuitSignal method * discovery: select on peer's QuitSignal to allow caller to unblock if disconnecting In this commit, we select on the peer's QuitSignal to allow the caller to unblock if the peer itself is disconnecting. With this change, we now ensure that it isn't possible for a peer to block on this method and prevent a graceful exit. * peer: before and after obtaining link for chan update, check quit signal * chainntnfs/height_hint_cache: add disable flag to hint cache * chainntnfs/height_hint_cache_test: add tests for disabled cache * chainntnfs/interface_test: run tests w/ disabled cache * lnwallet/interface_test: run tests with disabled hint cache * chainregistry: disable height hint cache * chainntnfs/bitcoindnotify: disable height hints in testing * chainntnfs/btcdnotify: disable height hint cache in testing * lnd_test: correct force close test re: htlc limbo funds This commit fixes a bug in the integration test, that reliably fails after disabling the height hint cache. The test originally asserted that the htlc funds were in limbo, but was reading a stale copy of the force close information. Recently, the test was amended to provided a valid read of the force close in 96a0798. However, the issue was not apparent until build against the disabled height hint cache. The test is now correct to assert that there are no funds in limbo, as the commitment output has been swept, but the htlcs are still in the contract court, so the nursery is unaware of them. We also add another sanity check to validate that there are no pending htlcs on the force close at that point in time. * peer: ensure readHandler doesn't block on AddMsg to msgStream In this commit, we add a quit channel to the AddMsg method of the msgStream struct. Before this commit, if the queue was full, the readHandler would block and be unable to exit. We remedy this by leveraging the existing quit channel of the peer as an additional select case within the AddMsg method. * lnd+cmd/lncli: bump version to 0.5-beta * discovery/gossiper: remove optimistic channel announcement request In this commit, we aim to resolve an issue with nodes requesting for channel announcements when receiving a channel update for a channel they're not aware of. This can happen if a node is not caught up with the chain or if they receive updates for zombie channels. This would lead to a spam issue, as if a node is not caught up with the chain, every new update they receive is premature, causing them to manually request the backing channel announcement. Ideally, we should be able to detect this as a potential DoS vector and ban the node responsible, but for now we'll simply remove this functionality. * lnrpc: change lightning-api repo link redirect * htlcswitch: return concrete error from GetLinksByInterface * server: avoid logging error if no links are found In this commit, we avoid logging an error when the links associated with a peer are not found within its termination watcher. We do this to prevent a benign log message as the links have already been removed from the switch. * autopilot: use updateBalance rather than tracking balance explicitly In this commit, we modify the balanceUpdate autopilot signal to update the balance according to what's returned to the WalletBalance callback rather than explicitly tracking the balance. This gives the agent a better sense of what the wallet's balance actually is. * pilot: remove disconnection logic from chanController In this commit, we remove the disconnection logic within the chanController when failing to open a channel with a peer. We do this as it's already done within the autopilot agent, where it should be, and because it's possible that we were already connected to this node and we happened to disconnect them anyway. * pilot: improve error when unable to reach any of a peer's addresses * autopilot: update the Node interface to return a raw bytes, not the key In this commit, we modify the Node interface to return a set of raw bytes, rather than the full pubkey struct. We do this as within the package, commonly we only require the pubkey bytes for fingerprinting purposes. Before this commit, we were forced to _always_ decompress the pubkey which can be expensive done thousands of times a second. * discovery: update autopilot.Node usage to match recent API changes * peer: ensure link failures are processed in peer life cycle * chainntnfs/interface_test: fix unreliable historical block ntfns test After joining the two forked chains, it is necessary to ensure they both agree on the same best hash before proceeding to UnsafeStart the notifier. This is because when the BitcoindClient starts, it retrieves its best known block then calls GetBlockHeaderVerbose on the hash of the retrieved block. This block could be a reorged block if JoinNodes has not completed sync. If it is the case that the best block retrieved has been reorged out of the chain, GetBlockHeaderVerbose errors because bitcoind sets the number of confirmations to -1 on reorged blocks, and the btcd rpc client panics when parsing a block whose number of confirmations is negative. This parsing error is expected to be fixed, and as a more permanent solution chain backends should ensure that the `best block` they retrieve during startup has not been reorged out of the chain. * lnd_test: prevent calling Fatal in goroutine This commit prevents an error that I've seen on travis, wherein the test fails because a call to Fatal happens after the test finishes. The root cause is that we call Fatal in a goroutine that is reading from the subscribe graph rpc call. To fix this, we now pass an err chan back into the main test context, where we can receive any errors and fail the test if one comes through. * lnwire: add new compatibility parsing more for onion error chan updates In this commit, we add a compatibility mode for older version of clightning to ensure that we're able to properly parse all their channel updates. An older version of c-lightning would send out encapsulated onion error message with an additional type byte. This would throw off our parsing as we didn't expect the type byte, and so we always 2 bytes off. In order to ensure that we're able to parse these messages and make adjustments to our path finding, we'll first check to see if the type byte is there, if so, then we'll snip off two bytes from the front and continue with parsing. if the bytes aren't found, then we can proceed as normal and parse the request. * lnd_test: add wait predicates to pending channel checks * peer: ensure we unlock the msgCond during peer msgConsumer exit * server: attempt reconnection to all known addresses * server: remove unused lightningID field * build: update to latest versions of neutrino+btcwallet In this commit, we update to the latest versions of btcwallet+neutrino that fix a number of bugs within lnd itself. Namely, we ensure that we no longer print out garbage bytes, properly reconnect btcd after being disconnected, ensure we don't add duplicate utxos, and finally ensure that we always start the rescan from the wallet's initial birthday. Fixes lightningnetwork#1775. Fixes lightningnetwork#1494. Fixes lightningnetwork#444. * routing/router: add assume chan valid * server: pass AssumeChannelValid to router * config: add experimental assumechanvalid flag * routing/conf: add experimental assume valid conf * Makefile: add tags env argument to build and install * contractcourt/channel_arbitrator: handle onchain close race on restart * autopilot/agent: move signal processing out of select * autopilot/agent: signal nodeUpdates on own channel We do this to avoid a huge amount of goroutines piling up on initial graph sync, as they will all block trying to send the node update on the stateUpdates channel. Now we instead make a new buffered channel nodeUpdates, and just return immediately if there is already a signal in the channel waiting to be processed. * autopilot/agent: signal chanOpenFailureUpdates on own channel We do this to avoid a huge amount of goroutines piling up when autopilot is trying to open many channels, as they will all block trying to send the update on the stateUpdates channel. Now we instead send them on a buffered channel, similar to what is done with the nodeUpdates. * autopilot/agent: signal chanPendingOpenUpdates on own channel * autopilot/agent: signal balanceUpdates on own channel * server: always backoff for unstable peers This commit modifies the connection peer backoff logic such that it will always backoff for "unstable" peers. Unstable in this context is determined by connections whose duration is shorter than 10 minutes. If a disconnect happens with a peer whose connection lasts longer than 10 minutes, we will scale back our stored backoff for that peer. This resolves an issue that would result in a tight connection loop with remote peers. This stemmed from the connection duration being very short, and always driving the backoff to the default backoff of 1 second. Short connections like this are now caught by the stable connection threshold. This also modifies the computation on the backoff relaxation to subtract the connection duration after applying randomized exponential backoff, which offers better stability when the connection duration and backoff are roughly equal. * server: stop requesting initial graph sync In this commit, we remove signaling for initial routing dumps, which create unnecessary log spam, bandwidth, and CPU. Now that gossip syncing is in full force, we will instead opt to use the more efficient querying/set reconciliation. Other nodes may still request initial gossip sync from us, and we will respond. * config: ensure ZMQ options when read from the config file are not equal * config: defer creating the base lnd dir until all flag parsing is done In this commit, we defer creating the base lnd directory until all flag parsing is done. We do this as it's possible that the config file specifies a lnddir, but it isn't actually used as the directory has already been created. * autopilot: optimize heavy loaded agent by fetching raw bytes for ChannelEdge In this commit, we implement an optimization to the autopilot agent to ensure that we don't spin and waste CPU when we either have a large graph, or a high max channel target for the agent. Before this commit, each time we went to read the state of a channel from disk, we would decompress the EC Point each time. However, for the case of the instal ChannlEdge struct to feed to the agent, we only actually need to obtain the pubkey, and can save the potentially expensive point decompression for each directional channel in the graph. * server: move gossip dispatch to peer See next commit msg for more detail. * peer: dispatch gossip sync in peer start This commit moves the gossip sync dispatch such that it is more tightly coupled to the life cycle of the peer. In testing, I noticed that the gossip syncer needs to be dispatched before the first gossip messages come across the wire. The prior spawn location in the server happens after starting all of the peer's goroutines, which could permit an ordering where the gossip syncer has not yet been registered. The new location registers the gossip syncer within the read handler such that the call is blocks before any messages are read. * peer: log disconnect to info, remove go-errors pkg * discovery/gossiper: require explict gossip syncer init This commit removes the fallback in fetchGossipSyncer that creates a gossip syncer if one is not registered w/in the gossiper. Now that we register gossip syncers explicitly before reading any gossip query messages, this should not longer be required. The fallback also did not honor the cfg.NoChanUpdates flag, which may have led to inconsistencies between configuration and actual behavior. * channeldb/graph: identify edge chan id on failure * channeldb/graph: refactor UpdateEdgePolicy to use existing db transaction * channeldb/graph: fix off-by-one public key slice In this commit, we fix an off-by-one error when slicing the public key from the serialized node info byte slice. This would cause us to write an extra byte to all edge policies. Even though the values were read correctly, when attempting to calculate the offset of an edge's update time going backwards, we'd always be incorrect, causing us to not properly prune the edge update index. * channeldb/graph: properly determine old update timestamp for an edge In this commit, we fix a lingering issue within the edge update index where entries were not being properly pruned due to an incorrect calculation of the offset of an edge's last update time. Since the offset is being determined from the end to the start, we need to subtract all the fields after an edge policy's last update time from the total amount of bytes of the serialized edge policy to determine the correct offset. This was also slightly off as the edge policy included an extra byte, which has been fixed in the previous commit. Instead of continuing the slicing approach however, we'll switch to deserializing the raw bytes of an edge's policy to ensure this doesn't happen in the future when/if the serialization methods change or extra data is included. * channeldb/graph_test: properly check entries within edge update index Due to entries within the edge update index having a nil value, the tests need to be modified to account for this. Previously, we'd assume that if we were unable to retrieve a value for a certain key that the entry was non-existent, which is why the improper pruning bug was not caught. Instead, we'll assert the number of entries to be the expected value and populate a lookup map to determine whether the correct entries exist within it. * channeldb/graph_test: extend prune edge update index test to update edges In this commit, we extend TestChannelEdgePruningUpdateIndexDeletion test to include one more update for each edge. By doing this, we can correctly determine whether old entries were properly pruned from the index once a new update has arrived. * channeldb: add migration to properly prune edge update index In this commit, we introduce a migration to fix some of the recent issues found w.r.t. the edge update index. The migration attempts to fix two things: 1) Edge policies include an extra byte at the end due to reading an extra byte for the node's public key from the serialized node info. 2) Properly prune all stale entries within the edge update index. As a result of this migration, nodes will have a slightly smaller in size channeldb. We will also no longer send stale edges to our peers in response to their gossip queries, which should also fix the fetching channel announcement for closed channels issue. * channeldb/graph_test: ensure policies for an edge have different timestamps In this commit, we ensure policies for edges we create in TestChanUpdatesInHorizon have different update timestamps. This ensures that there are two entries per edge in the edge update index. Because of this, the test will fail because ChanUpdatesInHorizon will return duplicate channel edges due to looking at all the entries within the edge update index. This will be addressed in a future commit to allow the set of tests to pass once again. * channeldb/graph: dedup channel edges returned from ChanUpdatesInHorizon In this commit, we ensure that we de-duplicate the set of channel edges returned from ChanUpdatesInHorizon. Other subsystems within lnd use this method to retrieve and send all the channels with updates within a time series to network peers. However, since the method looks at the edge update index, which can include up to two entries per edge for each policy, it's possible that we'd send channel announcements and updates twice, causing extra bandwidth. * lnwallet: ensure that each backend uses a distinct seed in integration tests In this commit, we add an additional degree of isolation to the set of integration tests. A bug was recently fixed to ensure that the wallet always starts rescans from _after_ it's birthday. In the past it would miss some funds that were deposited _right_ before the birthday of the wallet. Fixing this bug exposed a test flake wherein the btcd node would itself rescan back and collect some of the funds that were last sent to the bitcoind node. In order to fix this, we now ensure that each backend will use a unique HD seed such that the tests are still deterministic for each backend and role. * lnwire: add new ExtraOpaqueData field to gossip messages In this commit, we add a new field to all the existing gossip messages: ExtraOpqueData. We do this, as before this commit, if we came across a ChannelUpdate message with a set of optional fields, then we wouldn't be able to properly parse the signatures related to the message. If we never corrected this behavior, then we would violate the forwards compatible principle we use when parsing existing messages. As these messages can now be padded out to the max message size, we've increased the MaxPayloadLength value for all of these messages. Fixes lightningnetwork#1814. * lnwire: extend quickcheck parser tests to add extra data for gossip messages * channeldb: add new ExtraOpaqueData to edge policy+update and node ann In this commit, we add a mirror set of fields to the ones we recently added to the set of gossip wire messages. With these set of fields in place, we ensure that we'll be able to properly store and re-validate gossip messages that contain a set of extra/optional fields. * channeldb: extend set of graph tests to include opque data where pertinent * config: rename noencryptwallet to noseedbackup This commit renames the confusing noencryptwallet flag to noseedbackup, since this highlights the more crucial information of the flags behavior to the user. The description has also been capitalized to urge the user think twice about what they're doing. * docs/INSTALL: remove noencryptwallet from install docs We will be slowly phasing this out, though abruptly discontinuing support would be a more extensive change. For now, we will ensure that this feature is not recommended to users setting up a new wallet. * lntest/node: change noencryptwallet to noseedbackup * lnd: use default enc pw when noseedbackup is passed * cmd/lncli/commands: change noencryptwallet -> noseedbackup * walletunlocker/service: change noencryptwallet reference to noseedbackup * discovery: update utils to properly include opaque data in gossip msgs * discovery: ensure we populate ExtraOpaqueData when writing/reading from disk * discovery: add new set of tests to ensure we validate w/ extra data * docker/lnd/start-lnd.sh: pass noseedbackup instead of noencryptwallet * sample-lnd.conf: remove noencryptwallet flag from sample * macaroons/README: update noencryptwallet -> noseedbackup * chanseries+server: populate ExtraOpaqueData field when reading from disk * channeldb: add limit on the max number of opaque bytes per announcement In this commit, we add a new limit on the largest number of extra opaque bytes that we'll allow to be written per vertex/edge. We do this in order to limit the amount of disk space that we expose, as it's possible that nodes may start to pad their announcements adding an additional externalized cost as nodes may need to continue to store and relay these large announcements. * amount paid should be in satoshies as other fields used for amount * channeldb: also ignore the EOF error when trying to read ExtraOpaqueBytes In this commit, we account for the additional case wherein the announcement hasn't yet been written with the extra zero byte to indicate that there aren't any remaining bytes to be read. Before this commit, we accounted for the case where the announcement was written with the extra byte, but now we ensure that legacy nodes that upgrade will be able to boot properly. * dep: constrain siphash, x/sys, and errgo.v1, upd neutrino * peer: prune persistent peer connection on zero on-disk channels In this commit, we fix a small bug with regards to the persistent peer connection pruning logic. Before this commit, it'd be the case that we'd prune a persistent connection to a peer if all links happen to be inactive. This isn't ideal, as the channels are still open, so we should always be atttempting to connect to them. We fix this by looking at the set of channels on-disk instead and prune the persistent connection if there aren't any. * lnwire/features: add required gq to local features This commit adds the required feature name to our set of local known features. This will allow other peers connecting to us to set the required gossip queries feature bit. This is required for the subsequent commits, which instruct the server to set the bit depending on user configured preferences. * lnd: ensure that we set the wallet birthday for --noencryptwallet In this commit, we ensure that we always set the wallet birthday. If the user has provided a seed, or is creating a new one, then it will be overwritten below. However, before this commit, if a user started with the --noencryptwallet flag, then we would _always_ start to rescan from genesis with the recent bug fix to ensure that we always start after the birthday. * autopilot/agent: track pending connections This commit modifies the autopilot agent to track all pending connection requests, and forgo further attempts if a connection is already present. Previously, the agent would try and establish hundreds of requests to a node, especially if the connections were timing out and not returning. This resulted in an OOM OMM when cranking up maxchannels to 200, since there would be close to 10k pending connections before the program was terminated. The issue was compounded by periodic batch timeouts, causing autopilot to try and process thousands of triggers for failing connections to the same peer. With these fixes, autopilot will skip nodes that we are trying to connect to during heuristic selection. The CPU and memory utilization have been significantly reduced as a result. * autopilot/interface: expose NodeID in AttachmentDirective * autopilot/prefattach: set NodeID for selection candidates * autopilot/agent_test: ensure directives use unique keys This commit ensures that the mock attachment directives use unique keys, ensuring that they aren't skipped due to already having pending connection requests. The tests fail when they're all the same since they collide in the pendingConns map. * autopilot/agent_test: adds TestAgentSkipPendingConns Adds a test asserting that the agent prevents itself from making duplicate outstanding connection requests to the same peer. * autopilot/agent_test: remove Fatalf calls from goroutines * autopilot/multi: replace PubKey -> NodeKey on directive * lnwire/lnwire_test: assert nil 0-length sid reply msg * discovery/syncer: delay replies after initial sync to prevent DOS * discovery/gossiper: remove go-errors pkg * discovery/syncer_test: add DOS delayed replies test * chanseries: fix panic bug by ensuring we access a non-nil edge In this commit, we fix an existing but that would cause the daemon to at times crash. Before this commit, we access the wrong edge, which would possibly actually be nil, leading to a panic. In this commit we fix this by ensuring we access the proper edge which is known to be non-nil at this point in the control flow. * lntest: log the expected and last balance in WaitForBalance error case * test: fix flake in link node garbage collection test In this commit, we fix a flake in the link node garbage collection test by ensuring the channels have been fully closed on both sides before we attempt to restart and ensure that they don't actually establish connections. Without this check, it's possible that either side hasn't yet processed all the blocks, so they'll still reconnect to each other on start up. * build: update to latest version of neutrino In this commit we update to the latest version of neutrino which fixes a bug in the filter header syncing logic. * server: ensure persistent connections to tor addrs are made if active In this commit, we restrict the persistent connection logic on startup to only attempt to establish connections to Tor addresses if Tor outbound support is enabled. Otherwise, we'll continually attempt to reach the address even though we never will. * build: update btcwallet to latest version In this commit, we update btcwallet to the latest version which fixes a panic bug when attempting to notify a relevant transaction. * lnrpc: add field amt_paid_msat to Invoice * rpcserver: set AmtPaidMsat when creating RPC invoice * cmd/lncli+fundingmanager: use a default num min confs for 1 for openchannel * added missing spaces to Makefile * cmd/lncli: fix missing whitespaces on lncli queryroutes desc Adds whitespaces missing on command description, without those whitespaces at the end of the line it showed the words together * channeldb: remove unused fwdPackageLogBucket key In this commit, we move the declaration of the key for an unused bucket. In the past, this bucket was used to store the revocation forwarding package log. However, this has been moved under the key `fwdPackagesKey`. * channeldb: fix bucket creation hierarchy in createChannelDB In this commit, we fix a bug in the bucket creation code in createChannelDB. This bug can cause migrations on older nodes to fail, as we expect the bucket to already have been created. With this commit, we ensure that all the buckets under the main node and edges bucket are properly created. Otherwise, a set of the newer migrations will fail to apply for nodes updating from 0.4. * release: Add ARMv6 platform to release.sh * discovery/syncer: use rate limiter for gossip queries This commit replaces the simplistic rate limiting technique added in 557cb6e, to use the golang.org/x/time's rate limiter. This has the benefit of performing traffic shaping to meet a target maximum rate, and yet tolerate bursts. Bursts are expected during initial sync, though should become more rare afterwards. Performing traffic shaping with this mechanism should improve the ability of the gossip syncer to detect sustained bursts from the remote peer, and penalize them appropriately. This commit also modifies the default parameters to accept bursts of 10 queries, with a target rate of 1 reply every 5 seconds. * dep: add golang.org/x/time for rate limiter * lnd_test: make sure we stay disconnected using WaitInvariant in testGarbageCollectLinkNodes * lnd_test: restart nodes twice to trigger pruning logic * cmd/lncli: Fixed conflicting payinvoice amount output * channeldb: relax bucket assumptions for latest db migration In this commit, we no longer assume that the bucket hierarchy has been created properly when applying the latest DB migration. On older nodes that never obtained a channel graph, or updated _before_ the query sync stuff was added, then they're missing buckets that the migration expects them to have. We fix this by simply creating the buckets as we go, if needed. * contractcourt/chain_arbitrator: parallel chain watcher start This commit restructures the initialization procedure for chain watchers such that they can proceed in parallel. This is primarily to help nodes running with the neutrino backend, which otherwise forces a serial rescan for each active channel to check for spentness. Doing so allows the rescans to take advantage of batch scheduling in registering for the spend notifications, ensuring that only one or two passes are made, as opposed to one for each channel. Lastly, this commit ensures that the chain arb is properly shutdown if any of it's chain watchers or channel arbs fails to start, so as to cancel their goroutines before exiting. * tor/controller: add support for v3 onion services In this commit, we extend our Tor controller to also support creating v3 onion services, as they are now supported by the Tor daemon. We also refactor our existing AddOnion method to take in a config struct that houses all of the required options to create/restore an onion service. * tor/controller: add tor version number check In this commit, we add a check for the Tor server's version number to ensure it supports creating v3 onion services through its control port. * config: modify private key path to be onion type agnostic * server: update to lastest tor controller api * docs: update v3 tor docs * build: update dep to latest version of btcwallet * lnwallet: add test to ensure that all backends show unconf txns in ListTransactionDetails In this commit, we add a new test to the set of lnwallet integration tests. In this new test, we aim to ensure that all backends are able to display unconfirmed transactions in ListChainTransactions. As of this commit, this test fails as no backends will return unconfirmed transactions properly. * lnwallet: fix ListTransactionDetails to ensure unconfirmed transactions are installed In this commit, we fix a bug in the arguments to GetTransactions for the btcwallet implementation of the WalletController interface. Before this commit, we wouldn't properly return unconfirmed transactions. The issue was that we didn't specify the special mempool height of "-1", as the ending height. The mempool height is actually internally converted to the highest possible height that can fit into a int32. In this commit, we set the start to zero, and end to -1 (actually 2^32-1) to properly scan for unconfirmed transactions. Fixes lightningnetwork#1422. * lnwallet: update for new API change in NewBitcoindClient * lnwallet: add test to execrise unconfirmed transcations notified by SubscribeTransactions In this commit, we add a new test to ensure that all backends will properly send out notifications when an unconfirmed transcation that we send is inserted into the tx store. Before we updated the btcwallet build commit in dep, this would fail for neutrino but now passes. * multi: update to new BitcoindClient API * config: return an error if unable to parse values in the config file * channeldb: don't use KeyN in latest migration In this commit, we fix a bug in the latest migration that could cause the migration to end in a panic. Additionally, we modify the migration to exit early if the bucket wasn't found, as in this case, no migration is required. Fixes lightningnetwork#1874. * config: fix linter error * channeldb/invoices: extend invoice queries to allow backwards pagination * channeldb/invoice_test: refactor TestQueryInvoices and add reverse test cases * lnrpc: add fields to paginate invoices backwards * rpcserver: include new proto fields in ListInvoices rpc * cmd/lncli: add description and reversed flag to listinvoices * make: clean lnd-debug, lncli-debug and .vendor-new * cmd/lncli: update listinvoices docs with new pagination support * lnrpc: update ListInvoices RPC with new pagination support * discovery: fix logging for gossip syncer rate limiting * lnwire: check length of payload in decodeShortChanIDs * README: add security section in readme * build: update to point to latest version of neutrino In this commit, we update the build to point to the latest version of neutrino which includes a bug fix for a regression that would cause the daemon to spin when at chain tip attempting to always fetch the next set of headers though it was already fully up to date. * channeldb: fix bug in migration from 0.4 to 0.5 In this commit, we fix a bug in the latest database migration when migrating from 0.4 to 0.5. There's an issue in bolt db where if one deletes a bucket that has a key with a nil value, it thinks that's a sub bucket and attempts a bucket deletion. This will fail as it's not actually a sub-bucket. We get around this by using a cursor to manually delete items in the bucket. Fixes lightningnetwork#1907. * fix testing issue with duplicate invoices
dannypaz
added a commit
to sparkswap/lnd
that referenced
this issue
Oct 17, 2018
* contractcourt/chanarb test: expand TestChannelArbitratorCommitFailure to coop and local force close * macaroons: ensure path for macaroon DB exists Co-Authored-By: Karl Ranna <[email protected]> * config+cmd/lncli: make cleanAndExpandPath return empty string on empty path Co-Authored-By: Karl Ranna <[email protected]> * config+lnd: create macaroons files under the network dir Co-Authored-By: Karl Ranna <[email protected]> * config: remove default macaroon path Co-Authored-By: Karl Ranna <[email protected]> * cmd/lncli: add flags to determine the macaroon for the active network Co-Authored-By: Karl Ranna <[email protected]> * cmd/lncli: retrieve the macaroon for the current chain and network Co-Authored-By: Karl Ranna <[email protected]> * docs: update path to where macaroons are stored Co-Authored-By: Karl Ranna <[email protected]> * config: update comment for network directory * docker: update docker README to account for recent lncli/macaroon changes * cmd/lncli: properly parse cert and macaroon paths for all config variants In this commit, we ensure that we're able to properly parse the cert and macaroon paths for all relevant config variants. Before this commit, it would be the case that the macaroon path ended up empty if a user wasn't running the default (mainnet, lnddir) settings. In this commit, we remedy this by parsing each of the two (cert+macaroon) paths independently. * cmd/lncli: rename parseArgs to extractPathArgs, return error * autopilot/agent: add OnNodeUpdates signal Adds a new external signal alerting autopilot that new nodes have been added to the channel graph or an existing node has modified its channel announcment. This allows autopilot to examine its current state, and attempt to open channels if our target state is not yet met. * pilot: signal OnNodeUpdates to autopilot * autopilot/agent_test: test that agent recovers from initial empty graph * lnwire: recognize required query sync feature bit * lnwallet: fix logging message for fee floor * config: properly show list of sub-systems for show debuglevel Fixes lightningnetwork#1772. * chainntnfs: ensure proper fallback to scanning tx manually In this commit, we address a bug where it's possible that we still attempt to manually scan for a transaction to determine whether it's been included in the chain even after successfully checking the txindex and not finding it there. Now, we'll short-circuit this process by exiting early if the txindex lookup was successful but the transaction in question was not found. Otherwise, we'll fall back to the manual scan. * chainntnfs: extract helper test vars and functions In this commit, we extract some of the helper test variables and functions into their own file and guard them under a build flag. This is needed as some unit tests will be introduced in a future commit where most of the same functions within the interface tests are reused. In order to prevent these variables and functions from being exportable, we guard them by the "debug" build tag. * chainntnfs: guard test chain notifier under debug flag * multi: enable txindex on miner harness * build: update btcd+btcwallet+neutrino to latest versions * chainntnfs/neutrinonotify: update to latest API changes * routing/chainview: update to latest API changes * chainntnfs/bitcoind: initialize with height hint cache * chainntnfs/btcdnotify: initialize with height hint cache * chainntnfs/test_util: set 10ms trickleInterval for miner * chainntnfs/bitcoind tests: check found mempool case * chainntnfs/btcd: remove unnecessary check for tx==nil * chainntnfs/bitcoind: remove unnecessary check for tx==nil * chainntnfs: define common TxConfStatus type * chainntnfs/btcd+bitcoind: use common TxConfStatus * chainntnfs/btcd+bitcoind tests: use common TxConfStatus * chainntnfs/btcd: fallback to scan manually only in case of err != nil * chainntnfs/bitcoind: fallback to scan manually only in case of err != nil * chainntnfs/interface: fix TxConfStatus godocs for linter * htlcswitch/link_test: increase fwdpkg gc interval In this commit, we increase the fwdpkg gc interval to avoid having it conflict with switch tests that inspect forwarding packages. The current timeout is a little too short on travis, and sporadically fails TestChannelLinkCleanupSpuriousResponses, which was added recently. * peer: raise readHandler wait group done to defer statement In this commit, we raise the readHandler wait group done into a defer statement at the top of the method. This fixes an existing but that would cause the readHandler to declare it had exited, yet possibly still be waiting on the chan message stream below to exit. * peer: stop chanMsg streams in defer right after creation * peer: move atomic var increment in msgConsumer goroutine to defer In this commit we move the atomic var increment that signals the consumer goourtine has exited to the top of the method in a defer statement. This cleans up some duplicate code and also adheres to the pattern of using defers to signal cleaning up any dependent goroutine state on exit. * funding: add caller quit channel to waitUntilChannelOpen In this commit, we add a caller quit channel to waitUntilChannelOpen. This ensures that the caller won't block forever if it needs to exit before the funding manager exits, or the channel barrier is actually closed. * peer: allow msgStream goroutines to quit within the apply function In this commit, we thread through the quit of the peer to the execution of the apply function for a msgStream. This change ensures that if the target is still processing the message, then the peer is able to exit cleanly and not block insensately. * peer: allow msgConsumers to also exit on peer quit send * server: don't attempt to disable private channels * peer: don't attempt to re-enable any private channels * funding+peer: don't attempt to deliver messages if the peer is shutting down * build: update dep for golang.org/x/crypto to compile under golang 1.11 * lnpeer: extend Peer interface with new QuitSignal method In this commit, we extend the Peer interface with a new QuitSignal method. This method is meant to expose a read-only quit channel which will allow callers to cancel any actions based on the lifetime of the underlying peer. * peer: add QuitSignal to ensure peer struct adheres to lnpeer.Peer interface * htlcswitch+funding+discovery: update mock peers to add new QuitSignal method * discovery: select on peer's QuitSignal to allow caller to unblock if disconnecting In this commit, we select on the peer's QuitSignal to allow the caller to unblock if the peer itself is disconnecting. With this change, we now ensure that it isn't possible for a peer to block on this method and prevent a graceful exit. * peer: before and after obtaining link for chan update, check quit signal * chainntnfs/height_hint_cache: add disable flag to hint cache * chainntnfs/height_hint_cache_test: add tests for disabled cache * chainntnfs/interface_test: run tests w/ disabled cache * lnwallet/interface_test: run tests with disabled hint cache * chainregistry: disable height hint cache * chainntnfs/bitcoindnotify: disable height hints in testing * chainntnfs/btcdnotify: disable height hint cache in testing * lnd_test: correct force close test re: htlc limbo funds This commit fixes a bug in the integration test, that reliably fails after disabling the height hint cache. The test originally asserted that the htlc funds were in limbo, but was reading a stale copy of the force close information. Recently, the test was amended to provided a valid read of the force close in 96a0798. However, the issue was not apparent until build against the disabled height hint cache. The test is now correct to assert that there are no funds in limbo, as the commitment output has been swept, but the htlcs are still in the contract court, so the nursery is unaware of them. We also add another sanity check to validate that there are no pending htlcs on the force close at that point in time. * peer: ensure readHandler doesn't block on AddMsg to msgStream In this commit, we add a quit channel to the AddMsg method of the msgStream struct. Before this commit, if the queue was full, the readHandler would block and be unable to exit. We remedy this by leveraging the existing quit channel of the peer as an additional select case within the AddMsg method. * lnd+cmd/lncli: bump version to 0.5-beta * discovery/gossiper: remove optimistic channel announcement request In this commit, we aim to resolve an issue with nodes requesting for channel announcements when receiving a channel update for a channel they're not aware of. This can happen if a node is not caught up with the chain or if they receive updates for zombie channels. This would lead to a spam issue, as if a node is not caught up with the chain, every new update they receive is premature, causing them to manually request the backing channel announcement. Ideally, we should be able to detect this as a potential DoS vector and ban the node responsible, but for now we'll simply remove this functionality. * lnrpc: change lightning-api repo link redirect * htlcswitch: return concrete error from GetLinksByInterface * server: avoid logging error if no links are found In this commit, we avoid logging an error when the links associated with a peer are not found within its termination watcher. We do this to prevent a benign log message as the links have already been removed from the switch. * autopilot: use updateBalance rather than tracking balance explicitly In this commit, we modify the balanceUpdate autopilot signal to update the balance according to what's returned to the WalletBalance callback rather than explicitly tracking the balance. This gives the agent a better sense of what the wallet's balance actually is. * pilot: remove disconnection logic from chanController In this commit, we remove the disconnection logic within the chanController when failing to open a channel with a peer. We do this as it's already done within the autopilot agent, where it should be, and because it's possible that we were already connected to this node and we happened to disconnect them anyway. * pilot: improve error when unable to reach any of a peer's addresses * autopilot: update the Node interface to return a raw bytes, not the key In this commit, we modify the Node interface to return a set of raw bytes, rather than the full pubkey struct. We do this as within the package, commonly we only require the pubkey bytes for fingerprinting purposes. Before this commit, we were forced to _always_ decompress the pubkey which can be expensive done thousands of times a second. * discovery: update autopilot.Node usage to match recent API changes * peer: ensure link failures are processed in peer life cycle * chainntnfs/interface_test: fix unreliable historical block ntfns test After joining the two forked chains, it is necessary to ensure they both agree on the same best hash before proceeding to UnsafeStart the notifier. This is because when the BitcoindClient starts, it retrieves its best known block then calls GetBlockHeaderVerbose on the hash of the retrieved block. This block could be a reorged block if JoinNodes has not completed sync. If it is the case that the best block retrieved has been reorged out of the chain, GetBlockHeaderVerbose errors because bitcoind sets the number of confirmations to -1 on reorged blocks, and the btcd rpc client panics when parsing a block whose number of confirmations is negative. This parsing error is expected to be fixed, and as a more permanent solution chain backends should ensure that the `best block` they retrieve during startup has not been reorged out of the chain. * lnd_test: prevent calling Fatal in goroutine This commit prevents an error that I've seen on travis, wherein the test fails because a call to Fatal happens after the test finishes. The root cause is that we call Fatal in a goroutine that is reading from the subscribe graph rpc call. To fix this, we now pass an err chan back into the main test context, where we can receive any errors and fail the test if one comes through. * lnwire: add new compatibility parsing more for onion error chan updates In this commit, we add a compatibility mode for older version of clightning to ensure that we're able to properly parse all their channel updates. An older version of c-lightning would send out encapsulated onion error message with an additional type byte. This would throw off our parsing as we didn't expect the type byte, and so we always 2 bytes off. In order to ensure that we're able to parse these messages and make adjustments to our path finding, we'll first check to see if the type byte is there, if so, then we'll snip off two bytes from the front and continue with parsing. if the bytes aren't found, then we can proceed as normal and parse the request. * lnd_test: add wait predicates to pending channel checks * peer: ensure we unlock the msgCond during peer msgConsumer exit * server: attempt reconnection to all known addresses * server: remove unused lightningID field * build: update to latest versions of neutrino+btcwallet In this commit, we update to the latest versions of btcwallet+neutrino that fix a number of bugs within lnd itself. Namely, we ensure that we no longer print out garbage bytes, properly reconnect btcd after being disconnected, ensure we don't add duplicate utxos, and finally ensure that we always start the rescan from the wallet's initial birthday. Fixes lightningnetwork#1775. Fixes lightningnetwork#1494. Fixes lightningnetwork#444. * routing/router: add assume chan valid * server: pass AssumeChannelValid to router * config: add experimental assumechanvalid flag * routing/conf: add experimental assume valid conf * Makefile: add tags env argument to build and install * contractcourt/channel_arbitrator: handle onchain close race on restart * autopilot/agent: move signal processing out of select * autopilot/agent: signal nodeUpdates on own channel We do this to avoid a huge amount of goroutines piling up on initial graph sync, as they will all block trying to send the node update on the stateUpdates channel. Now we instead make a new buffered channel nodeUpdates, and just return immediately if there is already a signal in the channel waiting to be processed. * autopilot/agent: signal chanOpenFailureUpdates on own channel We do this to avoid a huge amount of goroutines piling up when autopilot is trying to open many channels, as they will all block trying to send the update on the stateUpdates channel. Now we instead send them on a buffered channel, similar to what is done with the nodeUpdates. * autopilot/agent: signal chanPendingOpenUpdates on own channel * autopilot/agent: signal balanceUpdates on own channel * server: always backoff for unstable peers This commit modifies the connection peer backoff logic such that it will always backoff for "unstable" peers. Unstable in this context is determined by connections whose duration is shorter than 10 minutes. If a disconnect happens with a peer whose connection lasts longer than 10 minutes, we will scale back our stored backoff for that peer. This resolves an issue that would result in a tight connection loop with remote peers. This stemmed from the connection duration being very short, and always driving the backoff to the default backoff of 1 second. Short connections like this are now caught by the stable connection threshold. This also modifies the computation on the backoff relaxation to subtract the connection duration after applying randomized exponential backoff, which offers better stability when the connection duration and backoff are roughly equal. * server: stop requesting initial graph sync In this commit, we remove signaling for initial routing dumps, which create unnecessary log spam, bandwidth, and CPU. Now that gossip syncing is in full force, we will instead opt to use the more efficient querying/set reconciliation. Other nodes may still request initial gossip sync from us, and we will respond. * config: ensure ZMQ options when read from the config file are not equal * config: defer creating the base lnd dir until all flag parsing is done In this commit, we defer creating the base lnd directory until all flag parsing is done. We do this as it's possible that the config file specifies a lnddir, but it isn't actually used as the directory has already been created. * autopilot: optimize heavy loaded agent by fetching raw bytes for ChannelEdge In this commit, we implement an optimization to the autopilot agent to ensure that we don't spin and waste CPU when we either have a large graph, or a high max channel target for the agent. Before this commit, each time we went to read the state of a channel from disk, we would decompress the EC Point each time. However, for the case of the instal ChannlEdge struct to feed to the agent, we only actually need to obtain the pubkey, and can save the potentially expensive point decompression for each directional channel in the graph. * server: move gossip dispatch to peer See next commit msg for more detail. * peer: dispatch gossip sync in peer start This commit moves the gossip sync dispatch such that it is more tightly coupled to the life cycle of the peer. In testing, I noticed that the gossip syncer needs to be dispatched before the first gossip messages come across the wire. The prior spawn location in the server happens after starting all of the peer's goroutines, which could permit an ordering where the gossip syncer has not yet been registered. The new location registers the gossip syncer within the read handler such that the call is blocks before any messages are read. * peer: log disconnect to info, remove go-errors pkg * discovery/gossiper: require explict gossip syncer init This commit removes the fallback in fetchGossipSyncer that creates a gossip syncer if one is not registered w/in the gossiper. Now that we register gossip syncers explicitly before reading any gossip query messages, this should not longer be required. The fallback also did not honor the cfg.NoChanUpdates flag, which may have led to inconsistencies between configuration and actual behavior. * channeldb/graph: identify edge chan id on failure * channeldb/graph: refactor UpdateEdgePolicy to use existing db transaction * channeldb/graph: fix off-by-one public key slice In this commit, we fix an off-by-one error when slicing the public key from the serialized node info byte slice. This would cause us to write an extra byte to all edge policies. Even though the values were read correctly, when attempting to calculate the offset of an edge's update time going backwards, we'd always be incorrect, causing us to not properly prune the edge update index. * channeldb/graph: properly determine old update timestamp for an edge In this commit, we fix a lingering issue within the edge update index where entries were not being properly pruned due to an incorrect calculation of the offset of an edge's last update time. Since the offset is being determined from the end to the start, we need to subtract all the fields after an edge policy's last update time from the total amount of bytes of the serialized edge policy to determine the correct offset. This was also slightly off as the edge policy included an extra byte, which has been fixed in the previous commit. Instead of continuing the slicing approach however, we'll switch to deserializing the raw bytes of an edge's policy to ensure this doesn't happen in the future when/if the serialization methods change or extra data is included. * channeldb/graph_test: properly check entries within edge update index Due to entries within the edge update index having a nil value, the tests need to be modified to account for this. Previously, we'd assume that if we were unable to retrieve a value for a certain key that the entry was non-existent, which is why the improper pruning bug was not caught. Instead, we'll assert the number of entries to be the expected value and populate a lookup map to determine whether the correct entries exist within it. * channeldb/graph_test: extend prune edge update index test to update edges In this commit, we extend TestChannelEdgePruningUpdateIndexDeletion test to include one more update for each edge. By doing this, we can correctly determine whether old entries were properly pruned from the index once a new update has arrived. * channeldb: add migration to properly prune edge update index In this commit, we introduce a migration to fix some of the recent issues found w.r.t. the edge update index. The migration attempts to fix two things: 1) Edge policies include an extra byte at the end due to reading an extra byte for the node's public key from the serialized node info. 2) Properly prune all stale entries within the edge update index. As a result of this migration, nodes will have a slightly smaller in size channeldb. We will also no longer send stale edges to our peers in response to their gossip queries, which should also fix the fetching channel announcement for closed channels issue. * channeldb/graph_test: ensure policies for an edge have different timestamps In this commit, we ensure policies for edges we create in TestChanUpdatesInHorizon have different update timestamps. This ensures that there are two entries per edge in the edge update index. Because of this, the test will fail because ChanUpdatesInHorizon will return duplicate channel edges due to looking at all the entries within the edge update index. This will be addressed in a future commit to allow the set of tests to pass once again. * channeldb/graph: dedup channel edges returned from ChanUpdatesInHorizon In this commit, we ensure that we de-duplicate the set of channel edges returned from ChanUpdatesInHorizon. Other subsystems within lnd use this method to retrieve and send all the channels with updates within a time series to network peers. However, since the method looks at the edge update index, which can include up to two entries per edge for each policy, it's possible that we'd send channel announcements and updates twice, causing extra bandwidth. * lnwallet: ensure that each backend uses a distinct seed in integration tests In this commit, we add an additional degree of isolation to the set of integration tests. A bug was recently fixed to ensure that the wallet always starts rescans from _after_ it's birthday. In the past it would miss some funds that were deposited _right_ before the birthday of the wallet. Fixing this bug exposed a test flake wherein the btcd node would itself rescan back and collect some of the funds that were last sent to the bitcoind node. In order to fix this, we now ensure that each backend will use a unique HD seed such that the tests are still deterministic for each backend and role. * lnwire: add new ExtraOpaqueData field to gossip messages In this commit, we add a new field to all the existing gossip messages: ExtraOpqueData. We do this, as before this commit, if we came across a ChannelUpdate message with a set of optional fields, then we wouldn't be able to properly parse the signatures related to the message. If we never corrected this behavior, then we would violate the forwards compatible principle we use when parsing existing messages. As these messages can now be padded out to the max message size, we've increased the MaxPayloadLength value for all of these messages. Fixes lightningnetwork#1814. * lnwire: extend quickcheck parser tests to add extra data for gossip messages * channeldb: add new ExtraOpaqueData to edge policy+update and node ann In this commit, we add a mirror set of fields to the ones we recently added to the set of gossip wire messages. With these set of fields in place, we ensure that we'll be able to properly store and re-validate gossip messages that contain a set of extra/optional fields. * channeldb: extend set of graph tests to include opque data where pertinent * config: rename noencryptwallet to noseedbackup This commit renames the confusing noencryptwallet flag to noseedbackup, since this highlights the more crucial information of the flags behavior to the user. The description has also been capitalized to urge the user think twice about what they're doing. * docs/INSTALL: remove noencryptwallet from install docs We will be slowly phasing this out, though abruptly discontinuing support would be a more extensive change. For now, we will ensure that this feature is not recommended to users setting up a new wallet. * lntest/node: change noencryptwallet to noseedbackup * lnd: use default enc pw when noseedbackup is passed * cmd/lncli/commands: change noencryptwallet -> noseedbackup * walletunlocker/service: change noencryptwallet reference to noseedbackup * discovery: update utils to properly include opaque data in gossip msgs * discovery: ensure we populate ExtraOpaqueData when writing/reading from disk * discovery: add new set of tests to ensure we validate w/ extra data * docker/lnd/start-lnd.sh: pass noseedbackup instead of noencryptwallet * sample-lnd.conf: remove noencryptwallet flag from sample * macaroons/README: update noencryptwallet -> noseedbackup * chanseries+server: populate ExtraOpaqueData field when reading from disk * channeldb: add limit on the max number of opaque bytes per announcement In this commit, we add a new limit on the largest number of extra opaque bytes that we'll allow to be written per vertex/edge. We do this in order to limit the amount of disk space that we expose, as it's possible that nodes may start to pad their announcements adding an additional externalized cost as nodes may need to continue to store and relay these large announcements. * amount paid should be in satoshies as other fields used for amount * channeldb: also ignore the EOF error when trying to read ExtraOpaqueBytes In this commit, we account for the additional case wherein the announcement hasn't yet been written with the extra zero byte to indicate that there aren't any remaining bytes to be read. Before this commit, we accounted for the case where the announcement was written with the extra byte, but now we ensure that legacy nodes that upgrade will be able to boot properly. * dep: constrain siphash, x/sys, and errgo.v1, upd neutrino * peer: prune persistent peer connection on zero on-disk channels In this commit, we fix a small bug with regards to the persistent peer connection pruning logic. Before this commit, it'd be the case that we'd prune a persistent connection to a peer if all links happen to be inactive. This isn't ideal, as the channels are still open, so we should always be atttempting to connect to them. We fix this by looking at the set of channels on-disk instead and prune the persistent connection if there aren't any. * lnwire/features: add required gq to local features This commit adds the required feature name to our set of local known features. This will allow other peers connecting to us to set the required gossip queries feature bit. This is required for the subsequent commits, which instruct the server to set the bit depending on user configured preferences. * lnd: ensure that we set the wallet birthday for --noencryptwallet In this commit, we ensure that we always set the wallet birthday. If the user has provided a seed, or is creating a new one, then it will be overwritten below. However, before this commit, if a user started with the --noencryptwallet flag, then we would _always_ start to rescan from genesis with the recent bug fix to ensure that we always start after the birthday. * autopilot/agent: track pending connections This commit modifies the autopilot agent to track all pending connection requests, and forgo further attempts if a connection is already present. Previously, the agent would try and establish hundreds of requests to a node, especially if the connections were timing out and not returning. This resulted in an OOM OMM when cranking up maxchannels to 200, since there would be close to 10k pending connections before the program was terminated. The issue was compounded by periodic batch timeouts, causing autopilot to try and process thousands of triggers for failing connections to the same peer. With these fixes, autopilot will skip nodes that we are trying to connect to during heuristic selection. The CPU and memory utilization have been significantly reduced as a result. * autopilot/interface: expose NodeID in AttachmentDirective * autopilot/prefattach: set NodeID for selection candidates * autopilot/agent_test: ensure directives use unique keys This commit ensures that the mock attachment directives use unique keys, ensuring that they aren't skipped due to already having pending connection requests. The tests fail when they're all the same since they collide in the pendingConns map. * autopilot/agent_test: adds TestAgentSkipPendingConns Adds a test asserting that the agent prevents itself from making duplicate outstanding connection requests to the same peer. * autopilot/agent_test: remove Fatalf calls from goroutines * autopilot/multi: replace PubKey -> NodeKey on directive * lnwire/lnwire_test: assert nil 0-length sid reply msg * discovery/syncer: delay replies after initial sync to prevent DOS * discovery/gossiper: remove go-errors pkg * discovery/syncer_test: add DOS delayed replies test * chanseries: fix panic bug by ensuring we access a non-nil edge In this commit, we fix an existing but that would cause the daemon to at times crash. Before this commit, we access the wrong edge, which would possibly actually be nil, leading to a panic. In this commit we fix this by ensuring we access the proper edge which is known to be non-nil at this point in the control flow. * lntest: log the expected and last balance in WaitForBalance error case * test: fix flake in link node garbage collection test In this commit, we fix a flake in the link node garbage collection test by ensuring the channels have been fully closed on both sides before we attempt to restart and ensure that they don't actually establish connections. Without this check, it's possible that either side hasn't yet processed all the blocks, so they'll still reconnect to each other on start up. * build: update to latest version of neutrino In this commit we update to the latest version of neutrino which fixes a bug in the filter header syncing logic. * server: ensure persistent connections to tor addrs are made if active In this commit, we restrict the persistent connection logic on startup to only attempt to establish connections to Tor addresses if Tor outbound support is enabled. Otherwise, we'll continually attempt to reach the address even though we never will. * build: update btcwallet to latest version In this commit, we update btcwallet to the latest version which fixes a panic bug when attempting to notify a relevant transaction. * lnrpc: add field amt_paid_msat to Invoice * rpcserver: set AmtPaidMsat when creating RPC invoice * cmd/lncli+fundingmanager: use a default num min confs for 1 for openchannel * added missing spaces to Makefile * cmd/lncli: fix missing whitespaces on lncli queryroutes desc Adds whitespaces missing on command description, without those whitespaces at the end of the line it showed the words together * channeldb: remove unused fwdPackageLogBucket key In this commit, we move the declaration of the key for an unused bucket. In the past, this bucket was used to store the revocation forwarding package log. However, this has been moved under the key `fwdPackagesKey`. * channeldb: fix bucket creation hierarchy in createChannelDB In this commit, we fix a bug in the bucket creation code in createChannelDB. This bug can cause migrations on older nodes to fail, as we expect the bucket to already have been created. With this commit, we ensure that all the buckets under the main node and edges bucket are properly created. Otherwise, a set of the newer migrations will fail to apply for nodes updating from 0.4. * release: Add ARMv6 platform to release.sh * discovery/syncer: use rate limiter for gossip queries This commit replaces the simplistic rate limiting technique added in 557cb6e, to use the golang.org/x/time's rate limiter. This has the benefit of performing traffic shaping to meet a target maximum rate, and yet tolerate bursts. Bursts are expected during initial sync, though should become more rare afterwards. Performing traffic shaping with this mechanism should improve the ability of the gossip syncer to detect sustained bursts from the remote peer, and penalize them appropriately. This commit also modifies the default parameters to accept bursts of 10 queries, with a target rate of 1 reply every 5 seconds. * dep: add golang.org/x/time for rate limiter * lnd_test: make sure we stay disconnected using WaitInvariant in testGarbageCollectLinkNodes * lnd_test: restart nodes twice to trigger pruning logic * cmd/lncli: Fixed conflicting payinvoice amount output * channeldb: relax bucket assumptions for latest db migration In this commit, we no longer assume that the bucket hierarchy has been created properly when applying the latest DB migration. On older nodes that never obtained a channel graph, or updated _before_ the query sync stuff was added, then they're missing buckets that the migration expects them to have. We fix this by simply creating the buckets as we go, if needed. * contractcourt/chain_arbitrator: parallel chain watcher start This commit restructures the initialization procedure for chain watchers such that they can proceed in parallel. This is primarily to help nodes running with the neutrino backend, which otherwise forces a serial rescan for each active channel to check for spentness. Doing so allows the rescans to take advantage of batch scheduling in registering for the spend notifications, ensuring that only one or two passes are made, as opposed to one for each channel. Lastly, this commit ensures that the chain arb is properly shutdown if any of it's chain watchers or channel arbs fails to start, so as to cancel their goroutines before exiting. * tor/controller: add support for v3 onion services In this commit, we extend our Tor controller to also support creating v3 onion services, as they are now supported by the Tor daemon. We also refactor our existing AddOnion method to take in a config struct that houses all of the required options to create/restore an onion service. * tor/controller: add tor version number check In this commit, we add a check for the Tor server's version number to ensure it supports creating v3 onion services through its control port. * config: modify private key path to be onion type agnostic * server: update to lastest tor controller api * docs: update v3 tor docs * build: update dep to latest version of btcwallet * lnwallet: add test to ensure that all backends show unconf txns in ListTransactionDetails In this commit, we add a new test to the set of lnwallet integration tests. In this new test, we aim to ensure that all backends are able to display unconfirmed transactions in ListChainTransactions. As of this commit, this test fails as no backends will return unconfirmed transactions properly. * lnwallet: fix ListTransactionDetails to ensure unconfirmed transactions are installed In this commit, we fix a bug in the arguments to GetTransactions for the btcwallet implementation of the WalletController interface. Before this commit, we wouldn't properly return unconfirmed transactions. The issue was that we didn't specify the special mempool height of "-1", as the ending height. The mempool height is actually internally converted to the highest possible height that can fit into a int32. In this commit, we set the start to zero, and end to -1 (actually 2^32-1) to properly scan for unconfirmed transactions. Fixes lightningnetwork#1422. * lnwallet: update for new API change in NewBitcoindClient * lnwallet: add test to execrise unconfirmed transcations notified by SubscribeTransactions In this commit, we add a new test to ensure that all backends will properly send out notifications when an unconfirmed transcation that we send is inserted into the tx store. Before we updated the btcwallet build commit in dep, this would fail for neutrino but now passes. * multi: update to new BitcoindClient API * config: return an error if unable to parse values in the config file * channeldb: don't use KeyN in latest migration In this commit, we fix a bug in the latest migration that could cause the migration to end in a panic. Additionally, we modify the migration to exit early if the bucket wasn't found, as in this case, no migration is required. Fixes lightningnetwork#1874. * config: fix linter error * channeldb/invoices: extend invoice queries to allow backwards pagination * channeldb/invoice_test: refactor TestQueryInvoices and add reverse test cases * lnrpc: add fields to paginate invoices backwards * rpcserver: include new proto fields in ListInvoices rpc * cmd/lncli: add description and reversed flag to listinvoices * make: clean lnd-debug, lncli-debug and .vendor-new * cmd/lncli: update listinvoices docs with new pagination support * lnrpc: update ListInvoices RPC with new pagination support * discovery: fix logging for gossip syncer rate limiting * lnwire: check length of payload in decodeShortChanIDs * README: add security section in readme * build: update to point to latest version of neutrino In this commit, we update the build to point to the latest version of neutrino which includes a bug fix for a regression that would cause the daemon to spin when at chain tip attempting to always fetch the next set of headers though it was already fully up to date. * channeldb: fix bug in migration from 0.4 to 0.5 In this commit, we fix a bug in the latest database migration when migrating from 0.4 to 0.5. There's an issue in bolt db where if one deletes a bucket that has a key with a nil value, it thinks that's a sub bucket and attempts a bucket deletion. This will fail as it's not actually a sub-bucket. We get around this by using a cursor to manually delete items in the bucket. Fixes lightningnetwork#1907. * fix testing issue with duplicate invoices
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@Roasbeef seemed to indicate on #lightning-dev that any excess data (or padding/null/dummy entries within addresses in node_announcements) in gossip messages will be dropped potentially before signatures are checked. This will potentially mean signatures are rejected (not actually sure on that one), but will definitely mean messages wont propagate.
This could become high-priority pretty fast if any other clients implement lightning/bolts#465 as suddenly the route graph seen by lnd nodes will get pretty sparse.
If the padding/null entries in addresses is removed (as suggested in lightning/bolts#471) implementing this can be as easy as tracking excess data at least pre-relay (ala https://github.com/rust-bitcoin/rust-lightning/pull/136/files#diff-9dc9910c41d9612ff2bf35f2c2b0e6e8R342).
The text was updated successfully, but these errors were encountered: