rebalance overhaul: internal pathfinding, economic counterparties

README.md

@@ -24,7 +24,6 @@ lndmanage is a command line tool for advanced channel management of an
 * Rebalancing command [```rebalance```](#channel-rebalancing)
   * different rebalancing strategies can be chosen
   * a target 'balancedness' can be specified (e.g. to empty the channel)
-* Circular self-payments ```circle```
 * Recommendation of good nodes [```recommend-nodes```](#channel-opening-strategies)
 * Batched channel opening [```openchannels```](#batched-channel-opening)
 * Support of [```lncli```](#lncli-support)
@@ -45,7 +44,6 @@ positional arguments:
     listchannels        lists channels with extended information [see also subcommands with -h]
     listpeers           lists peers with extended information
     rebalance           rebalance a channel
-    circle              circular self-payment
     recommend-nodes     recommends nodes [see also subcommands with -h]
     report              displays reports of activity on the node
     info                displays info on channels and nodes
@@ -141,45 +139,27 @@ Forwardings:
 ```
 
 ## Channel Rebalancing
-The workflow for rebalancing a channel goes as follows:
-
-* take a look at all your unbalanced channels with:
-
-  ```$ lndmanage listchannels rebalance```
-
-    The output will look like:
-```
-   -------- Description --------
-  cid        channel id
-  ub         unbalancedness [-1 ... 1] (0 is 50:50 balanced)
-  cap        channel capacity [sat]
-  lb         local balance [sat]
-  rb         remote balance [sat]
-  pbf        peer base fee [msat]
-  pfr        peer fee rate
-  annotation channel annotation
-  alias      alias
-  -------- Channels --------
-         cid            ub       cap        lb        rb    pbf       pfr alias
-  xxxxxxxxxxxxxxxxxx -0.78   1000000    888861     99480     10  0.000200   abc                
-  xxxxxxxxxxxxxxxxxx -0.63   1000000    814537    173768    300  0.000010   def
-  xxxxxxxxxxxxxxxxxx  0.55   2000000    450792   1540038     35  0.000002   ghi
-  xxxxxxxxxxxxxxxxxx  0.59    400000     81971    306335    400  0.000101   jkl
-  ...
-```
+Channels with depleted liquidity can be rebalanced with channels of saturated liquidity by circular self-payments, that is, a payment leaves from the saturated channel and arrives in the depleted channel. This operation has the cost of paying routing fees. Another often overlooked cost is *state accumulation*. Rebalancing is a time consuming process where many attemps are needed, because one is often trying to counter the natural flow within the network. This can lead to enlarged databases and therefore overall slower operations and strain on the network, which is why rebalancing should be used sparingly!
+
+Which channels could be rebalanced? Typically, we want to rebalance channels that give us high fee earning expecations [to see those channels, see `$ lndmanage listchannels forwardings`, fields `fo/w` (fees earned per week)  and `fi/w` (fees earned by other channels due to incoming payments per week)]. We want to increase local balance in channels with high `fo/w` and decrease local balance in channels with high `fi/w`. Another use case is to increase/decrease local balance for new channels to see whether there's demand for forwarding. Otherwise it is not helpful to perfectly rebalance a channel 50:50 in most cases.
+
+The workflow to rebalance a channel is as follows:
+* ideally your channels have meaningful fee rates set (see `$ lndmanage update-fees`)
+* the counterparty rebalance channels are chosen according to the expected earnings calculated from the fee rate that is set for the to-be-rebalanced channel, i.e., a channel with a low fee rate will send its liquidity to channels with high fee rates, such that this capital can earn fees in the future and vice-versa
+* take a look at all your channels with forwarding activity and select a channel with high `fo/w` or `fi/w`, or high absolute values of `ub`
+
 * the ```ub``` field tells you how unbalanced your channel is 
-  and in which direction
-* take a channel_id from the list you wish
-  to rebalance (target is a 50:50 balance)
+  and in which direction (negative means that the channel has mostly local balance - you may want to decrease the local balance)
+* take a `channel_id` or `node_id` you wish to rebalance
+* decide on whether you want to increase (positive `--amount`) or decrease your local balance (negative `--amount`), chose small amounts (< 1000000 sat) for better success
+* if `--amount` is not set, the sign of liquidity change is determined automatically
 * do a dry run to see what's waiting for you
 
-  ```$ lndmanage rebalance --max-fee-sat 20 --max-fee-rate 0.00001 channel_id```
+  ```$ lndmanage rebalance channel_id/node_id```
 
 * read the output and if everything looks well, 
   then run with the ```--reckless``` flag
-* in order to increase the success probability of your rebalancing you
-  can try to do it in smaller chunks, which can be set by the flag
-  `--chunksize 0.5` (in this example only half the amounts are used)
+* if you want to disable the fee rate selection of counterparty channels to increase success probability, run with the `--force` flag
 
 ## Forwarding Information
 A more sophisticated way to see if funds have to be reallocated is to 

lndmanage/lib/exceptions.py

@@ -22,23 +22,15 @@ class RebalanceFailure(Exception):
     pass
 
 
-class RoutesExhausted(RebalanceFailure):
-    pass
-
-
-class RebalanceCandidatesExhausted(RebalanceFailure):
-    pass
-
-
 class NoRebalanceCandidates(RebalanceFailure):
     pass
 
 
-class RebalancingTrialsExhausted(RebalanceFailure):
+class NotEconomic(RebalanceFailure):
     pass
 
 
-class MultichannelInboundRebalanceFailure(RebalanceFailure):
+class RebalancingTrialsExhausted(RebalanceFailure):
     pass
 
 
@@ -76,6 +68,12 @@ def __init__(self, payment):
         super().__init__()
 
 
+class TemporaryNodeFailure(PaymentFailure):
+    def __init__(self, payment):
+        self.payment = payment
+        super().__init__()
+
+
 class UnknownNextPeer(PaymentFailure):
     def __init__(self, payment):
         self.payment = payment
@@ -88,6 +86,12 @@ def __init__(self, payment):
         super().__init__()
 
 
+class IncorrectCLTVExpiry(PaymentFailure):
+    def __init__(self, payment):
+        self.payment = payment
+        super().__init__()
+
+
 class ChannelDisabled(PaymentFailure):
     def __init__(self, payment):
         self.payment = payment

lndmanage/lib/forwardings.py

@@ -7,7 +7,7 @@
 
 from lndmanage.lib.data_types import NodeProperties
 from lndmanage.lib.node import LndNode
-from lndmanage.lib.ln_utilities import channel_unbalancedness_and_commit_fee
+from lndmanage.lib.ln_utilities import local_balance_to_unbalancedness
 from lndmanage import settings
 
 logger = logging.getLogger(__name__)
@@ -44,7 +44,7 @@ def initialize_forwarding_stats(self, time_start: float, time_end: float):
         :param time_start: time interval start, unix timestamp
         :param time_end: time interval end, unix timestamp
         """
-        channel_id_to_node_id = self.node.get_channel_id_to_node_id()
+        channel_id_to_node_id = self.node.channel_id_to_node_id()
         self.channel_forwarding_stats = defaultdict(ForwardingStatistics)
         self.node_forwarding_stats = defaultdict(ForwardingStatistics)
 
@@ -565,7 +565,7 @@ def get_node_properites(
         f"Time interval (between first and last forwarding) is "
         f"{forwarding_analyzer.max_time_interval_days:6.2f} days."
     )
-    channel_id_to_node_id = node.get_channel_id_to_node_id(open_only=True)
+    channel_id_to_node_id = node.channel_id_to_node_id(open_only=True)
     node_ids_with_open_channels = {nid for nid in channel_id_to_node_id.values()}
     open_channels = node.get_open_channels()
 
@@ -641,7 +641,7 @@ def get_node_properites(
             "max_public_capacity": max(properties.public_capacities)
             if properties.public_capacities
             else 0,
-            "unbalancedness": channel_unbalancedness_and_commit_fee(
+            "unbalancedness": local_balance_to_unbalancedness(
                 local_balance, capacity, 0, False
             )[0],
         }
@@ -688,6 +688,8 @@ def get_node_properites(
         except ZeroDivisionError:
             node_properties_forwardings[node_id]["fees_in_per_week"] = float("nan")
 
+    # TODO: unify with information from liquidity hints
+
     return node_properties_forwardings
 
 

lndmanage/lib/info.py

@@ -1,9 +1,12 @@
-import re
 import datetime
-from typing import Tuple
+from typing import Dict
 
 from lndmanage.lib.network_info import NetworkAnalysis
-from lndmanage.lib import ln_utilities
+from lndmanage.lib.ln_utilities import (
+    parse_nodeid_channelid,
+    convert_channel_id_to_short_channel_id,
+    height_to_timestamp
+)
 from lndmanage import settings
 
 import logging
@@ -64,7 +67,7 @@ def parse_and_print(self, info):
 
         # analyzer = NetworkAnalysis(self.node)
         try:
-            channel_id, node_pub_key = self.parse(info)
+            channel_id, node_pub_key = parse_nodeid_channelid(info)
         except ValueError:
             logger.info("Info didn't represent neither a channel nor a node.")
             return
@@ -84,9 +87,9 @@ def parse_and_print(self, info):
             general_info['node2_alias'] = \
                 self.node.network.node_alias(general_info['node2_pub'])
             general_info['blockheight'] = \
-                ln_utilities.convert_channel_id_to_short_channel_id(
+                convert_channel_id_to_short_channel_id(
                     channel_id)[0]
-            general_info['open_timestamp'] = ln_utilities.height_to_timestamp(
+            general_info['open_timestamp'] = height_to_timestamp(
                 self.node, general_info['blockheight'])
 
             # TODO: if it's our channel, add extra info
@@ -106,46 +109,11 @@ def parse_and_print(self, info):
 
             self.print_node_info(general_info)
 
-    def parse(self, info: str) -> Tuple[int, str]:
-        """Parse whether info contains a channel id or node public key and hand
-        it back. If no info could be extracted, raise a ValueError.
-
-        :return: channel_id, node_pub_key
-        """
-        exp_channel_id = re.compile("^[0-9]{13,20}$")
-        exp_short_channel_id = re.compile("^[0-9]{6}x[0-9]{3}x[0-9]$")
-        exp_chan_point = re.compile("^[a-z0-9]{64}:[0-9]$")
-        exp_node_id = re.compile("^[a-z0-9]{66}$")
-
-        channel_id = None
-        node_pub_key = None
-
-        # prepare input string info
-        if exp_channel_id.match(info) is not None:
-            logger.debug("Info represents channel id.")
-            channel_id = int(info)
-        elif exp_short_channel_id.match(info) is not None:
-            logger.debug("Info represents short channel id.")
-            # TODO: convert short channel id to channel id
-            channel_id = 0
-        elif exp_chan_point.match(info) is not None:
-            # TODO: convert chan point to channel id
-            logger.debug("Info represents short channel id.")
-            channel_id = 0
-        elif exp_node_id.match(info) is not None:
-            logger.debug("Info represents node public key.")
-            node_pub_key = info
-        else:
-            raise ValueError("Info string doesn't match any pattern.")
-
-        return channel_id, node_pub_key
-
-    def print_channel_info(self, general_info):
+    def print_channel_info(self, general_info: Dict):
         """
         Prints the channel info with peer information.
 
         :param general_info: information about the channel in the public graph
-        :type general_info: dict
         """
 
         logger.info("-------- Channel info --------")
@@ -189,8 +157,9 @@ def print_channel_info(self, general_info):
         logger.info(f"{general_info['node1_alias']:^{COL_WIDTH}} | "
                     f"{general_info['node2_alias']:^{COL_WIDTH}}")
 
-        np1 = general_info['node1_policy']
-        np2 = general_info['node2_policy']
+        policies = general_info['policies']
+        np1 = policies[general_info['node1_pub'] > general_info['node2_pub']]
+        np2 = policies[general_info['node2_pub'] > general_info['node1_pub']]
         last_update_1 = np1['last_update']
         last_update_2 = np2['last_update']