multi: switch to sat/kw fees #1644

wpaulino · 2018-07-28T02:09:45Z

In this PR, we modify our FeeEstimator interface to return an estimated fee rate in sat/kw. Recently, due to low fees on the network, users have been experiencing failures broadcasting transactions due to not meeting specific fee requirements. This was happening more often than not, as the estimated fee returned by backend nodes (bitcoind and btcd) only takes into account vbytes, rather than weight. The fees returned are also expressed in sat/kb, so we must take care that we do not lose precision while converting to sat/kw. In the event that this happens, a fee floor of 253 sat/kw has been added. This fee rate originates from bitcoind rounding up the conversion from weight to vbytes.

Fixes #1571.
Fixes #1610.

Roasbeef

Did an initial pass through, and the set of changes reads well to me. We'll need to follow up with an additional few rounds of testing to ensure that we're not introducing some weird fee regression as a side effect of this PR.

I also updated the PR description to include the two issues that this directly affects. Once this goes in, I consider those to issues fixed for future users. However, for existing users that have already run into these issues, we may want to create a follow up patch (a component of the utxo pool really) which will simply re-sign if the fee is too low.

halseth · 2018-07-31T08:24:46Z

lnwallet/fee_estimator.go

-	b.minFeeRate = SatPerVByte(relayFee / 1000)
+	// The fee rate is expressed in sat/kb, so we'll manually convert it to
+	// our desired sat/kw rate.
+	minRelayFeePerKw := SatPerVByte(relayFee / 1000).FeePerKWeight()


This will lose some precision.

Why can't we just convert directly to sat/kw and skip the vbyte hop?

halseth · 2018-07-31T08:28:10Z

lnwallet/fee_estimator.go

-// EstimateFeePerVSize takes in a target for the number of blocks until an
-// initial confirmation and returns the estimated fee expressed in
-// satoshis/vbyte.
+// EstimateFeePer takes in a target for the number of blocks until an initial


update name

halseth · 2018-07-31T08:28:56Z

lnwallet/fee_estimator.go

-		satPerByte = b.minFeeRate
+	// Since we use fee rates in sat/kw internally, we'll convert the
+	// estimated fee rate from its sat/kb representation to sat/kw.
+	satPerKw := SatPerVByte(satPerKB / 1000).FeePerKWeight()


should probably create a SatPeKB->SatPerKw method to do this conversion without precision loss.

Actually, I suggest renaming SatPerVByte->SatPerKVByte, then we can do the conversion as is done here (removing the division by 1000 ofc)

halseth · 2018-07-31T08:32:11Z

contractcourt/contract_resolvers.go

+				lnwallet.OfferedHtlcSuccessWitnessSize,
+			)
+			weightEstimator.AddP2WKHOutput()
+			totalWeight := weightEstimator.Weight()


The chaining previously used read pretty nicely IMO :)

halseth · 2018-07-31T08:41:34Z

rpcserver.go


 	paymentMap := map[string]int64{in.Addr: in.Amount}
-	txid, err := r.sendCoinsOnChain(paymentMap, feeRate)
+	txid, err := r.sendCoinsOnChain(paymentMap, feePerVByte)


sendCoinsOnChain should also be modified to take sat/kw. I think it will be converted to sat/KB at some point, where which we can do the conversion.

cfromknecht

nice set of changes! conversion looks solid, my comments are mostly about making the units more explicit

cfromknecht · 2018-08-04T01:40:54Z

lnwallet/fee_estimator.go

+	// EstimateFee takes in a target for the number of blocks until an
+	// initial confirmation and returns the estimated fee expressed in
+	// sat/kw.
+	EstimateFee(numBlocks uint32) (SatPerKWeight, error)


change name to EstimateFeePerKW/EstimateFeePerKWeight

cfromknecht · 2018-08-04T01:45:43Z

lnwallet/fee_estimator.go

@@ -56,13 +67,13 @@ type FeeEstimator interface {
 type StaticFeeEstimator struct {
 	// FeeRate is the static fee rate in satoshis-per-vbyte that will be
 	// returned by this fee estimator.
-	FeeRate SatPerVByte
+	FeeRate SatPerKWeight


while we're changing this, can we set the field name to FeePerKW, so the units are clear in other parts of the code base? would recommend doing the same for btcd and bitcoind estimators

cfromknecht · 2018-08-04T01:47:33Z

chainregistry.go

@@ -37,13 +37,13 @@ const (
 	defaultBitcoinBaseFeeMSat   = lnwire.MilliSatoshi(1000)
 	defaultBitcoinFeeRate       = lnwire.MilliSatoshi(1)
 	defaultBitcoinTimeLockDelta = 144
-	defaultBitcoinStaticFeeRate = lnwallet.SatPerVByte(50)
+	defaultBitcoinStaticFeeRate = lnwallet.SatPerKVByte(50 * 1000)


would change these to be explicitly in SatPerKWeight

Agreed, but a comment with the feerate in sat/b.

halseth · 2018-08-06T10:21:55Z

chainregistry.go

@@ -37,13 +37,13 @@ const (
 	defaultBitcoinBaseFeeMSat   = lnwire.MilliSatoshi(1000)
 	defaultBitcoinFeeRate       = lnwire.MilliSatoshi(1)
 	defaultBitcoinTimeLockDelta = 144
-	defaultBitcoinStaticFeeRate = lnwallet.SatPerVByte(50)
+	defaultBitcoinStaticFeeRate = lnwallet.SatPerKVByte(50 * 1000)


Agreed, but a comment with the feerate in sat/b.

halseth · 2018-08-06T10:23:40Z

rpcserver.go

 	case feePerByte != 0:
-		return lnwallet.SatPerVByte(feePerByte), nil
+		return lnwallet.SatPerKVByte(feePerByte).FeePerKWeight(), nil


must be multiplied by 1000.

Nice catch! Don't blindly search and replace 🤫

Roasbeef · 2018-08-09T02:56:21Z

Needs rebase.

halseth

LGTM 🕺

halseth · 2018-08-09T08:48:56Z

lnwallet/fee_estimator.go

+
+	// By default, we'll use the backend node's minimum relay fee as the
+	// minimum fee rate we'll propose for transacations. However, if this
+	// happens to be lower than our fee floor, we'll enforce that instead.


halseth · 2018-08-09T08:52:43Z

contractcourt/contract_resolvers.go

-			AddP2PKHInput().
-			AddP2WKHOutput().VSize()
-		totalFees := feePerVSize.FeeForVSize(int64(totalVSize))
+		var weightEstimator lnwallet.TxWeightEstimator


nit: could keep the pattern, only changin to Weight()

halseth · 2018-08-09T08:54:42Z

chainregistry.go

 	defaultLitecoinDustLimit     = btcutil.Amount(54600)

+	// defaultBitcoinStaticFeePerKW is the fee rate of 50 sat/vbyte


halseth · 2018-08-09T08:54:57Z

chainregistry.go

 	defaultLitecoinDustLimit     = btcutil.Amount(54600)

+	// defaultBitcoinStaticFeePerKW is the fee rate of 50 sat/vbyte
+	// expressed in sat/kw.
+	//defaultBitcoinStaticFeePerKW = lnwallet.SatPerKVByte(50 * 1000)


halseth · 2018-08-09T08:56:40Z

fundingmanager.go

 	if err != nil {
 		msg.err <- err
 		return
 	}

-	// If the converted fee-per-kw is below the current widely used policy


halseth · 2018-08-09T08:58:04Z

htlcswitch/mock.go


 	quit chan struct{}
 }

-func (m *mockFeeEstimator) EstimateFeePerVSize(numBlocks uint32) (lnwallet.SatPerVByte, error) {
+func (m *mockFeeEstimator) EstimateFeePerKW(numBlocks uint32) (lnwallet.SatPerKWeight, error) {


nit: wrap line?

halseth · 2018-08-09T08:58:28Z

lntest/harness.go

@@ -172,7 +172,7 @@ func (n *NetworkHarness) SetUp(lndArgs []string) error {
 				PkScript: addrScript,
 				Value:    btcutil.SatoshiPerBitcoin,
 			}
-			if _, err := n.Miner.SendOutputs([]*wire.TxOut{output}, 30); err != nil {
+			if _, err := n.Miner.SendOutputs([]*wire.TxOut{output}, 7500); err != nil {


nit: wrap line

halseth · 2018-08-09T08:58:38Z

lntest/harness.go

@@ -1159,7 +1159,7 @@ func (n *NetworkHarness) sendCoins(ctx context.Context, amt btcutil.Amount,
 		PkScript: addrScript,
 		Value:    int64(amt),
 	}
-	if _, err := n.Miner.SendOutputs([]*wire.TxOut{output}, 30); err != nil {
+	if _, err := n.Miner.SendOutputs([]*wire.TxOut{output}, 7500); err != nil {


nit: wrap line?

In this commit, we modify our FeeEstimator interface to return an estimated fee rate in sat/kw. Recently, due to low fees on the network, users have been experiencing failures broadcasting transactions due to not meeting specific fee requirements. This was happening more often than not, as the estimated fee returned by backend nodes (bitcoind and btcd) only takes into account vbytes, rather than weight. The fees returned are also expressed in sat/kb, so we must take care that we do not lose precision while converting to sat/kw. In the event that this happens, a fee floor of 253 sat/kw has been added. This fee rate originates from bitcoind rounding up the conversion from weight to vbytes.

Due to a recent change within the codebase to return estimated fee rates in sat/kw, this commit ensures that we use this fee rate properly by calculing a transaction's fees using its weight. This includes all of the different transactions that are created within lnd (funding, sweeps, etc.). On-chain transactions still rely on a sat/vbyte fee rate since it's required by btcwallet.

In this commit, we explicitly convert sat/vbyte fee rates input by the user to sat/kw. We do this as users are typically more accustomed to sat/vbyte fee rates, rather than sat/kw.

Roasbeef

LGTM 🍡

Roasbeef mentioned this pull request Jul 30, 2018

Force closed channel stuck in pending channels forever #1571

Closed

Roasbeef added fees Related to the fees paid for transactions (both LN and funding/commitment transactions) P2 should be fixed if one has time utxo sweeping needs review PR needs review by regular contributors needs testing PR hasn't yet been actively tested on testnet/mainnet labels Jul 31, 2018

Roasbeef reviewed Jul 31, 2018

View reviewed changes

Roasbeef removed the needs review PR needs review by regular contributors label Jul 31, 2018

halseth reviewed Jul 31, 2018

View reviewed changes

Roasbeef added this to the 0.5 milestone Aug 1, 2018

wpaulino force-pushed the kw-fees branch from 2cb6e2f to af278ed Compare August 1, 2018 19:56

cfromknecht reviewed Aug 4, 2018

View reviewed changes

halseth suggested changes Aug 6, 2018

View reviewed changes

Roasbeef added the needs rebase PR has merge conflicts label Aug 9, 2018

wpaulino force-pushed the kw-fees branch from af278ed to 1f2bb9b Compare August 9, 2018 03:19

halseth previously approved these changes Aug 9, 2018

View reviewed changes

wpaulino added 4 commits August 9, 2018 17:29

multi: update to latest fee estimation interface

9d2eeb6

rpcserver: convert sat/vbyte fee rates input by the user to sat/kw

336f1c1

In this commit, we explicitly convert sat/vbyte fee rates input by the user to sat/kw. We do this as users are typically more accustomed to sat/vbyte fee rates, rather than sat/kw.

wpaulino dismissed halseth’s stale review via 336f1c1 August 10, 2018 00:30

wpaulino force-pushed the kw-fees branch from 1f2bb9b to 336f1c1 Compare August 10, 2018 00:30

Roasbeef approved these changes Aug 10, 2018

View reviewed changes

Roasbeef merged commit d2612e5 into lightningnetwork:master Aug 10, 2018

wpaulino deleted the kw-fees branch August 10, 2018 03:21

		defaultLitecoinDustLimit = btcutil.Amount(54600)

		// defaultBitcoinStaticFeePerKW is the fee rate of 50 sat/vbyte

multi: switch to sat/kw fees #1644

multi: switch to sat/kw fees #1644

Conversation

wpaulino commented Jul 28, 2018 • edited by Roasbeef Loading

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfromknecht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef commented Aug 9, 2018

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef left a comment

Choose a reason for hiding this comment

wpaulino commented Jul 28, 2018 •

edited by Roasbeef

Loading