BOLT 12 `offer` encoding and building #1719

jkczyz · 2022-09-13T22:08:33Z

Define an interface for BOLT 12 offer messages. The underlying format consists of the original bytes and the parsed contents.

The bytes are later needed when constructing an invoice_request message. This is because it must mirror all the offer TLV records, including unknown ones, which aren't represented in the contents.

The contents will be used in invoice_request messages to avoid duplication. Some fields while required in a typical user-pays-merchant flow may not be necessary in the merchant-pays-user flow (i.e., refund).

Also, defines a builder for constructing an offer given a required description and node id.

See #1597 for ongoing BOLT 12 work depending on this PR.

codecov-commenter · 2022-09-13T23:07:35Z

Codecov Report

Base: 90.79% // Head: 90.80% // Increases project coverage by +0.01% 🎉

Coverage data is based on head (e673125) compared to base (8f525c4).
Patch coverage: 94.52% of modified lines in pull request are covered.

❗ Current head e673125 differs from pull request most recent head 20da3f3. Consider uploading reports for the commit 20da3f3 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1719      +/-   ##
==========================================
+ Coverage   90.79%   90.80%   +0.01%     
==========================================
  Files          87       89       +2     
  Lines       47604    47971     +367     
  Branches    47604    47971     +367     
==========================================
+ Hits        43221    43562     +341     
- Misses       4383     4409      +26

Impacted Files	Coverage Δ
lightning/src/lib.rs	`100.00% <ø> (ø)`
lightning/src/ln/channelmanager.rs	`85.41% <ø> (ø)`
lightning/src/onion_message/packet.rs	`76.03% <ø> (ø)`
lightning/src/util/ser_macros.rs	`88.40% <80.00%> (-0.41%)`	⬇️
lightning/src/util/ser.rs	`91.80% <91.66%> (+0.13%)`	⬆️
lightning/src/offers/offer.rs	`94.59% <94.59%> (ø)`
lightning/src/ln/features.rs	`99.67% <100.00%> (+<0.01%)`	⬆️
lightning/src/onion_message/blinded_route.rs	`96.90% <100.00%> (+0.24%)`	⬆️
lightning/src/routing/gossip.rs	`92.48% <100.00%> (-0.03%)`	⬇️
lightning/src/util/events.rs	`37.62% <100.00%> (ø)`
... and 5 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

TheBlueMatt · 2022-09-14T18:08:48Z

lightning/src/util/ser_macros.rs

+		u8
+	};
+	(u16) => {
+		::util::ser::HighZeroBytesDroppedBigSize<u16>


I'm a bit uneasy about putting these directly in a struct - can we not just keep using them as wrappers at serialization-time rather than actually storing them anywhere?

Pushed a version that only uses them for decoding / encoding. It does remove a lot of intos, which is nice. Though I'd be more uneasy that the macros don't read / write correctly, whereas mistakes in the "struct using wrapper" approach can by caught be the compiler, FWIW.

But given its all hidden by a struct-defining macro we end up in the same place, no?

Yeah, mostly just saying more care needs to be taken when implementing and updating the macros. While making the change, I had an intermediate state where encoding and decoding were not compatible.

TheBlueMatt · 2022-09-14T18:10:27Z

lightning/src/util/ser_macros.rs

+				tlv_record_import!($fieldty$(<$gen>)?);
+			)*
+
+			pub(crate) struct $name<'a> {


Can't we just name this something slightly different rather than putting it in a module and having to guess the import path required (by listing the Rust prelude explicitly in our code)?

Mainly was trying to keep the usage site looking like a struct. Made it explicit now. Let me know if that is what you were thinking of.

TheBlueMatt · 2022-09-14T18:11:19Z

lightning/src/util/ser.rs

+impl<'a> From<&'a String> for WithoutLength<&'a String> {
+	fn from(s: &'a String) -> Self { Self(s) }
+}
+impl From<WithoutLength<String>> for String {


If we make these serialization-only wrappers we can drop the copy versions, which seems like the right thing to do.

We'd still need this to simplify deserialization, though it should be a move. I'd imagine it's equivalent to explicitly using .0.

lightning/src/util/ser.rs

lightning/src/offers/offer.rs

valentinewallace · 2022-09-14T18:58:31Z

lightning/src/offers/offer.rs

+
+//! Data structures and encoding for `offer` messages.
+//!
+//! An [`Offer`] is built by the merchant for the user-pays-merchants flow or may be parsed from a


Offers are also good for user-pays-user "Cash App" flow, would be nice to amend the docs to include this

The "user-pays-merchant" terminology is from the BOLT and is used to differentiate from "merchant-pays-user" (e.g., refunds, ATM) flow, which I use in a future commit. I could use "recipient" instead of the first use of "merchant" here to make it more generic. But not sure if I want to deviate from the spec by adding a new "flow". Open to other ideas so long as we can make it work with the invoice_request docs (see #1597 for current draft), which will be expanded with another builder for the "merchant-pays-user" flow.

Maybe we could do payer-offers-money vs payer-pays-recipient (maybe the word "offer" is confusing there)? ATM isn't really captured by "merchant" either.. I think we've historically not cared too much about sticking with spec naming/copy if there's a reason to deviate, like with wumbo, scid_privacy vs alias, etc.

Maybe we could do payer-offers-money vs payer-pays-recipient (maybe the word "offer" is confusing there)?

Hmm... payer-pays-recipient seems rather tautological. The "payer" is always the one paying a recipient. 😛

I guess "recipient" was a poor suggestion given we want to distinguish who is getting paid. Really, we are trying to distinguish who creates the "offer" (loosely since it could be an invoice_request offer) from who pays the invoice. Or even who creates the invoice.

In the user-pays-merchant flow, the merchant creates both the offer and the invoice. So the user pays the invoice.

In the merchant-pays-user flow, the merchant creates the "offer" but the user creates the invoice. So the merchant pays the invoice.

With that in mind, referring to them as "user" and "merchant" seems orthogonal to what we are trying to convey. That is, relative to who created the "offer", who is creating (or paying) the invoice? Or even, is the offer for a product/service or is the "offer" for money?

ATM isn't really captured by "merchant" either..

Well, the merchant owns the ATM, which accepts fiat and is using lighting to deliver the product (i.e. bitcoin). So I think the ATM and the refund cases are still that of the merchant paying bitcoin.

I think we've historically not cared too much about sticking with spec naming/copy if there's a reason to deviate, like with wumbo, scid_privacy vs alias, etc.

True, though we could contribute back if we have a better alternative in this case. Note that the BOLT says:

Here we use "user" as shorthand for the individual user's lightning node and "merchant" as the shorthand for the node of someone who is selling or has sold something.

Not sure if "individual user's lightning node" is really any clearer, though.

From my PoV, I'm fine with the BOLT copy because it's protocol-dev-facing and gets the point across. For wallet-dev-facing documentation, though, IMO there are different requirements and it'd be best to be inclusive of the range of use cases.

Given this, maybe we could have an expanded Use Cases section in the top-level docs, which include the user-pays-merchant/ATM/etc, and Offer could be documented as "an offer to be paid" and InvoiceRequest as "an offer of money", rather than each referencing a specific "flow"?

From my PoV, I'm fine with the BOLT copy because it's protocol-dev-facing and gets the point across. For wallet-dev-facing documentation, though, IMO there are different requirements and it'd be best to be inclusive of the range of use cases.

Could you enumerate these use cases?

Given this, maybe we could have an expanded Use Cases section in the top-level docs, which include the user-pays-merchant/ATM/etc, and Offer could be documented as "an offer to be paid" and InvoiceRequest as "an offer of money", rather than each referencing a specific "flow"?

Yeah, I agree that the offers top-level module should have an expanded explanation of use cases and how the various types relate to them. I'd prefer to wait until all the types exist before writing this.

In the offer submodule, I'd like to stick to showing basic example code, though, for the specific types involved. I've updated the wording a bit. Let me know what you think.

The updates look good, and that plan sgtm! The missing use case from my PoV was the Cash App flow (wouldn't want to exclude the people getting tattoos lol).

lightning/src/offers/offer.rs

TheBlueMatt

Basically looks good.

lightning/src/offers/offer.rs

TheBlueMatt · 2022-09-15T15:10:23Z

lightning/src/offers/offer.rs

+	}
+
+	/// The recipient's public key used to sign invoices.
+	pub fn node_id(&self) -> PublicKey {


Do we want to differentiate the naming here given we should encourage users to sign invoices with a different key than their node id?

Maybe signing_node_id?

Maybe just signing_public_key or so? I don't think we want to connect it to node ids at all?

Changed but left as node_id in the TLV stream to match the spec, for now.

TheBlueMatt · 2022-09-15T15:15:56Z

lightning/src/offers/offer.rs

+	}
+
+	/// Builds an [`Offer`] from the builder's settings.
+	pub fn build(self) -> Offer {


We should sign the offer here, right?

Offers no longer have signatures. The node_id is for signing invoices.

TheBlueMatt · 2022-09-15T20:01:23Z

Needs rebase on the features changes, should be easy.

jkczyz · 2022-09-15T15:40:32Z

lightning/src/offers/offer.rs

+	}
+
+	/// Builds an [`Offer`] from the builder's settings.
+	pub fn build(self) -> Offer {


Offers no longer have signatures. The node_id is for signing invoices.

jkczyz · 2022-09-15T15:41:23Z

lightning/src/offers/offer.rs

+	}
+
+	/// The recipient's public key used to sign invoices.
+	pub fn node_id(&self) -> PublicKey {


Maybe signing_node_id?

lightning/src/offers/offer.rs

jkczyz · 2022-09-15T20:53:35Z

lightning/src/offers/offer.rs

+impl Offer {
+	/// The chain used for paying the invoice.
+	pub fn chain(&self) -> ChainHash {
+		// TODO: Update once spec is finalized


@TheBlueMatt FYI, I don't think there was any resolution on this: lightning/bolts#798 (comment). Though I'm not sure how you are thinking about negotiating this given the sender would need to know what chains the recipient supports. Currently, the sender states which chain in the invoice_request based on what's in the offer. If it were left out, wouldn't you need another roundtrip?

FYI, also updated this to return a Vec and made some changes to rust-bitcoin to allow us to return a slice.

I was figuring the invoice would simply contain a list of supported chains and the sender would just pick which they want to use from that list.

Don't you need a chain to figure out how to send the invoice_request in the first place?

jkczyz · 2022-09-16T21:46:07Z

Pushed some changes mostly rearranging some of the fixups and cleaning up some things that had been missed.

jkczyz · 2022-09-19T00:17:24Z

Needs rebase on the features changes, should be easy.

Rebased

lightning/src/offers/offer.rs

valentinewallace · 2022-09-19T16:33:40Z

lightning/src/offers/offer.rs

+
+//! Data structures and encoding for `offer` messages.
+//!
+//! An [`Offer`] is built by the merchant for the user-pays-merchants flow or may be parsed from a


Maybe we could do payer-offers-money vs payer-pays-recipient (maybe the word "offer" is confusing there)? ATM isn't really captured by "merchant" either.. I think we've historically not cared too much about sticking with spec naming/copy if there's a reason to deviate, like with wumbo, scid_privacy vs alias, etc.

lightning/src/offers/offer.rs

valentinewallace · 2022-09-20T18:07:50Z

lightning/src/offers/offer.rs

+	///
+	/// Successive calls to this method will override the previous setting.
+	#[cfg(test)]
+	pub fn features(mut self, features: OfferFeatures) -> Self {


Is the plan to replace this with a basic_mpp() method when that feature bit is resolved?

I think so, but I do wonder if we'd prefer to default it somehow. Thoughts?

Hm, I guess I don't have a strong preference between a setter and an un-setter.

Oh, just reading through the updated BOLT 12 draft. There is a separate set of features for each message, so we only need to worry about this in the InvoiceBuilder.

lightning/src/offers/offer.rs

valentinewallace · 2022-09-20T18:45:44Z

lightning/src/offers/offer.rs

+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+//! Data structures and encoding for `offer` messages.


"Message" implies it's a p2p message to me, is there a way to disambiguate?

Meh... it's splitting hairs a bit, IMHO. invoice_request and invoice aren't really p2p messages either since they are delivered as part of an onion message. And invoice_request in the "send invoice" case doesn't use p2p either. But all of them are messages in a general sense. offer just has a different transport mechanism than the others. 😛

Do you have any alternative phrasing in mind?

valentinewallace · 2022-09-20T18:49:04Z

lightning/src/offers/offer.rs

+	metadata: Option<Vec<u8>>,
+	amount: Option<Amount>,
+	description: String,
+	features: Option<OfferFeatures>,


Could this just be empty() and then set features based on setters?

I preferred using Options otherwise special handling is needed for serializing empty features / Vecs. That said, for Option<Vec<_>> we could hide the Option from the API by returning a slice if Some or the empty slice if None. Similarly with Option<String> using &str. Unfortunately, I don't think we can do the same with OfferFeatures without returning a temporary. 😕

Do we prefer to hide the Option whenever possible? Seems it may make the API somewhat inconsistent but I'm open to the idea.

Could you clarify how that makes the API inconsistent? Seems a bit cleaner to avoid returning an Option where it makes sense. I thought the OfferTlvStreamRef could work around the serialization issues but maybe I'm missing something

Could you clarify how that makes the API inconsistent? Seems a bit cleaner to avoid returning an Option where it makes sense.

There's a few questions that we'd have to answer. If we return &OfferFeatures instead of Option<&OfferFeature>, should we return

&str or Option<&String>?

&[T] or Option<&Vec<T>>

u64 or Option<&Amount>

Duration or Option<Duration>

I can see the argument for &[T], but I'm not as sure about &str. Is it better for the user to check against None or the empty string to determine if a string is set. Is it ok to treat the empty string as not set, and vice versa?

For amount, we'll eventually need an exchange rate interface, so this could work for each variant. But for None, should we return 0? That said, presumably a wallet would want to display the offer's native currency, so would likely need to stick with Option or add a third variant to Amount for None.

Should we stick with Option for absolute expiry or return the maximum duration if None?

I thought the OfferTlvStreamRef could work around the serialization issues but maybe I'm missing something

Yeah, I suppose it could by doing:

let features = if self.features == OfferFeatures::empty() { None } else { Some(&self.features) };

But that raises a secondary concern. When should Option be used in OfferContents? In some places like amount it probably should be. But what about issuer or paths? We'd have to be comfortable with treating an empty TLV value as the same as if the TLV record had not been set. This also means an encoded TLV might not be equal with it after parsed and re-encoded because presumably we wouldn't write the record for empty values. Similarly for features.

My intuition is to stick with Option whenever possible, especially if InvoiceRequest or Invoice need to check against the presence. But we probably can go without Option around Vec and OfferFeatures with the above caveat in mind, assuming we're ok with additional conversion logic in as_tlv_stream here and try_from in later PRs. I'm non sure about String, regardless of what the API ultimately looks like.

Let me know what you think.

Ah, since we already don’t return Options for all getters I didn’t think of that as much of an API inconsistency(?).

u64 or Option<&Amount>

Duration or Option

I thought these two could make sense as options still. Certainly expiry being None makes sense IIUC? Fair point that amount could be weird though, is it possible to make Amounts contain NonZeroU64s?

&str or Option<&String>?

IIUC that’s only for issuer, IMO that one arguably makes sense as an Option<&String> for the reasons you stated, no strong feelings though.

features was the only one where I couldn’t see a reason for it to be None vs Some(OfferFeatures::empty() and therefore seemed weird in the public API.

We could get rid of the features() getter entirely and only have individual methods for individual features (offer.supports_mpp()). That way it could stay an Option under the hood so that parsing-and-reencoding always gets the same result. I admit I hadn’t considered that situation though.

Although, I thought since we save the offer bytes, parsing and re-encoding will always be the same anyway? Lmk what I'm missing, not sure I understood your point about how sticking with Option benefits InvReq/Inv

Ah, since we already don’t return Options for all getters I didn’t think of that as much of an API inconsistency(?).

Yes, very true.

u64 or Option<&Amount>

Duration or Option

I thought these two could make sense as options still. Certainly expiry being None makes sense IIUC? Fair point that amount could be weird though, is it possible to make Amounts contain NonZeroU64s?

Hadn't thought about NonZeroU64 for amount. Honestly, it's a little tough to work with, and the BOLT doesn't seem to care if it is zero. Just says it's a "minimum".

&str or Option<&String>?

IIUC that’s only for issuer, IMO that one arguably makes sense as an Option<&String> for the reasons you stated, no strong feelings though.

Yeah, just wasn't sure if we should try to do something similar to Vecs. I think I'm learning towards Option but Option<&str>, which removes some boilerplate from tests.

features was the only one where I couldn’t see a reason for it to be None vs Some(OfferFeatures::empty() and therefore seemed weird in the public API.

We could get rid of the features() getter entirely and only have individual methods for individual features (offer.supports_mpp()). That way it could stay an Option under the hood so that parsing-and-reencoding always gets the same result. I admit I hadn’t considered that situation though.

Let's use OfferFeatures::empty() without the Option for now. Could be nice to know if there are odd, unknown features.

Although, I thought since we save the offer bytes, parsing and re-encoding will always be the same anyway? Lmk what I'm missing, not sure I understood your point about how sticking with Option benefits InvReq/Inv

Ah, right! Literally ran into this yesterday when adding parsing failure tests in the next PR. Had to manually modify the TLV stream to make the test fail. 😛

Lmk what I'm missing, not sure I understood your point about how sticking with Option benefits InvReq/Inv

Sometime we need to check if a field in the Offer is present to determine how to handle the InvoiceRequest (e.g., amount). During parsing, it's simplest to convert the OfferTlvStream to OfferContents first before performing these checks. So having the Option in OfferContents can be useful.

lightning/src/offers/offer.rs

lightning/src/onion_message/mod.rs

valentinewallace · 2022-09-22T13:52:11Z

lightning/src/offers/offer.rs

+	///
+	/// Successive calls to this method will override the previous setting.
+	#[cfg(test)]
+	pub fn features(mut self, features: OfferFeatures) -> Self {


Hm, I guess I don't have a strong preference between a setter and an un-setter.

valentinewallace · 2022-09-22T14:03:22Z

lightning/src/offers/offer.rs

+
+//! Data structures and encoding for `offer` messages.
+//!
+//! An [`Offer`] is built by the merchant for the user-pays-merchants flow or may be parsed from a


The updates look good, and that plan sgtm! The missing use case from my PoV was the Cash App flow (wouldn't want to exclude the people getting tattoos lol).

lightning/src/offers/offer.rs

valentinewallace · 2022-09-22T14:37:52Z

lightning/src/offers/offer.rs

+	metadata: Option<Vec<u8>>,
+	amount: Option<Amount>,
+	description: String,
+	features: Option<OfferFeatures>,


Could you clarify how that makes the API inconsistent? Seems a bit cleaner to avoid returning an Option where it makes sense. I thought the OfferTlvStreamRef could work around the serialization issues but maybe I'm missing something

lightning/src/offers/offer.rs

lightning/src/util/ser.rs

jkczyz · 2022-09-22T23:02:38Z

@valentinewallace Offer API changes are in 7e932e0. Let me know if you'd prefer that I revert any of them. (cc: @TheBlueMatt)

lightning/src/offers/offer.rs

lightning/src/util/ser.rs

lightning/src/util/ser_macros.rs

lightning/src/offers/offer.rs

valentinewallace · 2022-09-23T19:04:24Z

This is looking pretty good to me, I think it would be good timing for a second reviewer to come in and I'll check out #1726 in the meantime

TheBlueMatt

Some docs need more detail, but is looking good.

lightning/src/offers/offer.rs

TheBlueMatt · 2022-09-26T11:16:36Z

lightning/src/offers/offer.rs

+pub type CurrencyCode = [u8; 3];
+
+tlv_stream!(OfferTlvStream, OfferTlvStreamRef, {
+	(2, chains: Vec<ChainHash>),


If these are all required, should we do something like we do for lightning-invoice in the builder to make it a compile-time error to miss a field?

It depends on the context, though note that all the records use Option in the TLV structs.

For Offer, only description and node_id are required. This is currently enforced at compile-time by OfferBuilder::new requiring both as parameters.

However, for InvoiceRequest used as a "send invoice" (e.g., refund), node_id MUST NOT be set.

I also avoided the lightning-invoice-style compile time checks because IIUC it can't be easily done in bindings since each builder function returns a different type, essentially.

lightning/src/offers/offer.rs

TheBlueMatt · 2022-09-26T11:17:01Z

lightning/src/offers/offer.rs

+			.unwrap_or_else(|| vec![ChainHash::using_genesis_block(Network::Bitcoin)])
+	}
+
+	/// Metadata set by the originator. Useful for authentication and validating fields.


Some of the above comments apply in the getters too.

Ended up expanding comments in Offer and linking to those methods in the OfferBuilder docs to avoid repetition.

TheBlueMatt · 2022-09-26T11:23:51Z

Oh feel free to squash, as far as I care.

TheBlueMatt · 2022-09-26T15:00:55Z

lightning/src/util/ser_macros.rs

+		u8
+	};
+	(u16) => {
+		::util::ser::HighZeroBytesDroppedBigSize<u16>


Another question here is how much magic we want in the macro - instead of having to write out a list of type -> wrapper structs here, we could just do it at the callsite, ensuring the callsite readably describes how things are being serialized, rather than it being hidden? That will also avoid having a list of things that could end up randomly doing the Wrong Thing in a surprising way.

Are you suggesting we expand encode_tlv (the call site) such that there is a tlv_record matcher for each of these types? Could do that, but it would result in a lot of repetitive boilerplate. Also, wouldn't prevent doing thing surprising given the same matchers here would have to be used there.

Note that these three helper macros are only used in one place each. I think I could define them as nested macros at the respective calls sites. But then relative to each other they would be spread all about, and it's kinda nice having them all in one place.

Well, I'm still a bit lost on why we need a new tlv_record thing at all - if all the fields are Option wrapped, everything should be option? Then its just a matter of having the tlv_stream macro understand what the field is, which would presumably mean passing u8, HighZeroBytesDroppedBigSize as separate parameters.

ie that the tlv_stream macro will then wrap the self.$field as $wrapper(self.$field).

(Note for discussion that u8 doesn't require HighZeroBytesDroppedBigSize since it is only one byte.)

Hmm... not sure if that's any better or all that straightforward. We'd still need special handling to distinguish between records that need a wrapper from those that don't (e.g., u8, char, and user-defined struct) when defining fields in each of the two TLV stream structs in the macro. This would require another (small) helper macro. Further, for the reference version, we'd still need to handle primitives differently, so tlv_record_ref_type would be needed still.

Additionally, for decoding (opposed to encoding), we aren't wrapping but rather specifying the type that must be used with Readable and then pulling out the inner value.

To your original question, the idea was for the tlv_stream macro to abstract away the encoding requirement of using minimal representations for integers and avoiding duplicating the length of strings and vectors. Seems making the caller explicitly state these puts the onus on them needlessly.

Putting vec_type aside, I can iterate on this to get rid of the tlv_record literal in favor of some variation of option that also takes a wrapper. This is pretty straightforward for encode_tlv. But for decode_tlv we already have a variation for specifying which Readable* trait along with optional args. So the question here is how do we want to formulate the matchers such that a $fieldty:tt can be passed through decode_tlv_stream to decode_tlv and handle either case.

Not quite sure I follow what you are thinking here. Currently, tlv_record assumes the underlying field is an Option, but generally that is not the case and specifically is not the case for vec_type.

Ah, right, I confused myself because its read into an Option but then unwrap'd like the required stuff is.

I can iterate on this to get rid of the tlv_record literal in favor of some variation of option that also takes a wrapper. This is pretty straightforward for encode_tlv

Yep, I'm pretty sure it is, given you already did :)

But for decode_tlv we already have a variation for specifying which Readable* trait along with optional args. So the question here is how do we want to formulate the matchers such that a $fieldty:tt can be passed through decode_tlv_stream to decode_tlv and handle either case.

Right, I don't think we want to pass the $fieldty:tt through, I think we'll want to split it like we do for the _tlv_stream_impl_writeable and then pass the wrapper through without the type itself. I had a patch to do this, but dropped it as I'd ended up rewriting more than necessary but it was pretty easy.

Right, I don't think we want to pass the $fieldty:tt through, I think we'll want to split it like we do for the _tlv_stream_impl_writeable and then pass the wrapper through without the type itself. I had a patch to do this, but dropped it as I'd ended up rewriting more than necessary but it was pretty easy.

Ah, I mean how should we pass it through to disambiguate from the matcher taking a trait:

($reader: expr, $field: ident, (option: $trait: ident $(, $read_arg: expr)?)) => {{

There, (option: $trait: ident $(, $read_arg: expr)?) is a token tree needed by decode_tlv_stream. I get that we want to pull the wrapper from $fieldty:tt, as you stated, before giving it to decode_tlv_stream. My question is more about what the matcher should look like in decode_tlv to differentiate it from the above matcher.

Could just be something like:

($reader: expr, $field: ident, (option, $wrapper:ident)) => {{

But the only difference between that and the earlier one is using a comma instead of a colon and dropping the optional read arg, which isn't very intuitive. Maybe use a wrapper literal?

($reader: expr, $field: ident, (option, wrapper: $wrapper:ident)) => {{

Ah, yea, no super strong opinion here. A wrapper literal makes sense to me - we can revisit it when we have a use for a wrapper that needs an argument.

Ok, I removed the tlv_record literals but wasn't able to pull out the wrapper from $fieldty:tt. I guess it can be done like _tlv_stream_impl_writeable, but I was hoping to avoid the extra () around any types not needing special encoding. Drawback is anyone using this outside of tlv_stream would have to give both the encoding and the type. I suppose it is at least explicit and should fail to compile if given the wrong type.

jkczyz

Working on updating docs.

jkczyz · 2022-09-26T21:11:36Z

lightning/src/util/ser_macros.rs

+		u8
+	};
+	(u16) => {
+		::util::ser::HighZeroBytesDroppedBigSize<u16>


Are you suggesting we expand encode_tlv (the call site) such that there is a tlv_record matcher for each of these types? Could do that, but it would result in a lot of repetitive boilerplate. Also, wouldn't prevent doing thing surprising given the same matchers here would have to be used there.

Note that these three helper macros are only used in one place each. I think I could define them as nested macros at the respective calls sites. But then relative to each other they would be spread all about, and it's kinda nice having them all in one place.

jkczyz · 2022-09-26T21:22:37Z

lightning/src/offers/offer.rs

+pub type CurrencyCode = [u8; 3];
+
+tlv_stream!(OfferTlvStream, OfferTlvStreamRef, {
+	(2, chains: Vec<ChainHash>),


It depends on the context, though note that all the records use Option in the TLV structs.

For Offer, only description and node_id are required. This is currently enforced at compile-time by OfferBuilder::new requiring both as parameters.

However, for InvoiceRequest used as a "send invoice" (e.g., refund), node_id MUST NOT be set.

I also avoided the lightning-invoice-style compile time checks because IIUC it can't be easily done in bindings since each builder function returns a different type, essentially.

jkczyz · 2022-11-01T19:13:55Z

This should be ready for review with the most recent spec changes, including disallowing Some(1) for quantity_max. #1726 has the parsing side of that change.

lightning/src/offers/offer.rs

valentinewallace · 2022-11-02T17:21:07Z

lightning/src/offers/offer.rs

+	}
+}
+
+tlv_stream!(OfferTlvStream, OfferTlvStreamRef, {


FYI, a chunk of unused warnings are introduced here and from Offer::bytes

Hmm... I only see them on intermediate commits. Are you seeing them on the HEAD commit?

in CI: https://github.com/lightningdevkit/rust-lightning/actions/runs/3381594439/jobs/5615665472

Hmm... I cna't seem to reproduce this locally. Can you? Maybe CI is using some flag that catches this.

I cannot :( may be related to cargo version as well

lightning/src/lib.rs

jkczyz · 2022-11-02T22:33:40Z

lightning/src/offers/offer.rs

@@ -163,7 +163,10 @@ impl OfferBuilder {
 	///
 	/// Successive calls to this method will override the previous setting.
 	pub fn supported_quantity(mut self, quantity: Quantity) -> Self {
-		self.offer.supported_quantity = quantity;
+		self.offer.supported_quantity = match quantity {
+			Quantity::Max(n) if n.get() == 1 => Quantity::One,


Wondering if I should instead have only two variants (Quantity::Max(n) and Quantity::Many) and default to Quantity::Max(1). Then would translate to the appropriate Option<u64> internally for the TLV stream. Thoughts?

Yea, I'm not a huge fan of the redundant One and Max(n) in the API. We could even just do a NonZeroU64 entirely as far as I'm concerned - we could just use u64::max for the Many entry. I also just generally don't want to overthink the quantity stuff, I have a strong feeling its not going to be heavily used.

FYI, I changed this to two variants Bounded(n) and Unbounded as discussed offline.

TheBlueMatt · 2022-11-03T23:15:58Z

Needs rebase, feel free to squash when you do so IMO.

TheBlueMatt

Basically LGTM.

lightning/src/offers/offer.rs

The offer message in BOLT 12 contains a features TLV record. Add a corresponding OfferFeatures type where the length is not included in the serialization as it would be redundant with the record length. Otherwise, define the features to be the same as InvoiceFeatures.

Strings defined by third parties may contain control characters. Provide a wrapper such that these are replaced when displayed. Useful in node aliases and offer fields.

Define an interface for BOLT 12 `offer` messages. The underlying format consists of the original bytes and the parsed contents. The bytes are later needed when constructing an `invoice_request` message. This is because it must mirror all the `offer` TLV records, including unknown ones, which aren't represented in the contents. The contents will be used in `invoice_request` messages to avoid duplication. Some fields while required in a typical user-pays-merchant flow may not be necessary in the merchant-pays-user flow (i.e., refund).

When serializing variable-length types as part of a TLV stream, the length does not need to be serialized as it is already encoded in TLV records. Add a WithoutLength wrapper for this encoding. Replace VecReadWrapper and VecWriteWrapper with this single type to avoid redundant encoders.

BOLT 12's offer message is encoded as a TLV stream (i.e., a sequence of TLV records). impl_writeable_tlv_based can't be used because it writes the overall length of the struct, whereas TLV streams only include the length of each TLV record. Add a `tlv_stream` macro for defining structs used in encoding. TLV records containing a single variable-length type should not encode the types length in the value since it is redundant. Add a wrapper type that can be used within a TLV stream to support the correct behavior during serialization and de-serialization.

jkczyz · 2022-11-04T20:12:27Z

Rebased and squashed.

lightning/src/offers/offer.rs

TheBlueMatt · 2022-11-04T20:54:47Z

lightning/src/offers/offer.rs

+	/// Builds an [`Offer`] from the builder's settings.
+	pub fn build(self) -> Result<Offer, ()> {
+		if let Some(Amount::Currency { .. }) = self.offer.amount {
+			return Err(());


Are we gonna document that this doesn't work? Why fail here, why not just not bother checking the amount is > MAX_MSAT?

Yeah, not sure why I'm erroring here. The next PR, which adds the error type, already skips the check. Looks like I just missed removing the error earlier.

TheBlueMatt · 2022-11-04T20:56:57Z

lightning/src/offers/offer.rs

+	/// Returns the amount in millisatoshi.
+	pub fn as_msats(&self) -> u64 {
+		match self {
+			Amount::Currency { .. } => unimplemented!(),


Isn't this user-reachable if we deserialize an Offer and the user calls amount.map(as_msats)? Maybe we should just drop the as_msats thing entirely?

That's why it's unimplemented and not unreachable. 🙂 Initially, I wasn't sure how I wanted to handle this. But then I figured we'd probably parameterize this method with a Forex trait or the like. Given the module is private, this could wait until later. Or maybe it's just simple enough to do now. Thoughts?

I'm not sure I really understand why we're providing this method at all? The user can match the Amount and look at the msat value or do their own forex conversion. Providing a util method that does that seems like it doesn't accomplish anything.

In InvoiceRequestBuilder, we need to check that the amount provided is sufficient for the Offer when accounting for both currency and quantity to be in accordance with the spec:

- MUST specify `invreq_amount`.`msat` as greater or equal to amount expected by `offer_amount` (and, if present, `offer_currency` and `invreq_quantity`)

So IIUC that means we have to do currency conversion when we receive an invoicerequest and check that the msat amount matches the given-currency amount? And for that you want to have a util on the Offer to do the conversion, presumably with some trait for currency conversion?

For now, let's just leave it all our, skip the conversion function, and reject invoicerequests with a currency other than bitcoin? That avoids adding broken code here, and we can figure out how/if we want to support currency conversion in the onion-message-handling pipeline later.

Yes, building an async trait to fetch the exchange rate in the middle of a sync message process is.... not going to be fun.

Could we do the same as we do for fee rates? Just have a background task that polls the latest exchange rates.

What I meant is, if the offer is for $20 USD, and we get an invoice_request for $10 in msats, we have to reject that, no?

Correct. Yeah, we're on the same page now. Needed both for a sender requesting an invoice from an offer denominated in USD and a recipient who publishes said offer when checking an invoice_request.

At least there we can force the user to provide it - the request_invoice method can require users pass in the msat value for offers that are in other currencies.

Sure, that's definitely an alternative API. But if we need to check an amount when handling an invoice_request, why not be consistent?

I don't buy this. If you post a permanent offer, it's probably not for a specific amount - you're asking for donations. If you have a specific amount in mind, it's for a specific payer, and probably only valid for some short time period (to pay an order).

Thinking more about it I'm not sure how much of a problem different currency denomination offers is solving. That horse is out of the barn now so not worth trying to revisit it, but I don't think we should bend over backwards building something clean for that use-case until we see how people are using offers in the wild.

I'd imagine it's useful for recurring payments / subscriptions.

Could we do the same as we do for fee rates? Just have a background task that polls the latest exchange rates.

We currently push that complexity on users, which kinda sucks. Further, for exchange rates things may be more complicated - you have to pre-fetch every exchange rate and store that, not just one three values but hundreds. Worse, I'm not sure what the appropriate answer is for a node that doesn't want to query some centralized website and give them a list of all users running lightning - how are we going to do exchange rate lookups for the sample node? We certainly can't query some centralized website, and we don't currently directly support SOCKS to pass it over tor, nor do I know that we want to require Tor to run it.

To make matters worse, most financial data APIs are paid, or at least paid over some rate threshold. A cursory glance at exchange rate data seems to indicate that's mostly true here too.

I'd imagine it's useful for recurring payments / subscriptions.

Mmm, recurring payments is a good point. IIRC that's been pushed from the v1 spec for now?

Sure, that's definitely an alternative API. But if we need to check an amount when handling an invoice_request, why not be consistent?

Right, my question is how to make this simple for users. On the receiving side, you only need to care if you ever provided an offer with a non-BTC currency, on the sending side you always need to care. I don't really have an answer here, mostly because I'm pretty confident all possible answers really really suck. I think it may be totally reasonable for a node to simply not support sending to non-BTC-denominated offers, sadly.

In any case, as for the current PR, I'd really rather not commit code that panics and can't really be fixed, and at least for the offer-issuance side we can trivially support only-btc-denominated offers for now, which seems like a simple way to go.

We currently push that complexity on users, which kinda sucks. Further, for exchange rates things may be more complicated - you have to pre-fetch every exchange rate and store that, not just one three values but hundreds. Worse, I'm not sure what the appropriate answer is for a node that doesn't want to query some centralized website and give them a list of all users running lightning - how are we going to do exchange rate lookups for the sample node? We certainly can't query some centralized website, and we don't currently directly support SOCKS to pass it over tor, nor do I know that we want to require Tor to run it.

To make matters worse, most financial data APIs are paid, or at least paid over some rate threshold. A cursory glance at exchange rate data seems to indicate that's mostly true here too.

Fair points. Though, FWIW, a user only needs to poll for the currency in which they denominated offers in. For paying offers, the implementation can fetch on demand or return an error if unsupported.

Mmm, recurring payments is a good point. IIRC that's been pushed from the v1 spec for now?

Correct.

Right, my question is how to make this simple for users. On the receiving side, you only need to care if you ever provided an offer with a non-BTC currency, on the sending side you always need to care. I don't really have an answer here, mostly because I'm pretty confident all possible answers really really suck. I think it may be totally reasonable for a node to simply not support sending to non-BTC-denominated offers, sadly.

If InvoiceRequestBuilder::amount_msat (or the builder itself) is parameterized by the exchange trait, then we could define a default implementation that errors, I suppose, if Offer::amount is a Currency.

In any case, as for the current PR, I'd really rather not commit code that panics and can't really be fixed, and at least for the offer-issuance side we can trivially support only-btc-denominated offers for now, which seems like a simple way to go.

So essentially have InvoiceRequestBuilder error if attempting to pay a non-BTC-denominated offer?

Changed OfferBuilder::amount to OfferBuilder::amount_msats and updated dependent PRs accordingly.

Fair points. Though, FWIW, a user only needs to poll for the currency in which they denominated offers in. For paying offers, the implementation can fetch on demand or return an error if unsupported.

Right, but how do we capture that in an interface that makes sense and is easy to use and hard to mis-use :(

So essentially have InvoiceRequestBuilder error if attempting to pay a non-BTC-denominated offer?

Yea, for now.

TheBlueMatt · 2022-11-08T17:39:04Z

lightning/src/offers/offer.rs

+					return Err(());
+				}
+			},
+			Some(Amount::Currency { .. }) => unreachable!(),


Can we just not fail here? I agree I think its unreachable but there's no reason to actually panic vs generating an offer that we may reject later.

FWIW, it already fails in the next PR when we have actual errors to differentiate (cadc69c). Do you mind if we just leave this as is until then to avoid more rebasing?

lightning/src/offers/offer.rs

Add a builder for creating offers given a required description and node_id. Other settings are optional and duplicative settings will override previous settings for non-Vec fields.

jkczyz force-pushed the 2022-09-offer-encoding branch from 7545c45 to 8081c46 Compare September 13, 2022 23:07

TheBlueMatt reviewed Sep 14, 2022

View reviewed changes

valentinewallace reviewed Sep 14, 2022

View reviewed changes

TheBlueMatt reviewed Sep 15, 2022

View reviewed changes

jkczyz commented Sep 15, 2022

View reviewed changes

jkczyz force-pushed the 2022-09-offer-encoding branch from a8ff89a to 6150762 Compare September 16, 2022 21:41

jkczyz force-pushed the 2022-09-offer-encoding branch from 6150762 to c4759e1 Compare September 19, 2022 00:17

valentinewallace reviewed Sep 19, 2022

View reviewed changes

jkczyz force-pushed the 2022-09-offer-encoding branch from c4759e1 to 50d3156 Compare September 19, 2022 21:18

valentinewallace reviewed Sep 20, 2022

View reviewed changes

valentinewallace reviewed Sep 22, 2022

View reviewed changes

lightning/src/offers/offer.rs Outdated Show resolved Hide resolved

lightning/src/offers/offer.rs Outdated Show resolved Hide resolved

lightning/src/util/ser.rs Outdated Show resolved Hide resolved

jkczyz force-pushed the 2022-09-offer-encoding branch from 3c999b9 to 51f6195 Compare September 22, 2022 22:59

valentinewallace reviewed Sep 23, 2022

View reviewed changes

lightning/src/offers/offer.rs Outdated Show resolved Hide resolved

jkczyz force-pushed the 2022-09-offer-encoding branch from 51f6195 to 5a4d91e Compare September 23, 2022 22:52

TheBlueMatt reviewed Sep 26, 2022

View reviewed changes

jkczyz commented Sep 26, 2022

View reviewed changes

jkczyz force-pushed the 2022-09-offer-encoding branch from 5a4d91e to b81931c Compare September 27, 2022 22:51

jkczyz force-pushed the 2022-09-offer-encoding branch from d5eae96 to 14431fd Compare October 28, 2022 22:21

valentinewallace reviewed Nov 2, 2022

View reviewed changes

lightning/src/offers/offer.rs Outdated Show resolved Hide resolved

lightning/src/offers/offer.rs Outdated Show resolved Hide resolved

lightning/src/offers/offer.rs Outdated Show resolved Hide resolved

lightning/src/offers/offer.rs Show resolved Hide resolved

valentinewallace reviewed Nov 2, 2022

View reviewed changes

jkczyz force-pushed the 2022-09-offer-encoding branch from 4e4b15f to 8274164 Compare November 2, 2022 22:30

jkczyz commented Nov 2, 2022

View reviewed changes

jkczyz force-pushed the 2022-09-offer-encoding branch 2 times, most recently from 5ccd8c7 to dc69f15 Compare November 3, 2022 21:52

TheBlueMatt reviewed Nov 3, 2022

View reviewed changes

lightning/src/offers/offer.rs Outdated Show resolved Hide resolved

jkczyz added 5 commits November 4, 2022 15:07

Add PrintableString utility

48bb9ed

Strings defined by third parties may contain control characters. Provide a wrapper such that these are replaced when displayed. Useful in node aliases and offer fields.

jkczyz force-pushed the 2022-09-offer-encoding branch from dc69f15 to e673125 Compare November 4, 2022 20:11

TheBlueMatt reviewed Nov 4, 2022

View reviewed changes

jkczyz force-pushed the 2022-09-offer-encoding branch 2 times, most recently from 20da3f3 to 0846064 Compare November 8, 2022 01:47

TheBlueMatt reviewed Nov 8, 2022

View reviewed changes

TheBlueMatt previously approved these changes Nov 8, 2022

View reviewed changes

TheBlueMatt reviewed Nov 8, 2022

View reviewed changes

lightning/src/offers/offer.rs Show resolved Hide resolved

Builder for creating offers

8ba09e0

Add a builder for creating offers given a required description and node_id. Other settings are optional and duplicative settings will override previous settings for non-Vec fields.

jkczyz dismissed TheBlueMatt’s stale review via 8ba09e0 November 8, 2022 19:48

jkczyz force-pushed the 2022-09-offer-encoding branch from 0846064 to 8ba09e0 Compare November 8, 2022 19:48

TheBlueMatt approved these changes Nov 8, 2022

View reviewed changes

valentinewallace approved these changes Nov 8, 2022

View reviewed changes

TheBlueMatt merged commit f1428fd into lightningdevkit:main Nov 8, 2022

jkczyz mentioned this pull request May 10, 2023

BOLT12 Offers Tracking Issue #1970

Open

60 tasks

BOLT 12 offer encoding and building #1719

BOLT 12 offer encoding and building #1719

Conversation

jkczyz commented Sep 13, 2022

codecov-commenter commented Sep 13, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Sep 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz commented Sep 16, 2022

jkczyz commented Sep 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz Sep 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz commented Sep 22, 2022

valentinewallace commented Sep 23, 2022

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Sep 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz Sep 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BOLT 12 `offer` encoding and building #1719

BOLT 12 `offer` encoding and building #1719

codecov-commenter commented Sep 13, 2022 •

edited

Loading

jkczyz Sep 21, 2022 •

edited

Loading

jkczyz Sep 27, 2022 •

edited

Loading