Generalise event lifetime preferences

[hoytech]

Background

Implementing NIP-33 and NIP-40 for my relay (strfry) caused me to think a bit about user-specified event lifetime preferences. We can roughly classify these into the following categories:

• Replaceable events, including parameterised replaceable (NIP-16, NIP-33): These indicate to a relay that an event can be removed when another more recent qualifying event is received
• Ephemeral events (NIP-16): These events should be broadcast to all connected users with a matching filter, but then deleted (or not stored in the first place)
• Expiring events (NIP-40): These events should be deleted when a specifed timestamp is reached

Note that if a relay supports NIP-09 deletions then all of the above could in principle be implemented with delete events:

• Replaceable events could be deleted immediately before posting a replacing event
• Ephemeral events could be deleted immediately after posting
• Expiring events could be deleted at their expiration time

By using deletions, any of these categories of event lifetimes could be implemented for events of any kind.

Problems

Replaceable, parameterised replaceable, and ephemeral events are specified to only apply to events within ranges of kinds. To me this seems sub-optimal:

• Protocols can't group their kinds together if they need different event lifetimes for different kinds. For instance, NIP-28 (public chat) reserves the range of kinds 40-49 but specified kind 41 to be replaceable, before NIP-16 was specified, I assume. With NIP-16, a non-contiguous kind in the 10000-19999 range would need to be reserved.
• Kinds can't compose with event lifetimes. A new protocol might like to have some messages of the same kind have different event lifetimes. For example, a protocol might like to support editable (replaceable) postings as well as "permanent" postings using the same kind.
• Users can't specify custom lifetimes for events of existing protocols. For example, you could imagine sending an ephemeral kind 1 message, as a "disappearing" temporary message that would pretty much just work on any existing client.
• The 10k blocks of ranges are arbitrarily sized, and they could be either too small (new ranges would have to be reserved in the future) or too big (prime kind real-estate will forever go unused).
• Users who aren't aware of the special handling of a particular kind mind start using it and get unexpected event lifetimes.

Importance of parameterised replaceable events

I believe NIP-33 is a very important specification and will be heavily used by future protocols.

The NIP itself doesn't contain any use-cases, but as a basic example, I could imagine using these types of event lifetimes for sharding and segregating your kind-3 contact lists. You could imagine having multiple different contact lists, like family, friends, co-workers, randos1, and randos2. This would future-proof contact lists growing too large, improve the efficiency of updates, and provide a natural organisation schema for users.

Furthermore, if it were possible to use parameterised replaceable events on kind 3, this would all work with minimal client changes and a reasonable backwards compatibility story. I'm not saying this is a good idea and we should do it, just pointing out the possibility.

Note that replaceable events are just a special case of parameterised replaceable events with a static replacement key.

Proposal

Here are two minor changes that I believe would generalise the protocol:

• Extend NIP-40 to special-case an expiration tag with value 0. This would indicate that the event is intended to be ephemeral, and should be treated as described in NIP-16, regardless of kind.
• Support the d tag on events of any kind. This indicates the event is to be treated as a parameterised replaceable event as described in NIP-33, using the value of this tag as a replacement key. This works for any kind, but the kinds of the replaced and replacing events must still match. The absence of a d tag does not result in an implicit empty d tag as described in NIP-33.

Disadvantages

• Events that want replaceable behaviour become slightly larger, since they need to include a d tag.
• A user could forget to include a d tag and unintentionally have multiple events of the same kind on a relay. However, when querying, clients that expect replaceable behaviour could take care to select the latest one (by created_at). Additionally, the user could correct this situation using a NIP-09 deletion.

Back-compat

The two above changes would not break any of the existing NIPs. However, I think it could make sense to create a NIP that supersedes NIPs 16 and 33, and mark the coupling of event lifetime with the 10k kind-ranges deprecated.

Additionally, here is how a relay could internally implement backwards compatibility using the d and expiration primitives:

• Kinds 20000-29999 have an implicit expiration tag with value 0 added to the front of the tag list. NIP-40 is clarified to specify that in the case of multiple expiration tags, the first tag's value is to be used.
• Kinds 0, 3, 41, 10000-19999, and 30000-39999 have implicit "d" tags with "" values added to the end of their tags list.

History

• The original version of this proposal suggested using a replace tag instead of NIP-33's d tag, but this would not have been queryable as pointed out by @barkyq below.

[barkyq]

I think the "d" tag is good to be queryable.

That way a client can ask for a given note by referring to the "d" tag rather than the changing event id.

I have no idea what "d" stands for, maybe "document", as I think one of the original use-cases for P.R,E. was editable documents.

[barkyq]

As an aside/old-man-complaint... I am not convinced of the utility of general purpose "expiration" tag. Just send a deletion event when you want your event to expire. Puts unnecessary burden on the relays IMO. Feel like most users will opt to not use the expiration "feature" and the relay will just always be checking for events to delete without ever being able to delete very much.

On the other hand, ephemeral events is super relay friendly 👍

Agreed that it would be nice to have an ephemeral event related to kind 1. Naively could just define 20001 as ephemeral kind 1. I guess it is not a very principled solution though. and many unanswered questions viz UI

[barkyq]

Also love strfry super impressive work, wish I understood C++ better. Very very good performance on wss://nos.lol

[hoytech]

Thank you!

You're completely right about the d tag being useful for querying. Especially if the user has a large number of PREs of the same kind. I'll edit the above proposal to get rid of my proposed replace tag in favour of using the NIP-33 d tag -- my mistake.

I partially agree about not using expiration and instead just sending a delete message when you want your events to go away since deletion is more general purpose. However there are some arguments in favour of expiration:

• You might not know all the relays that your event has propagated to, and there is no way to ensure that your delete message will reach all these relays too.
• You might not be available to sign the delete message right at the expiration time (network down, computer crashed, whatever). Even if you do sign it right on time, it won't necessarily propagate to all the relays on time. Clients that understand the expiration tag can immediately stop showing it when the time comes (and could even display a countdown or whatever too).

It does add a bit of extra load on the relays, but in strfry I've added an index on this field so the periodic checking for expired events is very inexpensive.

Good point that an event of 20001 could be standardised as an "ephemeral kind 1". However, clients would have to be re-coded to query for this kind, and display it. My proposal allows "ephemeralness" to compose with any kind so it would be more likely to work with existing clients.

[fiatjaf]

I like this. It is sad that we made bad choices in the past.

[fiatjaf]

@Semisol @cameri

[barkyq]

Seems easy enough to implement. Will need to reserve the d tag as a special tag for all events; do not suppose that should be added to "generic tag queries" NIP (something like the tags p, e, d are reserved). Would be a shame if some other NIP started using d tags on non 30000-39999 kinds.

Also expiration being less than created_at should automatically imply the event is treated as ephemeral? Seems like an easy enough trigger to implement in SQL or any relay implementation.

Seems like expiration < created_at would be totally useless unless expiration was queryable. If it was queryable, then setting low expiration values would give a defacto integer "kind" one could associate to ephemeral events. Most relays implementing NIP-40 will probably already have indices for the expiration. Although the way the queryable tags work, it would be very awkward to query for, say, all notes which will expire in 2 days.

Perhaps could add custom query rules to NIP-40, something like, in the filter JSON keys like:

"expiration": ["at", "1676088797"]
"expiration": ["before", "1676088797"]
"expiration": ["after", "1676088797"]

Obvious interpretation as filters on the "expiration" tag. Events which do not have the expiration tag should not be returned etc etc.

Would be kind of cool.

[barkyq]

Also, to reduce long-lived spam, popular people like jack could say stuff like, "I am filtering by expiration in less than 24 hours, let's have a discussion about github on nostr etc etc". Kind of a stoner thought and certainly many things potentially problematic but curious to hear what the author and others think.

[hoytech]

Also expiration being less than created_at should automatically imply the event is treated as ephemeral?

EDIT (mis-read this). This could work too. What would the different possible values for expiration represent, how ephemeral it is? :)

Maybe instead of overloading expiration, we should instead use a new tag like ephemeral. If present, the event should be considered ephemeral. This would probably have a bit better backwards compatibility actually, for relays that have implemented NIP-40 but not the 0 overload.

TBH I can't think of many use-cases for users querying by expiration and/or ephemeralness. I agree your method for querying numeric ranges is cool, but I don't think we should burden relay implementors with that unless we have some really solid use-cases.

[barkyq]

Yes I just figured that since it would be straightforward to check if expiration < created_at, it would add a new spot to put some extra information, without really increasing the size of the event very much.

Re: querying for expiring events. Yes I agree that it would be annoying to coordinate all the different relay codebases to add a custom filter parsing rule just to query expiration tags... The part which motivated me was that probably the DBs already have an index for this expiration integer. But yes a bit convoluted and not very intuitive.

Re: changing to ephemeral tag. This tag ephemeral would not take any arguments? Seems pretty elegant IMO.

---

If a d tag is present, the event is considered replaceable, and the second argument in the tag is considered as the "identifier" for the event (which can persist, i.e., the title of a manuscript / news article or something). Any kind can have a d tag.

One potential source of conflict is if people put d tags (with different values) on events which were historically considered replaceable like 0, 3, 41, 10000-19999. Like trying to implement a collection of kind 3 events for family, friends, etc. What do you think about that?

Also a bit strange if kind 5 (event deletions) got d tags, as it would be like deleting a deletion event, which I think is supposed to not be allowed.

---

If the ephemeral tag is present, the event is considered ephemeral. Are there any event kinds which should definitely not have an ephemeral tag? Cannot think of any right now.

Perhaps there could be a "fake" relay hint added to e tags when replying to ephemeral events. This could signal to clients to not bother looking for a given event (or even display some little box where the event should be saying "ephemeral event"). Probably beyond the scope of this NIP though.

One more question. Do you specify if there is an event with d tag and ephemeral tag? Two options, probably depending on the order people write SQL triggers. Guess one canonical order should be specified. Probably the old event with same d tag and kind gets deleted then the new one gets broadcasted without being stored (or gets stored very briefly for strfry)? At the end of the process there is no event with the same d tag, kind, and pubkey?

[hoytech]

One potential source of conflict is if people put d tags (with different values) on events which were historically considered replaceable like 0, 3, 41, 10000-19999

I think it makes the most sense for events with default-replaceable kinds to be treated as NIP-33 events. So if there is no d tag it effectively uses an empty string as the replacement key. That way the interpretation of events without d tags does not change. If people start adding events with non-empty d-tags, then these new events will exist in parallel with the events without d-tags. I agree this might cause some confusion for clients that only expect one kind 3 event, but clients really ought to handle this case anyway, since a relay does not have to implement NIP-02 (and even then, NIP-02 says replacement is "SHOULD").

Also a bit strange if kind 5 (event deletions) got d tags, as it would be like deleting a deletion event, which I think is supposed to not be allowed.

Agreed, deletions is an interesting case. NIP-09 says "Publishing a deletion event against a deletion has no effect. Clients and relays are not obliged to support "undelete" functionality."

Which doesn't seem definitive either way to me. Personally I see no reason not to support "undeletes" (either with subsequent NIP-09 deletions or with d-tags). The relay-side code is even easier that way.

Deletes are actually another interesting use-case for composing ephemeral events. The natural processing of an ephemeral deletion would be to delete the referenced events but not store the deletion event itself -- this might be useful for cleaning up old junk from relays without leaving a bunch of delete events lying around. Of course, somebody could re-submit the old events if they kept copies of them, but that's not always a concern.

I can't really think of any other events that shouldn't compose with ephemeral events. As mentioned above, I believe you can always emulate ephemeralness with a NIP-09 deletion anyway.

Somebody making a reply to an ephemeral event -- that's actually a tricky one and I could see it causing problems, although these problems are much the same as exist now when replying to an event that subsequently gets deleted. I think your suggestion about flagging ephemeral events in the UI is a good one.

If d and ephemeral are on the same event I think the natural order of processing is as you describe: Replacement happens, the event is broadcast, and then the event is deleted (or never stored). You're right this should probably be made explicit.

Thank you for thinking through all these edge cases!

[barkyq]

Alright. I am on board! Excited to hear what others think. 👍

[ryzizub]

I love this 🔥 Thanks @hoytech for pointing this out

[cameri]

I think the "d" tag is good to be queryable.

That way a client can ask for a given note by referring to the "d" tag rather than the changing event id.

I have no idea what "d" stands for, maybe "document", as I think one of the original use-cases for P.R,E. was editable documents.

d is for deduplication

[cameri]

@hoytech @fiatjaf I am pretty much onboard with these changes.

[scsibug]

I'm in agreement as well.

[fiatjaf]

OK, I think that gives us enough quorum to proceed with this.

[fiatjaf]

On the other hand I think this is a bad idea now. We don't want to create unnecessary burden on clients by allowing every event to become instantly replaceable. If kind:1 notes are replaceable that would mean a disproportionate amount of complexity added to otherwise simple social clients.

[scsibug]

Is it better for kinds to opt-in (as part of their NIP) to non-replaceability? And then have relays ignore the d tag (if one exists) in those cases? Agreed that kind:1 replaceability is not desirable.

[peterzion45]

Who understands these things except for the developers/programmers/and coders?
500ef1fdb07f3ac3a9d6c660c27df6af03acf661ba8fb9d651914a5c87e5bdf9

[arthurfranca]

Rise from grave \o/ cause this issue is masterpiece.

@fiatjaf On the other hand I think this is a bad idea now. We don't want to create unnecessary burden on clients by allowing every event to become instantly replaceable.
@scsibug Agreed that kind:1 replaceability is not desirable.

Yes. Though what makes an event "truly" replaceable for the viewing user is using an a tag when referencing the event.

For example, when clients use e to reply to a kind:1, if the author later replaces this `kind:1', the new version would look like a new note (no replies, no reactions, nothing). Thus same effect as deleting previous + creating new kind:1 note.

Am I wrong? @fiatjaf @scsibug

[fiatjaf]

The code for dealing with replaceable and non-replaceable events is very different at a fundamental level. Replaceable events have a big overhead. This proposal would allow anyone to turn any event kind into replaceable and that would cause havoc in all client and relay implementations.

[Semisol]

The original proposal for NIP-16 actually had an ephemeral and replaceable tag.

This was removed due to it increasing complexity: What would you do if you received a replaceable event for a kind that had a non-replaceable event also? What if the replaceable event was also ephemeral? So on.

This would also cause problems for applications that have some expectations, such as kind 1s being able to be marked as ephemeral.

[arthurfranca]

@fiatjaf This proposal would allow anyone to turn any event kind into replaceable

No, it depends. Bear with me.

1. Let's consider that the the spec says that kind:1 should be treated as a regular event (non-replaceable). It means people should not add d tags to kind:1s and replies to them should use e tags
2. Coracle and Gossip follow spec
3. Amethyst don't wanna follow spec and so it lets users edit their kind:1 posts (using d tag)

What happens is that Coracle and Gossip' replies to Amethyst's kind:1s would still use e tags instead of a tags. So Amethyst edited notes would continue being treated by them as non-replaceable; Amethyst edits would in fact appear as new notes (same as deleting and creating a new one as we have today), cause they would be referenced by id.

See my point?

[arthurfranca]

@Semisol What would you do if you received a replaceable event for a kind that had a non-replaceable event also?

Simple:

• At the relay side, everything can be replaced if it has a d tag, no matter the kind.
• At the client side, follow what specifc NIP say about each kind. If spec says that kind X is regular, fetch by and reference by id (e tag). If replaceable, use address/a tag and d tag is always set to "" (empty string). If parameterized, use variable d tag values.

edit: the ephemeral part, don't see any problem. If client sends event with ["expiration", "<30secs-ahead>"] tag, the event is ephemeral-ish.

edit2: this issue wants relays to treat expiration=0 as ephemeral (broadcast received event to whoever is listening with the right sub filter when relay receives it), though I think today some relays would treat it as expired and maybe don't broadcast it?

[fiatjaf]

See my point?

Yes, but couldn't your point be reduced to "anyone can do anything and if two people happen to do the same thing then they will be compatible, therefore we don't need any standards"?

[vitorpamplona]

From a client perspective:

1. Ephemeral works with expiration tag = now+10. There is no need to change anything on Nostr. The kind range for ephemeral events is completely useless.
2. Any event is replaceable if you create a filter with author and limit 1.
3. Any event is parametrized replaceable if you create a handle (like the d tag is) and do a filter with author, that handle and limit 1.

As a major plus, all versions of the "fake replaceables" are kept.

Event kind ranges can be deprecated.

[arthurfranca]

Any event is replaceable if you create a filter with author and limit 1.

Perfect. Though I'd say replaceables should set d-tag="" (good for relays, they would save disk space)

[arthurfranca]

@fiatjaf Yes, but couldn't your point be reduced to "anyone can do anything and if two people happen to do the same thing then they will be compatible, therefore we don't need any standards"?

The opposite. In a reality where kind range doesn't dictate replaceability of events for relays (relays would rely solely on d tag existence), standards are paramount! NIPs and NUDs must guide clients or else there is no interop.

The NIP where kind X is speced would need to define if kind X is regular, replaceable or PRE (and if ephemeral). This info would be used by clients to:

• Add or not d tag [to kind X]
• Add or not expiration tag
• Reference by id or by address
• Fetch by id or by address (or with limit 1 how @vitorpamplona said above)

[arthurfranca]

Example, all NIPs could add a table like below example for NIP-02:

Kinds used by this NIP

kind	type	d-tag	is ephemeral?	name
3	replaceable	""	❌	follow list

__

Example of how such table could look like for NIP-28:

Kinds used by this NIP

kind	type	d-tag	is ephemeral?	name
40	regular	N/A	❌	channel creation
41	PRE	"<kind-40-event-id>"	❌	channel metadata
42	regular	N/A	❌	channel message
43	PRE	"<kind-42-event-id>"	❌	mute message
44	PRE	"<kind-40-event-id>:<pubkey>"	❌	mute user

[Semisol]

The NIP where kind X is speced would need to define if kind X is regular, replaceable or PRE (and if ephemeral). This info would be used by clients to:

... Just use kind ranges. They are self documenting and don't need relays to handle all types of weird shit.

[arthurfranca]

@Semisol from the thing you quoted: "this info would be used by clients". Relays would not need to care about what NIPs other than NIP-01 say. Would not need to differentiate between event kinds. I will give an example of how NIP-01 would look like brb.

[arthurfranca]

@fiatjaf @Semisol

How a new NIP-01 lore could look like:

NIP-01

[...All mambojambo about basic things such as relays, events, clients, keys, event structure, searchable tags and non-searchable, how to calc event id and sign the id, client-relay and relay-client messages...]

Relay event treatment

Relays treat events received through a client-relay "EVENT" message the same way: no matter the kind,
they are saved on their DB if not expired (check event's expiration tag).

The only distinction made is between events with and without a d tag.
An event with a d tag set, replaces an older event from the same .kind, .pubkey and d-tag-value.

Client event treatment

Relay developers may ignore this section.
REPEATING: Relay developers may ignore this section!!111!!

For clients, event kinds are an alias for their real meaning. For example, kind:1 means "microblog message". This way, non-microblogging clients know they won't use these events.

There are three types of events: regular, replaceable (RE) and parameterized replaceable events (PRE). They are treated differently by clients, when structuring them, referencing them and when fetching them from relays.

To know the type of a specific event kind, check the NIP where the event kind is introduced.

Event structure by type

1. A regular event doesn't set a d tag
2. A RE set one d tag to "" (empty string)
3. A PRE set one d tag to a value specified on the NIP that introduced the specific event kind

Event referencing by type

1. A regular event is referenced by its id. That means the event referencing the regular event
   must add a searchable tag (which one is defined by the corresponding NIP) set to the regular event's id.
2. Similarly, an RE is referenced by an searchable tag but this time it is set to its address (<event-kind>:<event-pubkey>:).
3. A PRE is referenced by a searchable tag set to its address (<event-kind>:<event-pubkey>:<d-tag>).

Event fetching by type

1. When fetching a regular event at relays, filter by event id.
2. A RE should be filtered simultaneously by specific author, specific kind and be limited to one entry ({ limit: 1 }).
3. A PRE is similar to RE but should also be simultaneously filtered by one specifc d tag value.