Update NIP 11 to support relay recommendations

[staab]

The intention of this NIP is to create space in the protocol for specifying NIPs for more advanced use cases without forcing relays to implement them or be considered a partial implementation.

Some use cases that would be prohibitively expensive for hobby relays:

• Content recommendations and search
• Social graph analysis
• Network analytics

Without discoverability, clients will hard-code special-purpose relays, creating a centralizing pressure on the network. In addition, some relays might not be picked up by the known/hard-coded add-on services, leaving their data out of calculations.

Recommendations would allow relays to advertise complementary relays to clients. Complementary might mean that it's run by the same people, contains the same data, or is a preferred source for the given functionality. Master/slave configurations could be set up using this scheme too — master relays would not support requests, while slaves would not support publishing events.

The downside of this is that it relies on clients being smart and following these references. Someone told me about how dApps used to follow this model, but it slowed things down and developers ended up getting lazy and hardcoding services anyway.

[staab]

Here's another example:

[fiatjaf]

The intention of this NIP is to create space in the protocol for specifying NIPs for more advanced use cases without forcing relays to implement them or be considered a partial implementation.

But this is already the intent of the normal NIPs!

[staab]

Technically, yes, the supported_nips field exists. But just because you can find a relay that supports the NIP you're looking for doesn't mean you can find that support for a set of events you're interested in. So implementation is partial for both the relay, and the data on it.

In other words, this nip is focused not on just finding functionality, but on finding state. Picking a random relay that supports a given nip doesn't imply it has the data you want. A simple example is if you have a relay cluster with one master and multiple followers. The master supports publishing, the followers support requests, and they advertise each other.

[huumn]

Rather than extensions there might be reason to begin specifying a second layer over relays. I've been talking with @brugeman about this.

Keep relays dumb - IP routers are an apt metaphor. Let aggregation queries be done in another layer (TCP-like) with its own protocol and clients can pick from "aggregators" like they do relays ... Clients can fallback to "raw" relay and offer a limited feature set if aggregators aren't specified.

As more functionality/queries get stuffed into relays, fewer people will implement and run them.

[staab]

That's exactly the thought process I went through leading up to this NIP.

There are basically two options for supporting advanced functionality:

1. Add NIPs that relays can optionally implement
2. Create a second layer protocol

Option #1 violates the "keep relays dumb" rule. The more functionality that gets packed into relays, the less small relays will be able to keep up with increasing system resource requirements, and the more complex relay implementations will discourage smaller relay implementations (ala Browser Wars).

Option #2 breaks decentralization because 1. it weakens the guarantee of interoperability by creating a new protocol (or many protocols, or ad-hoc specifications, which I think is more likely), and 2. second-layer services don't fit into a homogenous relay topology. Relay hints, lists, and other places where relays are a first-class concept are incredibly valuable for allowing users to navigate the network and choose where they want to be. A second layer would need to piggyback on the relay recommendation problem somehow, in order to prevent some other mechanism from selecting gatekeepers for the network.

It seems to me the problems associated with option #2 are actually much worse than option #1. If we start with the topology of clients/relays as it already exists in the protocol and in practice, we have the hard problem of decentralization solved. Relays can then reference additional functionality, and clients can follow those references to fulfill their requirements. This has the slightly hidden benefit of improving the relevance of service discovery, since it relies on relays to tell clients which service provider will yield the best results for its users. Finding a content search service doesn't help me if it's indexing the wrong content.

Whether these second-layer services are implemented by relays or not is not actually that important. I think they should be, since a consistent API would reduce the complexity of implementation, and would keep the protocol coherent. The idea of adding new API to the protocol is already built in to NIPs, for example COUNT.

So, the key takeaways of the approach I'm suggesting:

• Relays are first-class concepts for the nostr protocol, and for users. They solve discoverability, and user customization. They should form the backbone of service discovery.
• Service discovery is more complex than finding a service that implements an interface. Services are generally going to be stateful, which means you need to fine the one that has what you're looking for.

Here are a few other ways I've considered for solving service discovery:

• Build it into the protocol, repeating relay hints/lists with a new type of thing.
• Add-on services could advertise which relays they cover, but how would a client find the right service for its users?
• Add-on services could take relays as a parameter to requests, and fetch the data if it's not in its cache already. I think this will result in either massively scaled up service providers, or a poor cache hit rate.
• Clients could hardcode urls for services. This encourages vertical scaling, centralization of service providers, and doesn't attempt to find data that isn't indexed by the service provider.

[fiatjaf]

I think we should talk in more concrete terms here. What do you think of NIP-50? Is that stuffing things into relays? I hope no one is expecting all relays to implement that. Why can't it just fit the normal list of NIPs?

[staab]

I agree indexers will have centralizing effects, the intent of this NIP is to get ahead of that, and articulate add-ons as discoverable through relays, rather than sitting in front of them.

All functionality for relays is optional, so all NIPs can be considered add-on services. That's why NIPs specifying sophisticated add-ons should keep the same relay protocol, since there's no fundamental difference between simple functionality like being able to publish to a relay and more advanced stuff like SUGGEST or whatever.

So for NIP 50, by all means add it to your relay implementation and advertise support via supported_nips. But if I have a user who wants to search relay a specifically for some content, but relay a doesn't support NIP 50, the client currently can't use the search functionality. That's sort of fine, but as a client author, I'd like to be able to search anything my user wants to search. Recommendations allow relay a to say "I don't support search, but I pay relay b, which is a relay instance optimized for search and nothing else, to index my database".

This allows for specially tuned relays to exist and build a business model through specialization, without as much of a risk of centralization.

[staab]

I'll also say that my plan for Coracle when I reach the stage where I need some additional piece of functionality that doesn't exist on relays and can't be performed client side is to create a coracle-specific indexer (actually my link preview stuff already sort of is that). I don't want to do that in a way that damages the integrity of the network though, adding my service to a relay implementation and letting my client discover it organically would be much preferred.

[huumn]

I moved my original comment to a gist: https://gist.github.com/huumn/5dbc53faad5ccac4bd2e6610ca777e69. I don't deserve to have an opinion here until I build more stuff on nostr. Unexperienced blabbering is unproductive. Apologies for the interjection.

[staab]

@pablof7z you talked about this problem a bunch in the relays panel yesterday, I'd be interested in your thoughts on this as a way to reduce centralization, encourage data locality, and ask clients to be smarter.

[huumn]

As I said in #522 I think, naive as I am, that all relay discovery efforts are probably worth exploring.

As such I don't really have much to add. I think this design pattern is consistent with the way clients currently discover relay functionality.

Another pattern that might be worth considering is one that's a little less stateful and therefore flexible like redirection. ie when client tries to use a nip that isn't supported, the relay responds with a NOTICE that tells them to try it on some other relay. The main advantage this would have over the proposal is it could be dynamic (the relay sees some sub-relay is down, etc) and the client would have a little less global state to manage.