Persistence with PubSub

[aschmahmann]

Background

Utilizing persistence with PubSub is useful for a variety of applications and has been a long requested feature.

There are a number of issues revolving around this request including libp2p/go-libp2p-pubsub#42 and ipfs/kubo#6447

Similarly, there have been a number of proposals for how to resolve this including:

1. Allow pubsub messages to reference each other and ask for previous messages
2. Just rebroadcast the state
3. Allow pluggable routers so developers can add their own semantics (Message Propagation/Gossip Modularity go-libp2p-pubsub#161, PubSub & GossipSub configuration proposal go-libp2p-pubsub#175, Added subscription specific configurations to PubSub go-libp2p-pubsub#171)

So far, each of these has been vetoed. 1 is too case specific, 2 has bad performance characteristics, and 3 involves committing to the interface design of PubSub routers.

Current Work

The current approach as described in libp2p/go-libp2p-pubsub#190 and libp2p/go-libp2p-pubsub-router#33. Does the following:

1. Adds a mechanism for PubSub to expose when it connects to a new peer (Add the ability to handle newly subscribed peers go-libp2p-pubsub#190)
2. (Ab)using stateful validators + application layer publishing to control which messages are broadcast. (Add Persistence Layer on top of PubSub go-libp2p-pubsub-router#33)
3. Registering a protocol for handling the "OnJoin" method for a given topic that gives peers the "Best Available Record" (Add Persistence Layer on top of PubSub go-libp2p-pubsub-router#33)

Questions

The above approach should work fine. However, there are some questions about the custom "OnJoin" protocol being implemented including:

1. Is this protocol push or pull?
   • Push saves half an RTT by just sending the data
     • Would this need an ACK?
     • Is this more abusable for DoS attacks?
   • Pull allows future protocol iterations to better control how they ask for data
2. Should the protocol be 1 or 2 rounds?
   • e.g. Should it send data when asked, or first send a small identifier (e.g. a CID) and only when asked for the identifier send the full message?
3. Is the lack of blacklisting, etc. a DoS problem?
4. Should the protocol be expandable to asking for the "Best Available Records" for multiple topics simultaneously (e.g. A connects to B then asks B for the best records on all topics they share, instead of a new request per topic)?
   • It's more efficient when requesting multiple topics (since no new streams)
   • It may be less efficient when only requesting a single topic if we naively do they same thing as with multiple topics (e.g. multiple topic responses might need IDs to go with them, where as one topic per stream has an implied response ID)

[aschmahmann]

Update after a call with @Stebalien @alanshaw @dirkmc @jimpick.

We're going with:

Pull protocol because the additional control definitely seems worth trading off half an RTT.

1 round because it's lower latency, simpler, and seems pretty good for most cases. Also worth noting that a second protocol like bitswap, graphsync, [insert your protocol here] could always be layered on top of the 1st round to give a second round.

1 topic per request because it's lower bandwidth in the 1 shared topic per peer use case which is fairly common. Additionally, we can always upgrade the protocol to deal with >1 shared topics and for some added complexity end up with more optimal bandwidth usage for both single and multiple topics.

[aschmahmann]

Closed by libp2p/go-libp2p-pubsub-router#33