feat: abstraction over filter and store for subscriptions with message recovery

[adklempner]

This is a bug report/feature request/support request/change request

Problem

If you run a filter subscription and go offline, any missed messages while offline will not be recovered. There can be other conditions that lead to missing messages from filter as well.

Proposed Solutions

We should explore an abstraction over both filter and store that is able to:

• identify if any messages were missed
• use store to recover missed messages
• (?) have a single stream that supplies these messages from both sources
  • duplicate messages need to be detected and filtered out

Notes

• Be aware of changes to store from v3
• Check if something like this already exists in status-go or chat-sdk

[danisharora099]

Agree.

This also ties in with using Store as a way to increase reliability in Filter, and related to #2154:

• store incoming messages from Filter in a memory cache
• run a store query every recurring interval (N seconds)
• compare the store query results with the filter memory cache and handle missed messages

sharing this for reference when the original issue is being addressed to contextualise implementation notes

[weboko]

Before attempting to implement this feature - we need to know if pings(store-v3 I assume) are light weight.

We need to be clear in the assumptions and check them:

• is Store reliable enough for us to be source of truth;
• (mentioned) how lightweight are pings;
• only persistent(not ephemeral) messages can be restored from before or in case of loss;
• API should be clear at communicating that messages received through such subscription can come in wrong order (in case of loss), meaning an old message from Store will be retrieved after Filter already delivered new messages;
• ideally

Side effect of the feature is: since right now Filter in js-waku and status-go has redundancy of 2-3 peers we might be able to drop this number to 1 with constantly checking if peer app subscribed to broadcasts all messages, if it is not the case - drop and replace with a new one.

Note: this issue is about delivering missing messages, where as #2017 is about scoring a peer by similar trick with Store.

[danisharora099]

Blocked by store v3: #2029

[danisharora099]

we need to know if pings(store-v3 I assume) are light weight.

Store V3 gives us an options to only return the stored message hashes, instead of the entire WakuMessage[] which is significantly more efficient for this usecase by includeData=false

is Store reliable enough for us to be source of truth

We will presume a Store node returning message hashes is reliable as previously concluded. This should further, internally, be improved with Store Sync.

only persistent(not ephemeral) messages can be restored from before or in case of loss

This is an important point IMO. To enable message recovery with Store, we will have to by default make all messages non-ephemeral. This perhaps comes under a good default for the js-waku SDK. Adding a description over the ephemeral option for message should come with a disclaimer that reliability MIGHT take a toll.

API should be clear at communicating that messages received through such subscription can come in wrong order (in case of loss), meaning an old message from Store will be retrieved after Filter already delivered new messages
Side effect of the feature is: since right now Filter in js-waku and status-go has redundancy of 2-3 peers we might be able to drop this number to 1 with constantly checking if peer app subscribed to broadcasts all messages, if it is not the case - drop and replace with a new one.

Sounds good!

@weboko
cc @chaitanyaprem might be relevant for your side of work on Filter reliability
cc @jm-clius do you have any thoughts re this?

[jm-clius]

Indeed. The Store v3 "ping" functionality is indeed there to have a (fairly) lightweight mechanism to retrieve only message hashes based on some query criteria.
We should consider "ephemeral" messages as best effort, also from a publishing POV - these messages will not make use of any Store-enhanced reliability mechanisms.
For now, assuming that Store is reliable is a reasonable abstraction for lightpush and filter.

My only comment would be that redundant Store checks should be avoided: lightpush can make use of regular Store checks to verify that a message has been published, filter can use the same checks to detect and retrieve missing received messages. In fact, this mechanism is agnostic as to what publish/subscribe methods are used and can be equally well applied to Relay.

[weboko]

Unblocked, moving to todo.