Pubsub mechanism for Parachains

[bkchr]

When parachains need to exchange information, they currently rely on sending XCM messages to each other or resorting to an off-chain protocol to obtain a storage proof from another chain. However, these solutions are complex and expensive, particularly when data should be disseminated amongst a large group of parachains, such as the price of DOT.

A potential solution would be the introduction of a publish-subscribe (pub-sub) mechanism. This would enable parachains to subscribe to (para_id, key) information within their runtime. Collators would then be responsible for integrating this information into the runtime during block creation. The information would be published by parachains as an output of their validation process, necessitating changes to the ValidationResult as specified in issue paritytech/polkadot-sdk#645.

The (k, v) information would be stored in the relay chain state, and it would likely be most efficient to maintain a separate child trie for each para_id. This would allow a collator to notify the parachain runtime when a change occurs in one of the subscribed items.

However, the parachain runtime cannot fully trust the collator to include the data. Therefore, we only want to include the data when changes occur to conserve space. In such cases, the collator could include the child trie root (as part of the relay chain proof), which would be sufficient to prove that no changes occurred or provide proof of the child trie with the changed data.

CC @gavofyork

[vstam1]

Hi @bkchr,
I was wondering if you have an update on the pubsub mechanism? I think a lot of parachains would really benefit from this feature, and I was wondering if there is a way we can contribute?
Thanks!

[JuaniRios]

I also could really use this feature

[wischli]

Same here!

[bkchr]

Yeah we have talked a little bit "off github" about this. The current plan is the following:

1. Introduce a new XCM message to publish certain information. This will be send from the parachain to the relay chain.
2. Then have the parachain read the relay chain state and put it into the ValidationData inherent. (I'm working on this already, but on some kind of optimized version.)

If you need this, could you may look into 1? Providing 2 in some "dirty" way should be possible quite easily.

[vstam1]

Thanks for your quick response. I was wondering if there is another way of publishing the information in step 1 without XCM? Or at least without a new XCM instruction. This new instruction will then probably be introduced in XCM v5 and that can still take months. Might we be able to use the Transact instruction to publish the information?

[bkchr]

@franciscoaguirre what is the status on XCMv4? Still possible to sneak in some new instruction?

[franciscoaguirre]

We can. We wanted to get it out as soon as possible but the PR still doesn't have enough reviews. A new instruction wouldn't hurt

[bkchr]

Nice! Could we add some kind of instruction like this:

Publish { data: Vec<(Key, Value)>

While Key and Value just being Vec<u8>.

[franciscoaguirre]

We can work on the implementation here: #2359, it's pointing to the v4 branch.

In the meantime, I created an RFC for this to discuss how it should work: polkadot-fellows/xcm-format#48
I feel like this could be a more generic instruction like Store that could potentially be used for storage parachains in the future. I see many people want this, please go and comment in there to make a nice new addition to the language :)

[franciscoaguirre]

Since XCMv4 was just a rename, this'll go in XCMv5. I urge you again to take a look at the RFC and comment. This will only go into v5 if we're all convinced about that RFC.