CIP-0137? | Decentralized Message Queue

[jpraynaud]

This proposal specifies a decentralized message queue leveraging the Cardano network layer. This protocol allows a topic based mechanism to diffuse messages from publishers to consumers in a decentralized way.

(rendered proposal)

[coot]

I'd suggest a different title for the PR/CIP; what about Mithril over Ouroboros-Network?

[rphair]

@coot (also responding to #876 (comment)) it depends on whether there will someday be another application of this method besides the one being developed for Mithril.

Will that be possible? From my level of understanding (not intimate) that possibility is suggested in the Motivation:

probably other protocols in the future

[rphair]

Some formatting changes that would be necessary to comply with CIP-0001 as mentioned here: #876 (comment)

[rphair]

(intended to appear in sequence with above review, but GitHub ate it)

[rphair]

(p.s.)

[rphair]

thanks @ch1bo for rationalising the header & document outline. Another thing that evolved in CIP-0001 (which always seemed pedantic to me but we have seen positive effects from in better organised, more readable CIP productions) are the more explicit titles below. At the very least you will be making your document headers consistent with all other merged CIPs :)

https://github.com/cardano-foundation/CIPs/blob/master/CIP-0001/README.md#structure

[rphair]

Tagging as Unconfirmed to reflect that this was positively received at CIP meeting # 94 but also that particulars like the Category: and CIP title are not definite yet. This tag will also press it toward review at a future CIP meeting but the editors will wait for confirmation from the authors first that this is relatively complete & ready for community review.

[coot]

@rphair replying to #876 (comment)

This opens many different applications, at least two of which were discussed internally recently: verification of tx-s making into the blockchain; distributing ledger peers in a decentralised way and many others.

[rphair]

OK @coot then it does sound like the current PR / CIP title Decentralized Message Queue is appropriately general & specific, and that the too-specific word Mithril should be kept out of the title (as per suggestion in #876 (comment)).

[coot]

@rphair I still think that my suggestion is better suited. If I recall correctly this CIPs doesn't really mention what Mithril will be used for (nor the name Decentralised Message Queue suggests it).

[rphair]

@coot I'll make sure that every time this proposal is reviewed at CIP meetings that we talk about how the name & scope can be best related. Also I hope to get more of the co-authors posting about this question. It's above my level to assess so I would be inclined to follow whatever you & the authors would ultimately decide together.

[coot]

@rphair could you please invite me to the CIP meeting when this CIP is discussed?

[rphair]

@coot I think enough issues have come up in the discussion that this would be ready for Review now, so I've put it on the agenda for the next meeting (or can postpone by 2-week intervals at the same time of day, at your or other authors' request): https://hackmd.io/@cip-editors/96

I would first want to assign this a CIP number but normally we don't do that for a CIP without a Category:. So the highest order for review (cc @Ryun1 @Crypto2099) would be for someone to summarise what's happening with the Consensus category for the CIP process so it can be assigned here, or possibly confirm Network as an alternative (#876 (comment) - cc @ch1bo @jpraynaud @abailly-iohk + @bwbush @dnadales from #872 (comment)).

We can also try to future-proof the title with respect to scope as per #876 (comment).

Meeting on CIP Discord (invite to channel: https://discord.gg/J8sGdCuKhs)

[jpraynaud]

@rphair I still think that my suggestion is better suited. If I recall correctly this CIPs doesn't really mention what Mithril will be used for (nor the name Decentralised Message Queue suggests it).

@coot the original idea is to have a generic design for the Decentralized Message Queue.

Mithril would be the first protocol using it but, as it is clearly mentioned in the Motivation section, other protocols (Leios, Peras, ...) will likely use it as well in the future.

[rphair]

@jpraynaud @abailly-iohk - applying freshly created label Category: Network Proposals belonging to the `Network` category. - please chime in with this issue with any feelings about how you see the CIP process participation for this category developing:

• Project Enlisting - Rethinking Post-Chang #898

In your opinion, would this be ready for Review at the next CIP meeting? (https://hackmd.io/@cip-editors/98) If not, please post something about how this proposal has been progressing & what we're waiting for.

[rphair]

p.s. to #876 (comment) @jpraynaud @abailly-iohk - another advantage to putting this up for Review again is that we could assign it a CIP number, especially now that questions about scope & categorisation seem to have been answered. RSVP

[jpraynaud]

p.s. to #876 (comment) @jpraynaud @abailly-iohk - another advantage to putting this up for Review again is that we could assign it a CIP number, especially now that questions about scope & categorisation seem to have been answered. RSVP

@rphair I think that we will be ready to have another review next Tuesday and eventually get a CIP number 👍
I will probably make few minor modifications in the mean time.

[rphair]

thanks @jpraynaud - I've put it on the agenda again & hope you can attend: https://hackmd.io/@cip-editors/98

[jpraynaud]

thanks @jpraynaud - I've put it on the agenda again & hope you can attend: https://hackmd.io/@cip-editors/98

Thanks @rphair, I'll be attending the next CIP meeting 👍.

[rphair]

Thanks @jpraynaud & team for such a close cooperation with the editors & community on this one: well confirmed as candidate in today's CIP meeting & therefore marking Last Check & therefore ready to merge unless any problems are identified in the next 2 weeks or so.

Please also update to the new CIP number by renaming the containing directory & change the proposal link in this PR's first posting accordingly. 🎉

Personally I believe this has passed all documentation requirements & peer review and will ✅ it as soon as this number assignment is made in the document:

[perturbing]

I believe the motivation section could benefit from a more generalized framing before diving into the specific application of Mithril. As it currently stands, the document immediately presents Mithril as the driving example, which might confuse readers who are looking for a broader understanding of the need for a decentralized message queue.

I would suggest starting with a more generalized motivation that emphasizes the overarching need for decentralized message diffusion across various protocols in the Cardano ecosystem. The benefits of using the existing Cardano infrastructure for this purpose—such as efficiency, security, and reduced development overhead—should be highlighted first. Mithril can then be introduced as a specific use case, perhaps noting that it will be used as an illustrative example throughout the document.

While I understand Mithril will be a fundamental first user of this DMQ, it's important that the document doesn't feel too Mithril-centric.

[jpraynaud]

Thanks @perturbing for this feedback. I will adjust the motivation section in that manner shortly 👍

[jpraynaud]

@perturbing I have updated the Motivation section to be less Mithril-centric in the commit cardano-scaling@7b304a5

[rphair]

thanks @jpraynaud - I think 7b304a5 does a great job of putting this CIP into context without being too specific with respect to Milthril.

[Quantumplation]

I might have missed it in my quick read, but does this deal with how long messages should be retained for diffusion by each node? even if it's not enforceable at the protocol level, if this is meant to be a generic mechanism, it'd be useful to know the bounds on availability for people intending to use this mechanism.

For example, does the node only propagate the messages to the currently connected peers? does it keep them available for download for 24 hours? does it keep them permanently? etc.

[jpraynaud]

I might have missed it in my quick read, but does this deal with how long messages should be retained for diffusion by each node? even if it's not enforceable at the protocol level, if this is meant to be a generic mechanism, it'd be useful to know the bounds on availability for people intending to use this mechanism.

For example, does the node only propagate the messages to the currently connected peers? does it keep them available for download for 24 hours? does it keep them permanently? etc.

Thanks @Quantumplation for raising this question.

The messages will have a lifetime configured a the topic level: messages will be considered invalid after that lifetime has elapsed, and will neither be diffused to other network nodes via the submission n2n submission mini-protocol nor accessible by the n2c notification mini-protocol.

I have detailed this invalidation mechanism and some figures for the overall extra storage requirements in the case of Mithril in the commit cardano-scaling@f97083e.

[Crypto2099]

All community feedback has been sufficiently addressed, ready to merge in.