From dedffe641d0ac7a7c5817ba253afc1d987a167a7 Mon Sep 17 00:00:00 2001 From: Wondertan Date: Mon, 12 Feb 2024 20:00:17 +0100 Subject: [PATCH 01/21] docs: initial Shwap CIP draft --- cips/cip-shwap_protocol.md | 60 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 cips/cip-shwap_protocol.md diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md new file mode 100644 index 0000000..e765c40 --- /dev/null +++ b/cips/cip-shwap_protocol.md @@ -0,0 +1,60 @@ +--- +title: Shwap Protocol +description: Shwap - the new extensible sampling protocol +author: Hlib Kanunnikov (@Wondertan) +discussions-to: https://forum.celestia.org/t/cip-shwap-protocol/1551 +status: Draft +type: Standards Track +category: Data Availability, Networking +created: 2024-02-02 +--- + +## Abstract + +This document describes the core of Shwap p2p protocol. Shwap provides scalable and extensible framework for exchanging +and swapping of shared data for Celestia's Data Availability network and beyond. + +## Motivation + +The Shwap protocol addresses network round trip inefficiencies in Celestia's Data Availability network, initially +introduced by adopting the Bitswap protocol and IPFS/IPLD's data access patterns for Data Availability Sampling (DAS) and +block synchronization. Despite transitioning to the optimized [ShrEx protocol][shrex] for block synchronization and integrating +[CAR and DAGStore-based storage][storage], data sampling performance was left untouched, preserving low network performance. Shwap +specifically addresses and mitigates these inefficiencies, improving operations and overall network efficiency. + +## Specification + +[Shwap Protocol Specification][spec] + +## Backwards Compatibility + +Shwap in incompatible with old sampling protocol. + +After rigorous investigation, celestia-node team decided against _implementing_ backwards compatibility with +the old protocol into the node client due to immense complications it brings. Instead, the simple and time efficient +strategy is to transiently deploy infrastructure for old and new versions, allowing network participants to gradually +migrate to the new version. Once, we know that majority has migrated, we will terminate old infrastructure and deprecate +old version. + +## Reference Implementation + +- [Go reference implementation][gimpl] +- [Rust implementation][rimpl] + +## Security Considerations + +Shwap does not change the security model of Celestia's Data Availability network and changes the underlying +protocol for data retrieval. + +Largely, the network and its codebase get simplified and requires less code and infrastructure to operate. This in turn +decreases amount of implementation vulnerabilities, DOS vectors, message amplification and resource exhaustion attacks. + +## Copyright + +Copyright and related rights waived via [CC0](../LICENSE). + +[shrex]: https://github.com/celestiaorg/celestia-node/blob/0abd16bbb05bf3016595498844a588ef55c63d2d/docs/adr/adr-013-blocksync-overhaul-part-2.md +[storage]: https://github.com/celestiaorg/celestia-node/blob/a33c80e20da684d656c7213580be7878bcd27cf4/docs/adr/adr-011-blocksync-overhaul-part-1.md +[gimpl]: https://github.com/celestiaorg/celestia-node/pull/2675 +[rimpl]: https://github.com/eigerco/lumina/blob/561640072114fa5c4ed807e94882473476a41dda/node/src/p2p/shwap.rs +[spec]: https://www.youtube.com/watch?v=dQw4w9WgXcQ \ No newline at end of file From 9fa0384db236c5476ec56a636ed0677a692dcb2b Mon Sep 17 00:00:00 2001 From: Wondertan Date: Tue, 13 Feb 2024 14:04:03 +0100 Subject: [PATCH 02/21] cip: clarify abstract and split motivation into technical and historical paragraphs --- cips/cip-shwap_protocol.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index e765c40..93320c8 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -11,16 +11,23 @@ created: 2024-02-02 ## Abstract -This document describes the core of Shwap p2p protocol. Shwap provides scalable and extensible framework for exchanging -and swapping of shared data for Celestia's Data Availability network and beyond. +This document describes a high level overview of the Shwap p2p protocol. Shwap provides scalable and extensible +framework for exchanging and swapping of shared data for Celestia's Data Availability network and beyond. ## Motivation -The Shwap protocol addresses network round trip inefficiencies in Celestia's Data Availability network, initially -introduced by adopting the Bitswap protocol and IPFS/IPLD's data access patterns for Data Availability Sampling (DAS) and -block synchronization. Despite transitioning to the optimized [ShrEx protocol][shrex] for block synchronization and integrating -[CAR and DAGStore-based storage][storage], data sampling performance was left untouched, preserving low network performance. Shwap -specifically addresses and mitigates these inefficiencies, improving operations and overall network efficiency. +Current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes O(log2(N)) network +round-trips(where N is the square size). This is not practical and does not scale for theoretically unlimited data square +Celestia network enables. The main motive here is a protocol with O(1) round-trip for _multiple_ samples, preserving +the assumption of having a 1/N honest peers connected. + +Initially, the Bitswap and IPLD were adopted as a base for the DA network protocols, including DAS, +block synchronization (BS) and blob/namespace data retrieval (ND). They gave battle tested protocols and tooling with +plugability to rapidly scaffold Celestia's DA network. However, it came with the price of scalability limits and +round-trips resulting in BS slower than block production. Before network launch, the transition +to the optimized [ShrEx protocol][shrex] for BS and integrating [CAR and DAGStore-based storage][storage] happened +optimizing BS and ND. However, DAS was left untouched preserving its weak scalability and roundtrip inefficiency. Shwap +specifically addresses these and additionally provides an extensible and flexible framework for BS, ND and beyond. ## Specification From cdf6409f2f4e61217e022bd611677ac0c0db8e73 Mon Sep 17 00:00:00 2001 From: Wondertan Date: Wed, 14 Feb 2024 12:38:33 +0100 Subject: [PATCH 03/21] cip: grammar and stylistic improvements --- cips/cip-shwap_protocol.md | 42 +++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 93320c8..2440ca6 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -16,18 +16,18 @@ framework for exchanging and swapping of shared data for Celestia's Data Availab ## Motivation -Current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes O(log2(N)) network -round-trips(where N is the square size). This is not practical and does not scale for theoretically unlimited data square -Celestia network enables. The main motive here is a protocol with O(1) round-trip for _multiple_ samples, preserving -the assumption of having a 1/N honest peers connected. - -Initially, the Bitswap and IPLD were adopted as a base for the DA network protocols, including DAS, -block synchronization (BS) and blob/namespace data retrieval (ND). They gave battle tested protocols and tooling with -plugability to rapidly scaffold Celestia's DA network. However, it came with the price of scalability limits and -round-trips resulting in BS slower than block production. Before network launch, the transition -to the optimized [ShrEx protocol][shrex] for BS and integrating [CAR and DAGStore-based storage][storage] happened -optimizing BS and ND. However, DAS was left untouched preserving its weak scalability and roundtrip inefficiency. Shwap -specifically addresses these and additionally provides an extensible and flexible framework for BS, ND and beyond. +The current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes log2(k) network +round-trips(where k is the square size). This is not practical and does not scale for the theoretically unlimited data +square that the Celestia network enables. The main motive here is a protocol with O(1) round-trip for _multiple_ samples, preserving +the assumption of having 1/n honest peers connected. + +Initially, Bitswap and IPLD were adopted as the basis for the DA network protocols, including DAS, +block synchronization (BS), and blob/namespace data retrieval (ND). They gave battle-tested protocols and tooling with +pluggability to rapidly scaffold Celestia's DA network. However, it came with the price of scalability limits and +round-trips resulting in BS slower than block production. Before the network launch, the transition +to the optimized [ShrEx protocol][shrex] for BS and integrating [CAR and DAGStore-based storage][storage] happened +optimizing BS and ND. However, DAS was left untouched, preserving its weak scalability and roundtrip inefficiency. Shwap +addresses these and provides an extensible and flexible framework for BS, ND, and beyond. ## Specification @@ -35,13 +35,13 @@ specifically addresses these and additionally provides an extensible and flexibl ## Backwards Compatibility -Shwap in incompatible with old sampling protocol. +Swap is incompatible with the old sampling protocol. -After rigorous investigation, celestia-node team decided against _implementing_ backwards compatibility with -the old protocol into the node client due to immense complications it brings. Instead, the simple and time efficient -strategy is to transiently deploy infrastructure for old and new versions, allowing network participants to gradually -migrate to the new version. Once, we know that majority has migrated, we will terminate old infrastructure and deprecate -old version. +After rigorous investigation, celestia-node team decided against _implementing_ backward compatibility with +the old protocol into the node client due to the immense complications it brings. Instead, the simple and time-efficient +strategy is transiently deploying infrastructure for old and new versions, allowing network participants to migrate +gradually to the latest version. We will first deprecate the old version, and once the majority has migrated, we will +terminate the old infrastructure. ## Reference Implementation @@ -50,11 +50,11 @@ old version. ## Security Considerations -Shwap does not change the security model of Celestia's Data Availability network and changes the underlying +Shwap does not change the security model of Celestia's Data Availability network and changes the underlying protocol for data retrieval. -Largely, the network and its codebase get simplified and requires less code and infrastructure to operate. This in turn -decreases amount of implementation vulnerabilities, DOS vectors, message amplification and resource exhaustion attacks. +Essentially, the network and its codebase get simplified and require less code and infrastructure to operate. This in turn +decreases the amount of implementation vulnerabilities, DOS vectors, message amplification, and resource exhaustion attacks. ## Copyright From 72adf41db19383d4c118adc01de196d6b15d2844 Mon Sep 17 00:00:00 2001 From: Wondertan Date: Wed, 14 Feb 2024 17:46:58 +0100 Subject: [PATCH 04/21] cip: add @rootulp 's suggestion --- cips/cip-shwap_protocol.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 2440ca6..527c879 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -12,7 +12,8 @@ created: 2024-02-02 ## Abstract This document describes a high level overview of the Shwap p2p protocol. Shwap provides scalable and extensible -framework for exchanging and swapping of shared data for Celestia's Data Availability network and beyond. +framework for exchanging and swapping of shared data for Celestia's Data Availability network and beyond. Shwap is a +portmanteau of share and swap. ## Motivation From 2cce54ebe9b6c52efcb80cffcdac5b406b3b18f1 Mon Sep 17 00:00:00 2001 From: Wondertan Date: Mon, 4 Mar 2024 18:38:29 +0100 Subject: [PATCH 05/21] merge in the spec and improve the CIP --- cips/cip-shwap_protocol.md | 344 ++++++++++++++++++++++++++++++++++--- 1 file changed, 324 insertions(+), 20 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 527c879..eec5cfb 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -11,51 +11,345 @@ created: 2024-02-02 ## Abstract -This document describes a high level overview of the Shwap p2p protocol. Shwap provides scalable and extensible -framework for exchanging and swapping of shared data for Celestia's Data Availability network and beyond. Shwap is a -portmanteau of share and swap. +This document specifies Shwap - the simple and expressive yet extensible and future-proof messaging framework aiming to +solve critical inefficiencies and standardize messaging of Celestia's Data Availability p2p network. + +Shwap defines a messaging framework to be exchanged around the DA p2p network in a trust-minimized way without enforcing +transport(QUIC/TCP or IP) or application layer protocol semantics(e.g., HTTP/x). Using this framework, Shwap +declares the most common messages and provides options for stacking them with lower-level protocols. +Shwap can be stacked together with application protocol like HTTP/x, [KadDHT][kaddht], [Bitswap][bitswap] or any custom +protocol. ## Motivation -The current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes log2(k) network -round-trips(where k is the square size). This is not practical and does not scale for the theoretically unlimited data -square that the Celestia network enables. The main motive here is a protocol with O(1) round-trip for _multiple_ samples, preserving -the assumption of having 1/n honest peers connected. +The current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes log2(k) +network roundtrips (where k is the square size). This is not practical and does not scale for the theoretically unlimited +data square that the Celestia network enables. The main motive here is a protocol with O(1) roundtrip for _multiple_ +samples, preserving the assumption of having 1/n honest peers connected. Initially, Bitswap and IPLD were adopted as the basis for the DA network protocols, including DAS, block synchronization (BS), and blob/namespace data retrieval (ND). They gave battle-tested protocols and tooling with pluggability to rapidly scaffold Celestia's DA network. However, it came with the price of scalability limits and -round-trips resulting in BS slower than block production. Before the network launch, the transition +roundtrips, resulting in slower BS than block production. Before the network launch, the transition to the optimized [ShrEx protocol][shrex] for BS and integrating [CAR and DAGStore-based storage][storage] happened -optimizing BS and ND. However, DAS was left untouched, preserving its weak scalability and roundtrip inefficiency. Shwap -addresses these and provides an extensible and flexible framework for BS, ND, and beyond. +optimizing BS and ND. However, DAS was left untouched, preserving its weak scalability and roundtrip inefficiency. + +Shwap messaging stacked together with Bitswap protocol directly addresses described inefficiency and provides a foundation +for efficient communication for BS, ND, and beyond. + +## Rationale + +The atomic primitive of Celestia's DA network is the share. Shwap standardizes messaging and serialization for shares. +Shares are grouped together, forming more complex data types(Rows, Blobs, etc.). These data types are encapsulated in +containers, e.g., Row container groups shares of a particular row. Containers can be identified with the share identifiers +in order to request, advertise or index the containers. The combination of containers and identifiers provides an extensible +and expressive messaging framework for groups of shares and enables efficient single roundtrip request-response +communication. + +Many share groups or containers are known in the Celestia network, and systemizing this is the main reason behind setting +up this simple messaging framework. A single place with all the possible Celestia DA messages must be defined, which node +software and protocol researchers can rely on and coordinate. Besides, this framework is designed to be +future-proof and sustain changes in the core protocol's data structures and proving system as long shares stay the +de facto atomic data type. + +Besides, there needs to be systematization and a joint knowledge base with all the edge cases for possible protocol +compositions of Shwap with lower-level protocols Bitswap, KadDHT, or Shrex, which Shwap aims to describe. ## Specification -[Shwap Protocol Specification][spec] +### Terms and Definitions + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in BCP +14 [RFC2119] [RFC8174] when, and only when, they appear in all +capitals, as shown here. + +Commonly used terms in this document are described below. + +_**Shwap**_: The protocol described by this document. Shwap is a +portmanteau name of words share and swap. + +_**[Share][shares]**_: The core data structure of DataSquare **"swapped"** between peers. + +_**[DataSquare][square]**_: The DA square format used by Celestia DA network. + +_**[DAH][dah]**_: The Data Availability Header with Row and Column commitments. + +_**[Namespace][ns]**_: The namespace grouping sets of shares. + +_**Peer**_: An entity that can participate in a Shwap protocol. There are three types of peers: +client, server, and node. + +_**Client**_: The Peer that requests content by content identifies over Shwap. + +_**Server**_: The Peer that responds with content over Shwap. + +_**Node**_: The peer that is both the client and the server. + +_**Proof**_: A Merkle inclusion proof of the data in the DataSquare. + +### Message Framework + +This section defines Shwap's messaging framework. Every group of shares that needs to be exchanged over the network +MUST define its [share identifier](#share-identifiers) and [share container](#share-containers), as well as, follow +their described rules. + +#### Share Identifiers + +Identifiers MUST have a fixed size for their fields. Subsequently, protobuf SHOULD NOT be used for CID serialization due +to varints and lack of fixed size arrays. Instead, identifiers use simple binary big endian serialization. + +Identifiers MAY embed each other to narrow down the scope of needed shares. For example, [SampleID](#sampleid) embeds +[RowID](#rowid) as every sample lay on a particular row. + +#### Share Containers + +Share containers encapsulate a set of data shares with [DAH][dah] inclusion proof. Share containers are identified by +[share identifiers](#share-identifiers). + +#### Versioning + +If a defined share container or identifier requires an incompatible change, the new message type MAY be introduced +suffixed with a new major version starting from v1. E.g., if the Row message needs a revision, RowV1 is created. + +### Messages + +This section defines all the supported Shwap messages, including share identifiers and containers. All the new +future messages should be described in the section. + +#### RowID + +RowID identifies the [Row shares container](#row-container) in a [DataSquare][square]. + +RowID identifiers are formatted as shown below: + +```text +RowID { + Height: u64; + RowIndex: u16; +} +``` + +The fields with validity rules that form RowID are: + +**Height**: A uint64 representing the chain height with the data square. It MUST be bigger than zero. + +**RowIndex**: An uint16 representing row index points to a particular row. It MUST not exceed the number of Row roots in +[DAH][dah]. + +Serialized RowID MUST have a length of 10 bytes. + +#### Row Container + +Row containers are protobuf formatted using the following proto3 schema: + +```protobuf +syntax = "proto3"; + +message Row { + bytes row_id = 1; + repeated bytes row_half = 2; +} +``` + +The fields with validity rules that form Row containers are: + +[**RowID**](#rowid): A RowID of the Row Container. It MUST follow [RowID](#rowid) formatting and field validity rules. + +**RowHalf**: A two-dimensional variable size byte arrays representing left half of shares in the row. It MUST equal the number of Columns roots in [DAH][dah] divided by two. These shares MUST only be from the left half of the row. +The right half is computed using Leopard GF16 Reed-Solomon erasure-coding. Afterward, the [NMT][nmt] is built over both +halves and the computed NMT root MUST be equal to the respective Row root in [DAH][dah]. + +#### SampleID + +SampleID identifies a Sample container of a single share in a [DataSquare][square]. + +SampleID identifiers are formatted as shown below: + +```text +SampleID { + RowID; + ColumnIndex: u16; +} +``` + +The fields with validity rules that form SampleID are: + +[**RowID**](#rowid): A RowID of the sample. It MUST follow [RowID](#rowid) formatting and field validity rules. + +**ColumnIndex**: A uint16 representing the column index of the sampled share; in other words, the share index in the row. It +MUST stay within the number of Column roots in [DAH][dah]. + +Serialized SampleID MUST have a length of 12 bytes. + +#### Sample Container + +Sample containers encapsulate single shares of the [DataSquare][square]. + +Sample containers are protobuf formatted using the following proto3 schema: + +```protobuf +syntax = "proto3"; + +message sample { + bytes sample_id = 1; + bytes sample_share = 2; + Proof sample_proof = 3; + ProofType proof_type = 4; +} + +enum ProofType { + RowProofType = 0; + ColProofType = 1; +} +``` + +The fields with validity rules that form Sample containers are: + +[**SampleID**](#sampleid): A SampleID of the Sample container. It MUST follow [SampleID](#sampleid) formatting and field +validity rules. + +**SampleShare**: A variable size array representing the share contained in the sample. Each share MUST follow [share +formatting and validity][shares-format] rules. + +**Proof**: A [protobuf formated][nmt-pb] [NMT][nmt] proof of share inclusion. It MUST follow [NMT proof verification][nmt-verify] +and be verified against the respective root from the Row or Column axis in [DAH][dah]. The axis is defined by the ProofType field. + +**ProofType**: An enum defining which axis root the Proof is coming from. It MUST be either RowProofType or ColumnProofType. + +#### DataID + +DataID identifies [namespace][ns] Data container of shares within a _single_ Row. That is, namespace shares spanning +over multiple Rows are identified with multiple identifiers. + +DataID identifiers are formatted as shown below: + +```text +DataID { + RowID; + Namespace; +} +``` + +The fields with validity rules that form DataID are: + +[**RowID**](#rowid): A RowID of the namespace data. It MUST follow [RowID](#rowid) formatting and field validity rules. + +[**Namespace**][ns]: A fixed-size bytes array representing the Namespace of interest. It MUST follow [Namespace][ns] +formatting and its validity rules. + +Serialized DataID MUST have a length of 39 bytes. + +#### Data Container + +Data containers encapsulate user-submitted data under [namespaces][ns]. + +Data containers are protobuf formatted using the following proto3 schema: + +```protobuf +syntax = "proto3"; + +message Data { + bytes data_id = 1; + repeated bytes data_shares = 2; + Proof data_proof = 3; +} +``` + +The fields with validity rules that form Data containers are: + +[**DataID**](#dataid): A DataID of the Data container. It MUST follow [DataID](#dataid) formatting and field validity +rules. + +**DataShares**: A two-dimensional variable size byte arrays representing left data shares of a namespace in the row. +Each share MUST follow [share formatting and validity][shares-format] rules. + +**Proof**: A [protobuf formated][nmt-pb] [NMT][nmt] proof of share inclusion. It MUST follow [NMT proof verification][nmt-verify] +and be verified against the respective root from the Row or Column axis in [DAH][dah]. The axis is defined by the ProofType field. + +Namespace data may span over multiple rows, in which case all the data is encapsulated in multiple containers. This is +done + +## Protocol Compositions + +This section specifies compositions of Shwap with other protocols. While Shwap is transport agnostic, there are rough +edges on the protocol integration, which every composition specification has to describe. + +### Bitswap + +[Bitswap][bitswap] is an application-level protocol for sharing verifiable data across peer-to-peer networks. +Bitswap operates as a dynamic want-list exchange among peers in a network. Peers continuously update and share their +want lists of desired data in real time. It is promptly fetched if at least one connected peer has the needed data. +This ongoing exchange ensures that as soon as any peer acquires the sought-after data, it can instantly share it with +those in need. + +Shwap is designed to be synergetic with Bitswap, as that is the primary composition to be deployed in Celestia's DA +network. Bitswap provides the 1/N peers guarantee and can parallelize fetching across multiple peers. Both of these properties +significantly contribute to Celestia's efficient DAS protocol. + +Bitswap runs over the libp2p stack, which provides QUIC transport integration. Subsequently, Shwap will benefit from features +libp2p provides together with transport protocol advancements introduced in QUIC. + +#### Multihashes and CID + +Bitswap is tightly coupled with Multihash and CID notions, establishing the [content addressability property][content-address]. +Bitswap operates over Blocks of data that are addressed and verified by CIDs. Based on that, Shwap integrates into +Bitswap by complying with both of these interfaces. The [Share Containers](#share-containers) are Blocks that are identified +via [Share Identifiers](#share-identifiers). + +Even though Shwap takes inspiration from content addressability, it breaks free from the hash-based model to optimize +message sizes and data request patterns. In some way, it hacks into multihash abstraction to make it contain data that +is not, in fact, a hash. Furthermore, the protocol does not include hash digests in the multihashes. The authentication of +the messages happens using externally provided data commitment. + +This creates a bunch of complexities with the [reference Golang implementation][gimpl] that are necessary if forking +and substantially diverging the upstream is not an option. The naive question would be: "Why not make content +verification after Bitswap provided it back over its API?" Intuitively, this would simplify much and would not require +"hacking" CID. However, this has an important downside - the Bitswap, in such a case, would consider the request finalized +and the content as fetched and valid, sending a DONT_WANT message to its peers. In contrast, the message might still be invalid +according to the verification rules. + +However, Bitswap still requires multihashes and CID codecs to be registered. Therefore, we provide a table for the +supported [share identifiers](#share-identifiers) with their respective multihash and CID codec codes. This table +should be extended whenever any new share identifier is added. + +| Name | Multihash | Codec | +|----------|-----------|--------| +| RowID | 0x7811 | 0x7810 | +| SampleID | 0x7801 | 0x7800 | +| DataID | 0x7821 | 0x7820 | ## Backwards Compatibility Swap is incompatible with the old sampling protocol. -After rigorous investigation, celestia-node team decided against _implementing_ backward compatibility with +After rigorous investigation, the celestia-node team decided against _implementing_ backward compatibility with the old protocol into the node client due to the immense complications it brings. Instead, the simple and time-efficient -strategy is transiently deploying infrastructure for old and new versions, allowing network participants to migrate -gradually to the latest version. We will first deprecate the old version, and once the majority has migrated, we will +strategy is transiently deploying infrastructure for old and new versions, allowing network participants to migrate +gradually to the latest version. We will first deprecate the old version, and once the majority has migrated, we will terminate the old infrastructure. -## Reference Implementation - -- [Go reference implementation][gimpl] -- [Rust implementation][rimpl] +## Considerations -## Security Considerations +### Security Shwap does not change the security model of Celestia's Data Availability network and changes the underlying protocol for data retrieval. Essentially, the network and its codebase get simplified and require less code and infrastructure to operate. This in turn decreases the amount of implementation vulnerabilities, DOS vectors, message amplification, and resource exhaustion attacks. +However, new bugs may be introduced, as with any new protocol. + +### Protobuf Serialization + +Protobuf is a widely adopted serialization format and is used within Celestia's protocols. This was quite an obvious choice +for consistency reasons, even though we could choose other more efficient and advanced formats like Cap'n Proto. + +## Reference Implementation + +- [Go reference implementation with Bitswap composition][gimpl] +- [Rust implementation with Bitswap composition][rimpl] ## Copyright @@ -63,6 +357,16 @@ Copyright and related rights waived via [CC0](../LICENSE). [shrex]: https://github.com/celestiaorg/celestia-node/blob/0abd16bbb05bf3016595498844a588ef55c63d2d/docs/adr/adr-013-blocksync-overhaul-part-2.md [storage]: https://github.com/celestiaorg/celestia-node/blob/a33c80e20da684d656c7213580be7878bcd27cf4/docs/adr/adr-011-blocksync-overhaul-part-1.md +[bitswap]: https://docs.ipfs.tech/concepts/bitswap/ +[content-address]: https://fission.codes/blog/content-addressing-what-it-is-and-how-it-works/ +[kaddht]: https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf +[square]: https://celestiaorg.github.io/celestia-app/specs/data_structures.html#2d-reed-solomon-encoding-scheme +[shares]: https://celestiaorg.github.io/celestia-app/specs/shares.html#abstract +[shares-format]: https://celestiaorg.github.io/celestia-app/specs/shares.html#share-format +[dah]: https://celestiaorg.github.io/celestia-app/specs/data_structures.html#availabledataheader +[ns]: https://celestiaorg.github.io/celestia-app/specs/namespace.html#abstract +[nmt]: https://github.com/celestiaorg/nmt/blob/master/docs/spec/nmt.md +[nmt-pb]: https://github.com/celestiaorg/nmt/blob/f5556676429118db8eeb5fc396a2c75ab12b5f20/pb/proof.proto +[nmt-verify]: https://github.com/celestiaorg/nmt/blob/master/docs/spec/nmt.md#namespace-proof-verification [gimpl]: https://github.com/celestiaorg/celestia-node/pull/2675 [rimpl]: https://github.com/eigerco/lumina/blob/561640072114fa5c4ed807e94882473476a41dda/node/src/p2p/shwap.rs -[spec]: https://www.youtube.com/watch?v=dQw4w9WgXcQ \ No newline at end of file From 1daa62b557cd90c4f17bc0bf35c4b2b740cdbb0e Mon Sep 17 00:00:00 2001 From: Wondertan Date: Tue, 5 Mar 2024 15:52:32 +0100 Subject: [PATCH 06/21] add a section on sampling being a separate concern --- cips/cip-shwap_protocol.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index eec5cfb..18cea92 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -346,6 +346,12 @@ However, new bugs may be introduced, as with any new protocol. Protobuf is a widely adopted serialization format and is used within Celestia's protocols. This was quite an obvious choice for consistency reasons, even though we could choose other more efficient and advanced formats like Cap'n Proto. +### Sampling and Reconstruction + +Shwap deliberately avoids specifying sampling and reconstruction logic The sampling concerns on randomness selection and +sample picking are out of Shwap's scope and a matter of following CIPs. Shwap only provides messaging for sampling(via +[SampleID](#sampleid) and [Sample container](#sample-container)). + ## Reference Implementation - [Go reference implementation with Bitswap composition][gimpl] From 247087ef6ec3079d7ed9cc3f2f138dfedce96692 Mon Sep 17 00:00:00 2001 From: Wondertan Date: Wed, 6 Mar 2024 16:20:32 +0100 Subject: [PATCH 07/21] Suggestions from @jcstein --- cips/cip-shwap_protocol.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 18cea92..7081b98 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -15,23 +15,23 @@ This document specifies Shwap - the simple and expressive yet extensible and fut solve critical inefficiencies and standardize messaging of Celestia's Data Availability p2p network. Shwap defines a messaging framework to be exchanged around the DA p2p network in a trust-minimized way without enforcing -transport(QUIC/TCP or IP) or application layer protocol semantics(e.g., HTTP/x). Using this framework, Shwap +transport (QUIC/TCP or IP) or application layer protocol semantics (e.g., HTTP/x). Using this framework, Shwap declares the most common messages and provides options for stacking them with lower-level protocols. Shwap can be stacked together with application protocol like HTTP/x, [KadDHT][kaddht], [Bitswap][bitswap] or any custom protocol. ## Motivation -The current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes log2(k) +The current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes log₂(k) network roundtrips (where k is the square size). This is not practical and does not scale for the theoretically unlimited data square that the Celestia network enables. The main motive here is a protocol with O(1) roundtrip for _multiple_ -samples, preserving the assumption of having 1/n honest peers connected. +samples, preserving the assumption of having 1/N honest peers connected. Initially, Bitswap and IPLD were adopted as the basis for the DA network protocols, including DAS, block synchronization (BS), and blob/namespace data retrieval (ND). They gave battle-tested protocols and tooling with pluggability to rapidly scaffold Celestia's DA network. However, it came with the price of scalability limits and roundtrips, resulting in slower BS than block production. Before the network launch, the transition -to the optimized [ShrEx protocol][shrex] for BS and integrating [CAR and DAGStore-based storage][storage] happened +to the optimized [ShrEx protocol][shrex] for BS and integrating [CAR and DAGStore-based storage][storage] happened, optimizing BS and ND. However, DAS was left untouched, preserving its weak scalability and roundtrip inefficiency. Shwap messaging stacked together with Bitswap protocol directly addresses described inefficiency and provides a foundation @@ -40,7 +40,7 @@ for efficient communication for BS, ND, and beyond. ## Rationale The atomic primitive of Celestia's DA network is the share. Shwap standardizes messaging and serialization for shares. -Shares are grouped together, forming more complex data types(Rows, Blobs, etc.). These data types are encapsulated in +Shares are grouped together, forming more complex data types (Rows, Blobs, etc.). These data types are encapsulated in containers, e.g., Row container groups shares of a particular row. Containers can be identified with the share identifiers in order to request, advertise or index the containers. The combination of containers and identifiers provides an extensible and expressive messaging framework for groups of shares and enables efficient single roundtrip request-response @@ -269,7 +269,7 @@ Each share MUST follow [share formatting and validity][shares-format] rules. and be verified against the respective root from the Row or Column axis in [DAH][dah]. The axis is defined by the ProofType field. Namespace data may span over multiple rows, in which case all the data is encapsulated in multiple containers. This is -done +done. ## Protocol Compositions @@ -348,7 +348,7 @@ for consistency reasons, even though we could choose other more efficient and ad ### Sampling and Reconstruction -Shwap deliberately avoids specifying sampling and reconstruction logic The sampling concerns on randomness selection and +Shwap deliberately avoids specifying sampling and reconstruction logic. The sampling concerns on randomness selection and sample picking are out of Shwap's scope and a matter of following CIPs. Shwap only provides messaging for sampling(via [SampleID](#sampleid) and [Sample container](#sample-container)). From 5fd040a3fcd99c7eda76836e4fe39ecb97697177 Mon Sep 17 00:00:00 2001 From: Wondertan Date: Wed, 6 Mar 2024 16:40:55 +0100 Subject: [PATCH 08/21] Suggestions from @YazzyYaz --- cips/cip-shwap_protocol.md | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 7081b98..ab90dce 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -1,6 +1,6 @@ --- title: Shwap Protocol -description: Shwap - the new extensible sampling protocol +description: Shwap - a new messaging framework for DA and sampling author: Hlib Kanunnikov (@Wondertan) discussions-to: https://forum.celestia.org/t/cip-shwap-protocol/1551 status: Draft @@ -11,7 +11,7 @@ created: 2024-02-02 ## Abstract -This document specifies Shwap - the simple and expressive yet extensible and future-proof messaging framework aiming to +This document specifies Shwap - the simple and expressive yet extensible messaging framework aiming to solve critical inefficiencies and standardize messaging of Celestia's Data Availability p2p network. Shwap defines a messaging framework to be exchanged around the DA p2p network in a trust-minimized way without enforcing @@ -30,9 +30,9 @@ samples, preserving the assumption of having 1/N honest peers connected. Initially, Bitswap and IPLD were adopted as the basis for the DA network protocols, including DAS, block synchronization (BS), and blob/namespace data retrieval (ND). They gave battle-tested protocols and tooling with pluggability to rapidly scaffold Celestia's DA network. However, it came with the price of scalability limits and -roundtrips, resulting in slower BS than block production. Before the network launch, the transition -to the optimized [ShrEx protocol][shrex] for BS and integrating [CAR and DAGStore-based storage][storage] happened, -optimizing BS and ND. However, DAS was left untouched, preserving its weak scalability and roundtrip inefficiency. +roundtrips, resulting in slower BS than block production. Before the network launch, we transitioned to the optimized +[ShrEx protocol][shrex] for BS and integrated [CAR and DAGStore-based storage][storage] optimizing BS and ND. However, +DAS was left untouched, preserving its weak scalability and roundtrip inefficiency. Shwap messaging stacked together with Bitswap protocol directly addresses described inefficiency and provides a foundation for efficient communication for BS, ND, and beyond. @@ -41,16 +41,15 @@ for efficient communication for BS, ND, and beyond. The atomic primitive of Celestia's DA network is the share. Shwap standardizes messaging and serialization for shares. Shares are grouped together, forming more complex data types (Rows, Blobs, etc.). These data types are encapsulated in -containers, e.g., Row container groups shares of a particular row. Containers can be identified with the share identifiers -in order to request, advertise or index the containers. The combination of containers and identifiers provides an extensible -and expressive messaging framework for groups of shares and enables efficient single roundtrip request-response -communication. +containers, For example, row container groups shares of a particular row. Containers can be identified with the share +identifiers in order to request, advertise or index the containers. The combination of containers and identifiers +provides an extensible and expressive messaging framework for groups of shares and enables efficient single roundtrip +request-response communication. Many share groups or containers are known in the Celestia network, and systemizing this is the main reason behind setting up this simple messaging framework. A single place with all the possible Celestia DA messages must be defined, which node -software and protocol researchers can rely on and coordinate. Besides, this framework is designed to be -future-proof and sustain changes in the core protocol's data structures and proving system as long shares stay the -de facto atomic data type. +software and protocol researchers can rely on and coordinate. Besides, this framework is designed to be sustain changes +in the core protocol's data structures and proving system as long shares stay the de facto atomic data type. Besides, there needs to be systematization and a joint knowledge base with all the edge cases for possible protocol compositions of Shwap with lower-level protocols Bitswap, KadDHT, or Shrex, which Shwap aims to describe. From c40334f7d5159ea9739371a40fc139e65b66df3f Mon Sep 17 00:00:00 2001 From: Wondertan Date: Wed, 6 Mar 2024 16:55:15 +0100 Subject: [PATCH 09/21] reorder mhes and codecs --- cips/cip-shwap_protocol.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index ab90dce..b759cf4 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -315,8 +315,8 @@ should be extended whenever any new share identifier is added. | Name | Multihash | Codec | |----------|-----------|--------| -| RowID | 0x7811 | 0x7810 | -| SampleID | 0x7801 | 0x7800 | +| RowID | 0x7801 | 0x7800 | +| SampleID | 0x7811 | 0x7810 | | DataID | 0x7821 | 0x7820 | ## Backwards Compatibility From 6b165f6236d157548c62113c91844cde8d3be96c Mon Sep 17 00:00:00 2001 From: Wondertan Date: Wed, 6 Mar 2024 16:56:47 +0100 Subject: [PATCH 10/21] clarify data possession --- cips/cip-shwap_protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index b759cf4..dab74e0 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -25,7 +25,7 @@ protocol. The current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes log₂(k) network roundtrips (where k is the square size). This is not practical and does not scale for the theoretically unlimited data square that the Celestia network enables. The main motive here is a protocol with O(1) roundtrip for _multiple_ -samples, preserving the assumption of having 1/N honest peers connected. +samples, preserving the assumption of having 1/N honest peers connected possessing the data. Initially, Bitswap and IPLD were adopted as the basis for the DA network protocols, including DAS, block synchronization (BS), and blob/namespace data retrieval (ND). They gave battle-tested protocols and tooling with From 082ebfce7539e50025e4d0b5311c18f024d8163f Mon Sep 17 00:00:00 2001 From: Wondertan Date: Wed, 6 Mar 2024 18:35:11 +0100 Subject: [PATCH 11/21] fixing review comments from @ebuchman --- cips/cip-shwap_protocol.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index dab74e0..fcc6df0 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -80,9 +80,9 @@ _**[Namespace][ns]**_: The namespace grouping sets of shares. _**Peer**_: An entity that can participate in a Shwap protocol. There are three types of peers: client, server, and node. -_**Client**_: The Peer that requests content by content identifies over Shwap. +_**Client**_: The Peer that requests content by share identifiers over Shwap. -_**Server**_: The Peer that responds with content over Shwap. +_**Server**_: The Peer that responds with shares over Shwap. _**Node**_: The peer that is both the client and the server. @@ -91,8 +91,7 @@ _**Proof**_: A Merkle inclusion proof of the data in the DataSquare. ### Message Framework This section defines Shwap's messaging framework. Every group of shares that needs to be exchanged over the network -MUST define its [share identifier](#share-identifiers) and [share container](#share-containers), as well as, follow -their described rules. +MUST define its [share identifier](#share-identifiers) and [share container](#share-containers) and follow their described rules. #### Share Identifiers @@ -156,9 +155,10 @@ The fields with validity rules that form Row containers are: [**RowID**](#rowid): A RowID of the Row Container. It MUST follow [RowID](#rowid) formatting and field validity rules. -**RowHalf**: A two-dimensional variable size byte arrays representing left half of shares in the row. It MUST equal the number of Columns roots in [DAH][dah] divided by two. These shares MUST only be from the left half of the row. -The right half is computed using Leopard GF16 Reed-Solomon erasure-coding. Afterward, the [NMT][nmt] is built over both -halves and the computed NMT root MUST be equal to the respective Row root in [DAH][dah]. +**RowHalf**: A two-dimensional variable size byte array representing left half of shares in the row. Its length MUST be +equal to the number of Column roots in [DAH][dah] divided by two. These shares MUST only be from the left half of the +row. The right half is computed using Leopard GF16 Reed-Solomon erasure-coding. Afterward, the [NMT][nmt] is built over +both halves and the computed NMT root MUST be equal to the respective Row root in [DAH][dah]. #### SampleID @@ -235,7 +235,7 @@ The fields with validity rules that form DataID are: [**RowID**](#rowid): A RowID of the namespace data. It MUST follow [RowID](#rowid) formatting and field validity rules. -[**Namespace**][ns]: A fixed-size bytes array representing the Namespace of interest. It MUST follow [Namespace][ns] +[**Namespace**][ns]: A fixed-size 29 bytes array representing the Namespace of interest. It MUST follow [Namespace][ns] formatting and its validity rules. Serialized DataID MUST have a length of 39 bytes. @@ -261,14 +261,15 @@ The fields with validity rules that form Data containers are: [**DataID**](#dataid): A DataID of the Data container. It MUST follow [DataID](#dataid) formatting and field validity rules. -**DataShares**: A two-dimensional variable size byte arrays representing left data shares of a namespace in the row. +**DataShares**: A two-dimensional variable size byte array representing left data shares of a namespace in the row. Each share MUST follow [share formatting and validity][shares-format] rules. **Proof**: A [protobuf formated][nmt-pb] [NMT][nmt] proof of share inclusion. It MUST follow [NMT proof verification][nmt-verify] -and be verified against the respective root from the Row or Column axis in [DAH][dah]. The axis is defined by the ProofType field. +and be verified against the respective root from the Row root axis in [DAH][dah]. -Namespace data may span over multiple rows, in which case all the data is encapsulated in multiple containers. This is -done. +Namespace data may span over multiple rows, in which case all the data is encapsulated in multiple containers. This +enables parallelization of namespace data retrieval and certain [compositions](#protocol-compositions) may get advantage +of that by requesting containers of a single namespace from multiple servers simultaneously. ## Protocol Compositions @@ -321,7 +322,7 @@ should be extended whenever any new share identifier is added. ## Backwards Compatibility -Swap is incompatible with the old sampling protocol. +Shwap is incompatible with the old sampling protocol. After rigorous investigation, the celestia-node team decided against _implementing_ backward compatibility with the old protocol into the node client due to the immense complications it brings. Instead, the simple and time-efficient From 1dcd5ee3a6f2a0419a40c5cf43996820d70fd30f Mon Sep 17 00:00:00 2001 From: Wondertan Date: Thu, 7 Mar 2024 17:48:17 +0100 Subject: [PATCH 12/21] clarify serialization --- cips/cip-shwap_protocol.md | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index fcc6df0..3c41833 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -101,6 +101,11 @@ to varints and lack of fixed size arrays. Instead, identifiers use simple binary Identifiers MAY embed each other to narrow down the scope of needed shares. For example, [SampleID](#sampleid) embeds [RowID](#rowid) as every sample lay on a particular row. +##### Serialization + +Share identifiers MUST be serialized by concatenating big-endian representations of fields in the order defined by their +respective formatting section. + #### Share Containers Share containers encapsulate a set of data shares with [DAH][dah] inclusion proof. Share containers are identified by @@ -133,10 +138,10 @@ The fields with validity rules that form RowID are: **Height**: A uint64 representing the chain height with the data square. It MUST be bigger than zero. -**RowIndex**: An uint16 representing row index points to a particular row. It MUST not exceed the number of Row roots in -[DAH][dah]. +**RowIndex**: An uint16 representing row index points to a particular row. The 16 bit limit fits data squares up to 2TB. +It MUST not exceed the number of Row roots in[DAH][dah]. -Serialized RowID MUST have a length of 10 bytes. +[Serialized](#serialization) RowID MUST have a length of 10 bytes. #### Row Container @@ -177,10 +182,10 @@ The fields with validity rules that form SampleID are: [**RowID**](#rowid): A RowID of the sample. It MUST follow [RowID](#rowid) formatting and field validity rules. -**ColumnIndex**: A uint16 representing the column index of the sampled share; in other words, the share index in the row. It -MUST stay within the number of Column roots in [DAH][dah]. +**ColumnIndex**: A uint16 representing the column index of the sampled share; in other words, the share index in the row. +The 16 bit limit fits data squares up to 2TB. It MUST stay within the number of Column roots in [DAH][dah]. -Serialized SampleID MUST have a length of 12 bytes. +[Serialized](#serialization) SampleID MUST have a length of 12 bytes. #### Sample Container @@ -238,7 +243,7 @@ The fields with validity rules that form DataID are: [**Namespace**][ns]: A fixed-size 29 bytes array representing the Namespace of interest. It MUST follow [Namespace][ns] formatting and its validity rules. -Serialized DataID MUST have a length of 39 bytes. +[Serialized](#serialization) DataID MUST have a length of 39 bytes. #### Data Container @@ -343,8 +348,9 @@ However, new bugs may be introduced, as with any new protocol. ### Protobuf Serialization -Protobuf is a widely adopted serialization format and is used within Celestia's protocols. This was quite an obvious choice -for consistency reasons, even though we could choose other more efficient and advanced formats like Cap'n Proto. +Protobuf is used t serialize [share containers](#share-containers). It is a widely adopted serialization format and is +used within Celestia's protocols. This was quite an obvious choice for consistency reasons, even though we could choose +other more efficient and advanced formats like Cap'n Proto. ### Sampling and Reconstruction From 80c53f42acf93a308e7dcab2422f501264740f52 Mon Sep 17 00:00:00 2001 From: Wondertan Date: Mon, 11 Mar 2024 19:31:25 +0100 Subject: [PATCH 13/21] more serialization improvements; describe issues with multihashes, more rules to the spec --- cips/cip-shwap_protocol.md | 64 +++++++++++++++++++++++++------------- 1 file changed, 43 insertions(+), 21 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 3c41833..1a577be 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -91,25 +91,35 @@ _**Proof**_: A Merkle inclusion proof of the data in the DataSquare. ### Message Framework This section defines Shwap's messaging framework. Every group of shares that needs to be exchanged over the network -MUST define its [share identifier](#share-identifiers) and [share container](#share-containers) and follow their described rules. +MUST define its [share identifier](#share-identifiers) and [share container](#share-containers) and follow their +described rules. Every identifier and container MUST define its serialization format, which MAY NOT be consistent with +other identifiers and containers. + + +#### Share Containers + +Share containers encapsulate a set of data shares with proof. Share containers are identified by [share identifiers](#share-identifiers). + +Containers SHOULD contain shares within a single DataSquare and MAY NOT be adjacent. Containers MUST have a [DAH][dah] +inclusion proof field defined. + +##### Serialization + +Share containers are RECOMMENDED to use protobuf (proto3) encoding, and other formats MAY be used for serialization. A +container MAY define multiple serialization formats. #### Share Identifiers -Identifiers MUST have a fixed size for their fields. Subsequently, protobuf SHOULD NOT be used for CID serialization due -to varints and lack of fixed size arrays. Instead, identifiers use simple binary big endian serialization. +Share identifiers identify [share containers](#share-containers). Identifiers are not collision-resistant and there MAY +be different identifiers referencing the same container. Identifiers MAY embed each other to narrow down the scope of needed shares. For example, [SampleID](#sampleid) embeds [RowID](#rowid) as every sample lay on a particular row. ##### Serialization -Share identifiers MUST be serialized by concatenating big-endian representations of fields in the order defined by their -respective formatting section. - -#### Share Containers - -Share containers encapsulate a set of data shares with [DAH][dah] inclusion proof. Share containers are identified by -[share identifiers](#share-identifiers). +Share identifiers SHOULD be serialized by concatenating big-endian representations of fields in the order defined by +their respective formatting section. Serialized identifiers SHOULD have constant and predetermined lengths in bytes. #### Versioning @@ -298,24 +308,27 @@ libp2p provides together with transport protocol advancements introduced in QUIC #### Multihashes and CID -Bitswap is tightly coupled with Multihash and CID notions, establishing the [content addressability property][content-address]. +Bitswap is tightly coupled with [Multihash][mh] and [CID][cid] notions, establishing the [content addressability property][content-address]. Bitswap operates over Blocks of data that are addressed and verified by CIDs. Based on that, Shwap integrates into Bitswap by complying with both of these interfaces. The [Share Containers](#share-containers) are Blocks that are identified via [Share Identifiers](#share-identifiers). Even though Shwap takes inspiration from content addressability, it breaks free from the hash-based model to optimize message sizes and data request patterns. In some way, it hacks into multihash abstraction to make it contain data that -is not, in fact, a hash. Furthermore, the protocol does not include hash digests in the multihashes. The authentication of -the messages happens using externally provided data commitment. +is not, in fact, a hash. Furthermore, the protocol does not include hash digests in the multihashes. The authentication +of the messages happens using externally provided data commitment. -This creates a bunch of complexities with the [reference Golang implementation][gimpl] that are necessary if forking -and substantially diverging the upstream is not an option. The naive question would be: "Why not make content -verification after Bitswap provided it back over its API?" Intuitively, this would simplify much and would not require -"hacking" CID. However, this has an important downside - the Bitswap, in such a case, would consider the request finalized -and the content as fetched and valid, sending a DONT_WANT message to its peers. In contrast, the message might still be invalid -according to the verification rules. +However, breaking-free from hashes creates issues necessary to be solved on the implementation level, particularly in +[the reference Golang implementation][gimpl], if forking and substantially diverging from the upstream is not an option. +CIDs are required to have fixed and deterministic sizes. Making share identifiers compliant with CID +prevents protobuf usage due to its reliance on varints and dynamic byte arrays serialization in. -However, Bitswap still requires multihashes and CID codecs to be registered. Therefore, we provide a table for the +The naive question would be: "Why not make content verification after Bitswap provided it back over its API?" Intuitively, +this would simplify much and would not require "hacking" CID. However, this has an important downside - the Bitswap, in +such a case, would consider the request finalized and the content as fetched and valid, sending a DONT_WANT message to +its peers. In contrast, the message might still be invalid according to the verification rules. + +Bitswap still requires multihashes and CID codecs to be registered. Therefore, we provide a table for the supported [share identifiers](#share-identifiers) with their respective multihash and CID codec codes. This table should be extended whenever any new share identifier is added. @@ -348,10 +361,17 @@ However, new bugs may be introduced, as with any new protocol. ### Protobuf Serialization -Protobuf is used t serialize [share containers](#share-containers). It is a widely adopted serialization format and is +Protobuf is recommended used to serialize [share containers](#share-containers). It is a widely adopted serialization format and is used within Celestia's protocols. This was quite an obvious choice for consistency reasons, even though we could choose other more efficient and advanced formats like Cap'n Proto. +### Constant-size Identifier Serialization + +Share identifiers should be of a constant size according to the spec. This is needed to support [Bitswap composition](#bitswap), +which has an implementation level limitation that enforces constant size identifiers. Ideally, this should be avoided as +Shwap aims to be protocol agnostic, and future iterations of Shwap may introduce dynamically sized identifiers if constant +ever becomes problematic. + ### Sampling and Reconstruction Shwap deliberately avoids specifying sampling and reconstruction logic. The sampling concerns on randomness selection and @@ -370,6 +390,8 @@ Copyright and related rights waived via [CC0](../LICENSE). [shrex]: https://github.com/celestiaorg/celestia-node/blob/0abd16bbb05bf3016595498844a588ef55c63d2d/docs/adr/adr-013-blocksync-overhaul-part-2.md [storage]: https://github.com/celestiaorg/celestia-node/blob/a33c80e20da684d656c7213580be7878bcd27cf4/docs/adr/adr-011-blocksync-overhaul-part-1.md [bitswap]: https://docs.ipfs.tech/concepts/bitswap/ +[cid]: https://docs.ipfs.tech/concepts/content-addressing/ +[mh]: https://multiformats.io/multihash/ [content-address]: https://fission.codes/blog/content-addressing-what-it-is-and-how-it-works/ [kaddht]: https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf [square]: https://celestiaorg.github.io/celestia-app/specs/data_structures.html#2d-reed-solomon-encoding-scheme From 9cd5a3933a0ed9e1f6094c384cb0e558e6095853 Mon Sep 17 00:00:00 2001 From: Hlib Kanunnikov Date: Tue, 12 Mar 2024 12:43:07 +0100 Subject: [PATCH 14/21] Update cips/cip-shwap_protocol.md Co-authored-by: Josh Stein <46639943+jcstein@users.noreply.github.com> --- cips/cip-shwap_protocol.md | 1 + 1 file changed, 1 insertion(+) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 1a577be..62af5ab 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -1,4 +1,5 @@ --- +cip: 18 title: Shwap Protocol description: Shwap - a new messaging framework for DA and sampling author: Hlib Kanunnikov (@Wondertan) From fabe64c0673dcfd7280df468907d8390406e9f52 Mon Sep 17 00:00:00 2001 From: Hlib Kanunnikov Date: Tue, 12 Mar 2024 18:40:17 +0100 Subject: [PATCH 15/21] Apply suggestions from code review Co-authored-by: Rootul P --- cips/cip-shwap_protocol.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-shwap_protocol.md index 62af5ab..fd48103 100644 --- a/cips/cip-shwap_protocol.md +++ b/cips/cip-shwap_protocol.md @@ -12,7 +12,7 @@ created: 2024-02-02 ## Abstract -This document specifies Shwap - the simple and expressive yet extensible messaging framework aiming to +This document specifies Shwap (a portmanteau of share and swap) - the simple, expressive, and extensible messaging framework aiming to solve critical inefficiencies and standardize messaging of Celestia's Data Availability p2p network. Shwap defines a messaging framework to be exchanged around the DA p2p network in a trust-minimized way without enforcing @@ -24,7 +24,7 @@ protocol. ## Motivation The current Data Availability Sampling (DAS) network protocol is inefficient. A _single_ sample operation takes log₂(k) -network roundtrips (where k is the square size). This is not practical and does not scale for the theoretically unlimited +network roundtrips (where k is the extended square size). This is not practical and does not scale for the theoretically unlimited data square that the Celestia network enables. The main motive here is a protocol with O(1) roundtrip for _multiple_ samples, preserving the assumption of having 1/N honest peers connected possessing the data. @@ -42,7 +42,7 @@ for efficient communication for BS, ND, and beyond. The atomic primitive of Celestia's DA network is the share. Shwap standardizes messaging and serialization for shares. Shares are grouped together, forming more complex data types (Rows, Blobs, etc.). These data types are encapsulated in -containers, For example, row container groups shares of a particular row. Containers can be identified with the share +containers. For example, a row container groups the shares of a particular row. Containers can be identified with the share identifiers in order to request, advertise or index the containers. The combination of containers and identifiers provides an extensible and expressive messaging framework for groups of shares and enables efficient single roundtrip request-response communication. @@ -130,7 +130,7 @@ suffixed with a new major version starting from v1. E.g., if the Row message nee ### Messages This section defines all the supported Shwap messages, including share identifiers and containers. All the new -future messages should be described in the section. +future messages should be described in this section. #### RowID @@ -150,7 +150,7 @@ The fields with validity rules that form RowID are: **Height**: A uint64 representing the chain height with the data square. It MUST be bigger than zero. **RowIndex**: An uint16 representing row index points to a particular row. The 16 bit limit fits data squares up to 2TB. -It MUST not exceed the number of Row roots in[DAH][dah]. +It MUST not exceed the number of Row roots in [DAH][dah]. [Serialized](#serialization) RowID MUST have a length of 10 bytes. From 7eac32ec167e5f045a9fea846dcc3e3e380a0b1c Mon Sep 17 00:00:00 2001 From: Josh Stein <46639943+jcstein@users.noreply.github.com> Date: Thu, 14 Mar 2024 14:43:21 -0400 Subject: [PATCH 16/21] Rename cip-shwap_protocol.md to cip-18.md --- cips/{cip-shwap_protocol.md => cip-18.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename cips/{cip-shwap_protocol.md => cip-18.md} (100%) diff --git a/cips/cip-shwap_protocol.md b/cips/cip-18.md similarity index 100% rename from cips/cip-shwap_protocol.md rename to cips/cip-18.md From 0324e1bfb4675fef49ee215cf1487e2c37811adf Mon Sep 17 00:00:00 2001 From: Wondertan Date: Thu, 14 Mar 2024 19:51:19 +0100 Subject: [PATCH 17/21] clarify versioning for new cid codecs and multihashes --- cips/cip-18.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cips/cip-18.md b/cips/cip-18.md index fd48103..f0b54f6 100644 --- a/cips/cip-18.md +++ b/cips/cip-18.md @@ -331,7 +331,7 @@ its peers. In contrast, the message might still be invalid according to the veri Bitswap still requires multihashes and CID codecs to be registered. Therefore, we provide a table for the supported [share identifiers](#share-identifiers) with their respective multihash and CID codec codes. This table -should be extended whenever any new share identifier is added. +should be extended whenever any new share identifier or new version of an existing identifier is added. | Name | Multihash | Codec | |----------|-----------|--------| From ef245a7fe5f672fb7955bfc7aaa9a7c53ed5fdd8 Mon Sep 17 00:00:00 2001 From: jcstein Date: Tue, 19 Mar 2024 11:30:19 -0400 Subject: [PATCH 18/21] docs: name to cip-19 --- cips/{cip-18.md => cip-19.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename cips/{cip-18.md => cip-19.md} (99%) diff --git a/cips/cip-18.md b/cips/cip-19.md similarity index 99% rename from cips/cip-18.md rename to cips/cip-19.md index f0b54f6..161abaa 100644 --- a/cips/cip-18.md +++ b/cips/cip-19.md @@ -1,5 +1,5 @@ --- -cip: 18 +cip: 19 title: Shwap Protocol description: Shwap - a new messaging framework for DA and sampling author: Hlib Kanunnikov (@Wondertan) From ca771db7ea6991a002a8d75a82bedad072155092 Mon Sep 17 00:00:00 2001 From: jcstein Date: Wed, 20 Mar 2024 10:55:46 -0400 Subject: [PATCH 19/21] docs: add to readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index a827818..c42b2cc 100644 --- a/README.md +++ b/README.md @@ -34,3 +34,4 @@ Read [CIP-1](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-1.md) for in | [14](/cips/cip-14.md) | ICS-27 Interchain Accounts | Susannah Evans (@womensrights), Aidan Salzmann (@asalzmann), Sam Pochyly (@sampocs) | | [15](/cips/cip-15.md) | Discourage memo usage | Rootul Patel (@rootulp), NashQueue (@nashqueue) | | [16](/cips/cip-16.md) | Make Security Related Governance Parameters Immutable | Mingpei CAO (@caomingpei) | +| [19](/cips/cip-19.md) | Shwap Protocol | Hlib Kanunnikov (@Wondertan) | From 7dfca947ca99f2b0d1d6174c68480213f31600c5 Mon Sep 17 00:00:00 2001 From: jcstein Date: Wed, 20 Mar 2024 11:15:28 -0400 Subject: [PATCH 20/21] docs: linting --- cips/cip-19.md | 49 ++++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/cips/cip-19.md b/cips/cip-19.md index 161abaa..2d2652a 100644 --- a/cips/cip-19.md +++ b/cips/cip-19.md @@ -31,8 +31,8 @@ samples, preserving the assumption of having 1/N honest peers connected possessi Initially, Bitswap and IPLD were adopted as the basis for the DA network protocols, including DAS, block synchronization (BS), and blob/namespace data retrieval (ND). They gave battle-tested protocols and tooling with pluggability to rapidly scaffold Celestia's DA network. However, it came with the price of scalability limits and -roundtrips, resulting in slower BS than block production. Before the network launch, we transitioned to the optimized -[ShrEx protocol][shrex] for BS and integrated [CAR and DAGStore-based storage][storage] optimizing BS and ND. However, +roundtrips, resulting in slower BS than block production. Before the network launch, we transitioned to the optimized +[ShrEx protocol][shrex] for BS and integrated [CAR and DAGStore-based storage][storage] optimizing BS and ND. However, DAS was left untouched, preserving its weak scalability and roundtrip inefficiency. Shwap messaging stacked together with Bitswap protocol directly addresses described inefficiency and provides a foundation @@ -42,14 +42,14 @@ for efficient communication for BS, ND, and beyond. The atomic primitive of Celestia's DA network is the share. Shwap standardizes messaging and serialization for shares. Shares are grouped together, forming more complex data types (Rows, Blobs, etc.). These data types are encapsulated in -containers. For example, a row container groups the shares of a particular row. Containers can be identified with the share -identifiers in order to request, advertise or index the containers. The combination of containers and identifiers -provides an extensible and expressive messaging framework for groups of shares and enables efficient single roundtrip +containers. For example, a row container groups the shares of a particular row. Containers can be identified with the share +identifiers in order to request, advertise or index the containers. The combination of containers and identifiers +provides an extensible and expressive messaging framework for groups of shares and enables efficient single roundtrip request-response communication. Many share groups or containers are known in the Celestia network, and systemizing this is the main reason behind setting up this simple messaging framework. A single place with all the possible Celestia DA messages must be defined, which node -software and protocol researchers can rely on and coordinate. Besides, this framework is designed to be sustain changes +software and protocol researchers can rely on and coordinate. Besides, this framework is designed to be sustain changes in the core protocol's data structures and proving system as long shares stay the de facto atomic data type. Besides, there needs to be systematization and a joint knowledge base with all the edge cases for possible protocol @@ -96,7 +96,6 @@ MUST define its [share identifier](#share-identifiers) and [share container](#sh described rules. Every identifier and container MUST define its serialization format, which MAY NOT be consistent with other identifiers and containers. - #### Share Containers Share containers encapsulate a set of data shares with proof. Share containers are identified by [share identifiers](#share-identifiers). @@ -104,22 +103,22 @@ Share containers encapsulate a set of data shares with proof. Share containers a Containers SHOULD contain shares within a single DataSquare and MAY NOT be adjacent. Containers MUST have a [DAH][dah] inclusion proof field defined. -##### Serialization +##### Serialization for Share Containers -Share containers are RECOMMENDED to use protobuf (proto3) encoding, and other formats MAY be used for serialization. A +Share containers are RECOMMENDED to use protobuf (proto3) encoding, and other formats MAY be used for serialization. A container MAY define multiple serialization formats. #### Share Identifiers -Share identifiers identify [share containers](#share-containers). Identifiers are not collision-resistant and there MAY +Share identifiers identify [share containers](#share-containers). Identifiers are not collision-resistant and there MAY be different identifiers referencing the same container. Identifiers MAY embed each other to narrow down the scope of needed shares. For example, [SampleID](#sampleid) embeds [RowID](#rowid) as every sample lay on a particular row. -##### Serialization +##### Serialization for Share Identifiers -Share identifiers SHOULD be serialized by concatenating big-endian representations of fields in the order defined by +Share identifiers SHOULD be serialized by concatenating big-endian representations of fields in the order defined by their respective formatting section. Serialized identifiers SHOULD have constant and predetermined lengths in bytes. #### Versioning @@ -171,9 +170,9 @@ The fields with validity rules that form Row containers are: [**RowID**](#rowid): A RowID of the Row Container. It MUST follow [RowID](#rowid) formatting and field validity rules. -**RowHalf**: A two-dimensional variable size byte array representing left half of shares in the row. Its length MUST be -equal to the number of Column roots in [DAH][dah] divided by two. These shares MUST only be from the left half of the -row. The right half is computed using Leopard GF16 Reed-Solomon erasure-coding. Afterward, the [NMT][nmt] is built over +**RowHalf**: A two-dimensional variable size byte array representing left half of shares in the row. Its length MUST be +equal to the number of Column roots in [DAH][dah] divided by two. These shares MUST only be from the left half of the +row. The right half is computed using Leopard GF16 Reed-Solomon erasure-coding. Afterward, the [NMT][nmt] is built over both halves and the computed NMT root MUST be equal to the respective Row root in [DAH][dah]. #### SampleID @@ -285,7 +284,7 @@ and be verified against the respective root from the Row root axis in [DAH][dah] Namespace data may span over multiple rows, in which case all the data is encapsulated in multiple containers. This enables parallelization of namespace data retrieval and certain [compositions](#protocol-compositions) may get advantage -of that by requesting containers of a single namespace from multiple servers simultaneously. +of that by requesting containers of a single namespace from multiple servers simultaneously. ## Protocol Compositions @@ -319,14 +318,14 @@ message sizes and data request patterns. In some way, it hacks into multihash ab is not, in fact, a hash. Furthermore, the protocol does not include hash digests in the multihashes. The authentication of the messages happens using externally provided data commitment. -However, breaking-free from hashes creates issues necessary to be solved on the implementation level, particularly in -[the reference Golang implementation][gimpl], if forking and substantially diverging from the upstream is not an option. -CIDs are required to have fixed and deterministic sizes. Making share identifiers compliant with CID +However, breaking-free from hashes creates issues necessary to be solved on the implementation level, particularly in +[the reference Golang implementation][gimpl], if forking and substantially diverging from the upstream is not an option. +CIDs are required to have fixed and deterministic sizes. Making share identifiers compliant with CID prevents protobuf usage due to its reliance on varints and dynamic byte arrays serialization in. The naive question would be: "Why not make content verification after Bitswap provided it back over its API?" Intuitively, -this would simplify much and would not require "hacking" CID. However, this has an important downside - the Bitswap, in -such a case, would consider the request finalized and the content as fetched and valid, sending a DONT_WANT message to +this would simplify much and would not require "hacking" CID. However, this has an important downside - the Bitswap, in +such a case, would consider the request finalized and the content as fetched and valid, sending a DONT_WANT message to its peers. In contrast, the message might still be invalid according to the verification rules. Bitswap still requires multihashes and CID codecs to be registered. Therefore, we provide a table for the @@ -362,8 +361,8 @@ However, new bugs may be introduced, as with any new protocol. ### Protobuf Serialization -Protobuf is recommended used to serialize [share containers](#share-containers). It is a widely adopted serialization format and is -used within Celestia's protocols. This was quite an obvious choice for consistency reasons, even though we could choose +Protobuf is recommended used to serialize [share containers](#share-containers). It is a widely adopted serialization format and is +used within Celestia's protocols. This was quite an obvious choice for consistency reasons, even though we could choose other more efficient and advanced formats like Cap'n Proto. ### Constant-size Identifier Serialization @@ -375,9 +374,9 @@ ever becomes problematic. ### Sampling and Reconstruction -Shwap deliberately avoids specifying sampling and reconstruction logic. The sampling concerns on randomness selection and +Shwap deliberately avoids specifying sampling and reconstruction logic. The sampling concerns on randomness selection and sample picking are out of Shwap's scope and a matter of following CIPs. Shwap only provides messaging for sampling(via -[SampleID](#sampleid) and [Sample container](#sample-container)). +[SampleID](#sampleid) and [Sample container](#sample-container)). ## Reference Implementation From feba38bc987564fcd0468a65a5705f93915917e6 Mon Sep 17 00:00:00 2001 From: jcstein Date: Wed, 20 Mar 2024 11:29:38 -0400 Subject: [PATCH 21/21] docs: link headings properly --- cips/cip-19.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cips/cip-19.md b/cips/cip-19.md index 2d2652a..53c95a5 100644 --- a/cips/cip-19.md +++ b/cips/cip-19.md @@ -151,7 +151,7 @@ The fields with validity rules that form RowID are: **RowIndex**: An uint16 representing row index points to a particular row. The 16 bit limit fits data squares up to 2TB. It MUST not exceed the number of Row roots in [DAH][dah]. -[Serialized](#serialization) RowID MUST have a length of 10 bytes. +[Serialized](#serialization-for-share-identifiers) RowID MUST have a length of 10 bytes. #### Row Container @@ -195,7 +195,7 @@ The fields with validity rules that form SampleID are: **ColumnIndex**: A uint16 representing the column index of the sampled share; in other words, the share index in the row. The 16 bit limit fits data squares up to 2TB. It MUST stay within the number of Column roots in [DAH][dah]. -[Serialized](#serialization) SampleID MUST have a length of 12 bytes. +[Serialized](#serialization-for-share-identifiers) SampleID MUST have a length of 12 bytes. #### Sample Container @@ -253,7 +253,7 @@ The fields with validity rules that form DataID are: [**Namespace**][ns]: A fixed-size 29 bytes array representing the Namespace of interest. It MUST follow [Namespace][ns] formatting and its validity rules. -[Serialized](#serialization) DataID MUST have a length of 39 bytes. +[Serialized](#serialization-for-share-identifiers) DataID MUST have a length of 39 bytes. #### Data Container