From 60b1a2b418d50d6cedd642f8a35999a6e6820aab Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Wed, 1 Mar 2023 15:59:22 +0100 Subject: [PATCH 01/31] feat: filecoin spec --- w3-filecoin.md | 187 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 w3-filecoin.md diff --git a/w3-filecoin.md b/w3-filecoin.md new file mode 100644 index 0000000..4f23d83 --- /dev/null +++ b/w3-filecoin.md @@ -0,0 +1,187 @@ +# Filecoin Protocol + +![status:wip](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) + +## Editors + +- [Vasco Santos], [Protocol Labs] + +## Authors + +- [Vasco Santos], [Protocol Labs] + +# Abstract + +This spec describes the filecoin protocol used by web3.storage's "w3up" platform to interact with Spade ♠️ in order to get user/application CAR files into Filecoin deals. + +## Language + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://datatracker.ietf.org/doc/html/rfc2119). + +# Terminology + +## Roles + +There are several roles in the authorization flow: + +| Name | Description | +| ----------- | ----------- | +| Storage Provider | [Principal] identified by [`did:web`] identifier, representing a storage provider like w3up | +| Authority | [Principal] that represents service provider that executes invoked capabilities | +| Verifier | Component of the [authority] that performs UCAN validation | + +### Storage Provider + +A _Storage Provider_ is a type of [principal] identified by a [`did:web`] identifier. + +A Storage Providers facilitates data storage services to applications and users, getting the requested data stored into Filecoin deals asynchronously. + +### Authority + +_Authority_ is a [principal] that executes invoked capabilities. + +### Verifier + +Component of the [authority] that performs UCAN validation + +# Protocol + +## Overview + +A Storage Provider is the entry point for user/application data into the web3. It will act on behalf of users to move data around into different storage points. One of the key storage points is Filecoin deals. + +## Authorization + +Storage Provider (web3.storage) must be given capabilities by a Filecoin broker Authority (Spade) to invoke capabilities to get its stored content into Filecoin deals. + +```mermaid +sequenceDiagram + participant Authority as 🌐

did:web:spade.storage + participant Storage Provider as 🌐

did:web:web3.storage #32; + + Authority->>Storage Provider: delegate `filecoin/add` +``` + +## Filecoin deals + +When a storage provider has enough + +```mermaid +sequenceDiagram + participant Authority as 🌐

did:web:spade.storage + participant Storage Provider as 🌐

did:web:web3.storage #32; + + Storage Provider->>Authority: invoke `filecoin/add` + Note left of Authority: Prepare for deal + Authority-->>Storage Provider: receipt + Note left of Authority: Land deal in Filecoin + Authority->>Storage Provider: invoke `filecoin/deal` + Authority-->>Storage Provider: invoke `filecoin/fail` +``` + +## Capabilities + +This section describes the capabilities that form the w3 Filecoin protocol, along with details relevant to invoking capabilities with a service provider. + +### `filecoin/add` + +A Filecoin broker Authority MAY delegate capabilities to any storage provider. + +> `did:web:spade.storage` delegates `filecoin/add` capabilities to `did:web:web3.storage` + +```json +{ + "iss": "did:web:spade.storage", + "aud": "did:web:web3.storage", + "att": [{ "with": "did:web:spade.storage", "can": "filecoin/add" }], + "exp": null, + "sig": "..." +} +``` + +> `did:web:web3.storage` invokes capability from `did:web:spade.storage` + +```json +{ + "iss": "did:web:web3.storage", + "aud": "did:web:spade.storage", + "att": [{ + "with": "did:web:spade.storage", + "can": "filecoin/add", + "nb": : { + "aggregateId": "id...", + "cars": [ + { + "link": "bag...", + "size": 1234, + "md5": "md5...", + "commP": "commP...", + "url": "https://..." + } + ] + } + }], + "prof": [ + "iss": "did:web:spade.storage", + "aud": "did:web:web3.storage", + "att": [{ "with": "did:web:spade.storage", "can": "filecoin/add" }], + "exp": null, + "sig": "..." + ], + "sig": "..." +} +``` + +### `filecoin/deal` + +A Filecoin broker authority MAY invoke deal capability when one aggregate lands into a Filecoin deal. It should include the CID of the request CAR cid from `filecoin/add` invocation. + +```json +{ + "iss": "did:web:spade.storage", + "aud": "did:web:spade.storage", + "att": [{ + "with": "did:web:spade.storage", + "can": "filecoin/deal", + "nb": : { + "aggregateId": "id...", + "requestCarCid": "bag...", + "commP": "commP..." + } + }], + "prof": [], + "sig": "..." +} +``` + +### `filecoin/fail` + +A Filecoin broker authority MAY invoke deal capability when one aggregate fails to land into a Filecoin deal. It should include the CID of the request CAR cid from `filecoin/add` invocation. + +```json +{ + "iss": "did:web:spade.storage", + "aud": "did:web:spade.storage", + "att": [{ + "with": "did:web:spade.storage", + "can": "filecoin/deal", + "nb": : { + "aggregateId": "id...", + "requestCarCid": "bag...", + "reason": "tbd" + } + }], + "prof": [], + "sig": "..." +} +``` + +--- + +[`did:web`]: https://w3c-ccg.github.io/did-method-web/ +[ucan]: https://github.com/ucan-wg/spec/ +[principal]: https://github.com/ucan-wg/spec/#321-principals +[authority]:#authority + +[Protocol Labs]:https://protocol.ai/ +[Vasco Santos]:https://github.com/vasco-santos From 898f2ebe6026a6bf49d94b7ee69dbe3c9c88a1b3 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Mon, 27 Mar 2023 13:54:39 +0200 Subject: [PATCH 02/31] fix: new iteration with receipts and aggregate --- .github/workflows/words-to-ignore.txt | 7 + w3-aggregation.md | 293 ++++++++++++++++++++++++++ w3-filecoin.md | 187 ---------------- 3 files changed, 300 insertions(+), 187 deletions(-) create mode 100644 w3-aggregation.md delete mode 100644 w3-filecoin.md diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index 7e91338..86d67c3 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -94,3 +94,10 @@ flowAuthorityPrincipal capabilitiesVerifierComponent Verifier sudo +Filecoin +Storefront +capabilty +commPs +dag-cbor +NameDescriptionStorefrontPrincipal +w3upAuthorityPrincipal \ No newline at end of file diff --git a/w3-aggregation.md b/w3-aggregation.md new file mode 100644 index 0000000..cc31218 --- /dev/null +++ b/w3-aggregation.md @@ -0,0 +1,293 @@ +# CAR Aggregation Protocol + +![status:wip](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) + +## Editors + +- [Vasco Santos], [Protocol Labs] + +## Authors + +- [Vasco Santos], [Protocol Labs] + +# Abstract + +This spec describes a [UCAN] protocol allowing an implementer to receive an aggregate of CAR files for inclusion in a Filecoin deal. + +## Language + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://datatracker.ietf.org/doc/html/rfc2119). + +# Terminology + +## Roles + +There are several roles in the authorization flow: + +| Name | Description | +| ----------- | ----------- | +| Storefront | [Principal] identified by [`did:web`] identifier, representing a storage aggregator like w3up | +| Authority | [Principal] that represents service provider that executes invoked capabilities | +| Verifier | Component of the [authority] that performs UCAN validation | + +### Storefront + +A _Storefront_ is a type of [principal] identified by a [`did:web`] identifier. + +A Storefront facilitates data storage services to applications and users, getting the requested data stored into Filecoin deals asynchronously. + +### Authority + +_Authority_ is a [principal] that executes invoked capabilities. + +### Verifier + +A Component of the [authority] that performs UCAN validation + +# Protocol + +## Overview + +A Storefront is the entry point for user/application data into the web3. It will act on behalf of users to move data around into different storage points. One of the key storage presences may be Filecoin Storage Providers. + +## Authorization + +A Filecoin broker authority (e.g. Spade) MUST give capabilities to Storefront principals (e.g. web3.storage) to invoke capabilities to get stored content into Filecoin deals. + +```mermaid +sequenceDiagram + participant Authority as 🌐

did:web:spade.storage + participant Storefront as 🌐

did:web:web3.storage + + Authority->>Storefront: delegate `aggregate/*` +``` + +Note that this can be done out of band with presentation of a delegation signature. + +## Deal creation + +When a Storefront has enough content to fulfill an aggregate (each broker might have different requirements), a Filecoin deal for an aggregate SHALL be requested by a `aggregate/submit` invocations. Deal negotiations with Filecoin Storage Providers will be handled out off band. Therefore, Broker should generate a receipt to acknowledge received request. This receipt contains an [effect](https://github.com/ucan-wg/invocation/#7-effect) to be performed (`.fx` field pointing to another task) that will eventually get a receipt with either `ok` or `error`. + +```mermaid +sequenceDiagram + participant Storefront as 🌐

did:web:web3.storage + participant Authority as 🌐

did:web:spade.storage + + Storefront->>Authority: invoke `aggregate/submit` + Note left of Authority: Prepare for deal + Authority-->>Storefront: receipt + Note left of Authority: Land deal in Filecoin +``` + +## Deal information + +Storefront users MAY want to check details about deals from the content they previously stored. These deals will change over time as they get renewed. Therefore, Storefront should invoke `aggregate/deals` capability to gather information about given aggregate identifier. + +```mermaid +sequenceDiagram +participant Storefront as 🌐

did:web:web3.storage + participant Authority as 🌐

did:web:spade.storage + + Storefront->>Authority: invoke `aggregate/deals` +``` + +## Capabilities + +This section describes the capabilities that form the w3 aggregation protocol, along with the details relevant for invoking capabilities with a service provider. + +In this document, we will be looking at `spade.storage` as an implementer of the `aggregate` and `deals/*` protocol. + +### `aggregate/*` + +A Filecoin broker Authority MAY delegate capabilities to any Storefront principal. + +> `did:web:spade.storage` delegates `aggregate/*` capabilities to `did:web:web3.storage` + +```json +{ + "iss": "did:web:spade.storage", + "aud": "did:web:web3.storage", + "att": [{"with": "did:web:spade.storage", "can": "aggregate/*" }], + "exp": null, + "sig": "..." +} +``` + +### `aggregate/submit` + +A Storefront principal can invoke a capabilty to submit an aggregate ready for deals. + +> `did:web:web3.storage` invokes capability from `did:web:spade.storage` + +```json +{ + "iss": "did:web:web3.storage", + "aud": "did:web:spade.storage", + "att": [{ + "with": "did:web:spade.storage", + "can": "aggregate/submit", + "nb": { + "link": "bagy...aggregate", + } + }], + "prf": [ + "iss": "did:web:spade.storage", + "aud": "did:web:web3.storage", + "att": [{ "with": "did:web:spade.storage", "can": "aggregate/*" }], + "exp": null, + "sig": "..." + ], + "sig": "..." +} +``` + +This capability is invoked to submit a request to a broker service when an aggregate is ready for a Filecoin deal. The `nb.link` is the CAR CID that refers to a "Ferry" aggregate, a collection of dag-cbor blocks with format: + +```json +{ + "link": "bag...", + "size": 110101, + "commP": "commP...", + "url": "https://.../bag(...).car" +} +``` + +A receipt will be generated to acknowledge the received request. This receipt contains an [effect](https://github.com/ucan-wg/invocation/#7-effect) to be performed (`.fx` field pointing to another task) that will eventually get a receipt with either `ok` or `error`. + +```json +{ + "ran": "bafy...invocation", + "out": { + "ok": { + "status": "queued" + } + }, + "fx": { + "fork": [ + "bafy...dealinvocation1", + "bafy...dealinvocation2", + "bafy...dealinvocation3" + ] + }, + "meta": {}, + "iss": "did:web:spade.storage", + "prf": [] +} +``` + +Open questions: + +- can we get a `commP` of the aggregate with `commP` of every CAR that is part of it? +- perhaps the `nb` should also include the `commP` of the aggregate? + +### `aggregate/deals` + +A Storefront principal can invoke a capabilty to submit an aggregate ready for deals. + +> `did:web:web3.storage` invokes capability from `did:web:spade.storage` + +```json +{ + "iss": "did:web:web3.storage", + "aud": "did:web:spade.storage", + "att": [{ + "with": "did:web:spade.storage", + "can": "aggregate/deals", + "nb": { + "link": "bagy...aggregate", + } + }], + "prf": [ + "iss": "did:web:spade.storage", + "aud": "did:web:web3.storage", + "att": [{ "with": "did:web:spade.storage", "can": "aggregate/*" }], + "exp": null, + "sig": "..." + ], + "sig": "..." +} +``` + +Open questions: + +- should the `nb` include only the `commP` of the aggregate as only field? + +### `deal/offer` + +When a broker receives an `aggregate/submit` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with multiple fork tasks to be performed asynchronously. + +```json +{ + "iss": "did:web:spade.storage", + "aud": "did:web:web3.storage", + "att": [{ + "with": "did:web:spade.storage", + "can": "deal/offer", + "nb": { + "commP": "commP", + } + }], + "prf": [], + "sig": "..." +} +``` + +Once this invocation is executed, a receipt is generated with the result of the operation. + +A successful deal receipt will include the details of the deal: + +```json +{ + "ran": "bafy...invocation", + "out": { + "ok": { + "dealId": 142334312, + "storageProvider": "f07...", + "status": "Active", + "pieceCid": "bag...", + "dataCid": "bafy...", + "dataModelSelector": "Links/...", + "activation": "2023-04-13T01:58:00+00:00", + "expiration": "2024-09-05T01:58:00+00:00", + "created": "2023-04-11T17:57:30.522198+00:00", + "updated": "2024-04-12T03:42:26.928993+00:00" + } + }, + "fx": { + "fork": [] + }, + "meta": {}, + "iss": "did:web:spade.storage", + "prf": [] +} +``` + +If a deal fails, the failing commPs will be reported: + +```json +{ + "ran": "bafy...invocation", + "out": { + "error": { + "parts": [{ + "commP": "commP", + "reason": "reasonCode", + }], + }, + }, + "fx": { + "fork": [] + }, + "meta": {}, + "iss": "did:web:spade.storage", + "prf": [] +} +``` + +[`did:web`]: https://w3c-ccg.github.io/did-method-web/ +[UCAN]: https://github.com/ucan-wg/spec/ +[principal]: https://github.com/ucan-wg/spec/#321-principals +[authority]:#authority + +[Protocol Labs]:https://protocol.ai/ +[Vasco Santos]:https://github.com/vasco-santos diff --git a/w3-filecoin.md b/w3-filecoin.md deleted file mode 100644 index 4f23d83..0000000 --- a/w3-filecoin.md +++ /dev/null @@ -1,187 +0,0 @@ -# Filecoin Protocol - -![status:wip](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) - -## Editors - -- [Vasco Santos], [Protocol Labs] - -## Authors - -- [Vasco Santos], [Protocol Labs] - -# Abstract - -This spec describes the filecoin protocol used by web3.storage's "w3up" platform to interact with Spade ♠️ in order to get user/application CAR files into Filecoin deals. - -## Language - -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://datatracker.ietf.org/doc/html/rfc2119). - -# Terminology - -## Roles - -There are several roles in the authorization flow: - -| Name | Description | -| ----------- | ----------- | -| Storage Provider | [Principal] identified by [`did:web`] identifier, representing a storage provider like w3up | -| Authority | [Principal] that represents service provider that executes invoked capabilities | -| Verifier | Component of the [authority] that performs UCAN validation | - -### Storage Provider - -A _Storage Provider_ is a type of [principal] identified by a [`did:web`] identifier. - -A Storage Providers facilitates data storage services to applications and users, getting the requested data stored into Filecoin deals asynchronously. - -### Authority - -_Authority_ is a [principal] that executes invoked capabilities. - -### Verifier - -Component of the [authority] that performs UCAN validation - -# Protocol - -## Overview - -A Storage Provider is the entry point for user/application data into the web3. It will act on behalf of users to move data around into different storage points. One of the key storage points is Filecoin deals. - -## Authorization - -Storage Provider (web3.storage) must be given capabilities by a Filecoin broker Authority (Spade) to invoke capabilities to get its stored content into Filecoin deals. - -```mermaid -sequenceDiagram - participant Authority as 🌐

did:web:spade.storage - participant Storage Provider as 🌐

did:web:web3.storage #32; - - Authority->>Storage Provider: delegate `filecoin/add` -``` - -## Filecoin deals - -When a storage provider has enough - -```mermaid -sequenceDiagram - participant Authority as 🌐

did:web:spade.storage - participant Storage Provider as 🌐

did:web:web3.storage #32; - - Storage Provider->>Authority: invoke `filecoin/add` - Note left of Authority: Prepare for deal - Authority-->>Storage Provider: receipt - Note left of Authority: Land deal in Filecoin - Authority->>Storage Provider: invoke `filecoin/deal` - Authority-->>Storage Provider: invoke `filecoin/fail` -``` - -## Capabilities - -This section describes the capabilities that form the w3 Filecoin protocol, along with details relevant to invoking capabilities with a service provider. - -### `filecoin/add` - -A Filecoin broker Authority MAY delegate capabilities to any storage provider. - -> `did:web:spade.storage` delegates `filecoin/add` capabilities to `did:web:web3.storage` - -```json -{ - "iss": "did:web:spade.storage", - "aud": "did:web:web3.storage", - "att": [{ "with": "did:web:spade.storage", "can": "filecoin/add" }], - "exp": null, - "sig": "..." -} -``` - -> `did:web:web3.storage` invokes capability from `did:web:spade.storage` - -```json -{ - "iss": "did:web:web3.storage", - "aud": "did:web:spade.storage", - "att": [{ - "with": "did:web:spade.storage", - "can": "filecoin/add", - "nb": : { - "aggregateId": "id...", - "cars": [ - { - "link": "bag...", - "size": 1234, - "md5": "md5...", - "commP": "commP...", - "url": "https://..." - } - ] - } - }], - "prof": [ - "iss": "did:web:spade.storage", - "aud": "did:web:web3.storage", - "att": [{ "with": "did:web:spade.storage", "can": "filecoin/add" }], - "exp": null, - "sig": "..." - ], - "sig": "..." -} -``` - -### `filecoin/deal` - -A Filecoin broker authority MAY invoke deal capability when one aggregate lands into a Filecoin deal. It should include the CID of the request CAR cid from `filecoin/add` invocation. - -```json -{ - "iss": "did:web:spade.storage", - "aud": "did:web:spade.storage", - "att": [{ - "with": "did:web:spade.storage", - "can": "filecoin/deal", - "nb": : { - "aggregateId": "id...", - "requestCarCid": "bag...", - "commP": "commP..." - } - }], - "prof": [], - "sig": "..." -} -``` - -### `filecoin/fail` - -A Filecoin broker authority MAY invoke deal capability when one aggregate fails to land into a Filecoin deal. It should include the CID of the request CAR cid from `filecoin/add` invocation. - -```json -{ - "iss": "did:web:spade.storage", - "aud": "did:web:spade.storage", - "att": [{ - "with": "did:web:spade.storage", - "can": "filecoin/deal", - "nb": : { - "aggregateId": "id...", - "requestCarCid": "bag...", - "reason": "tbd" - } - }], - "prof": [], - "sig": "..." -} -``` - ---- - -[`did:web`]: https://w3c-ccg.github.io/did-method-web/ -[ucan]: https://github.com/ucan-wg/spec/ -[principal]: https://github.com/ucan-wg/spec/#321-principals -[authority]:#authority - -[Protocol Labs]:https://protocol.ai/ -[Vasco Santos]:https://github.com/vasco-santos From 1c76c3870675116b2be5fb0d8d3c67ce545c1059 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 10:38:25 +0200 Subject: [PATCH 03/31] chore: apply suggestions from code review Co-authored-by: Irakli Gozalishvili --- w3-aggregation.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index cc31218..5a2636e 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -66,7 +66,7 @@ Note that this can be done out of band with presentation of a delegation signatu ## Deal creation -When a Storefront has enough content to fulfill an aggregate (each broker might have different requirements), a Filecoin deal for an aggregate SHALL be requested by a `aggregate/submit` invocations. Deal negotiations with Filecoin Storage Providers will be handled out off band. Therefore, Broker should generate a receipt to acknowledge received request. This receipt contains an [effect](https://github.com/ucan-wg/invocation/#7-effect) to be performed (`.fx` field pointing to another task) that will eventually get a receipt with either `ok` or `error`. +When a Storefront has enough content to fulfill an aggregate (each broker might have different requirements), a Filecoin deal for an aggregate SHALL be requested by a `aggregate/submit` invocations. Deal negotiations with Filecoin Storage Providers will be handled out off band. Therefore, Broker should generate a receipt to acknowledge received request. This receipt MUST contain a followup task in the (`.fx.join` field) that is run when submitted request is processed which MAY succeed (if aggregate was accepted) or fail (e.g. if aggregated was determined to be invalid). Result of the subsequent task CAN be looked up using it's receipt. ```mermaid sequenceDiagram @@ -107,7 +107,7 @@ A Filecoin broker Authority MAY delegate capabilities to any Storefront principa { "iss": "did:web:spade.storage", "aud": "did:web:web3.storage", - "att": [{"with": "did:web:spade.storage", "can": "aggregate/*" }], + "att": [{"with": "did:web:web3.storage", "can": "aggregate/*" }], "exp": null, "sig": "..." } @@ -124,7 +124,7 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for d "iss": "did:web:web3.storage", "aud": "did:web:spade.storage", "att": [{ - "with": "did:web:spade.storage", + "with": "did:web:web3.storage", "can": "aggregate/submit", "nb": { "link": "bagy...aggregate", @@ -152,7 +152,7 @@ This capability is invoked to submit a request to a broker service when an aggre } ``` -A receipt will be generated to acknowledge the received request. This receipt contains an [effect](https://github.com/ucan-wg/invocation/#7-effect) to be performed (`.fx` field pointing to another task) that will eventually get a receipt with either `ok` or `error`. +A receipt will be generated to acknowledge the received request. This receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate) ```json { @@ -180,9 +180,9 @@ Open questions: - can we get a `commP` of the aggregate with `commP` of every CAR that is part of it? - perhaps the `nb` should also include the `commP` of the aggregate? -### `aggregate/deals` +### `aggregate/get` -A Storefront principal can invoke a capabilty to submit an aggregate ready for deals. +A Storefront principal can invoke a capability to get state of the accepted aggregate. > `did:web:web3.storage` invokes capability from `did:web:spade.storage` @@ -191,8 +191,8 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for d "iss": "did:web:web3.storage", "aud": "did:web:spade.storage", "att": [{ - "with": "did:web:spade.storage", - "can": "aggregate/deals", + "with": "did:web:web3.storage", + "can": "aggregate/get", "nb": { "link": "bagy...aggregate", } @@ -201,7 +201,6 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for d "iss": "did:web:spade.storage", "aud": "did:web:web3.storage", "att": [{ "with": "did:web:spade.storage", "can": "aggregate/*" }], - "exp": null, "sig": "..." ], "sig": "..." @@ -262,7 +261,7 @@ A successful deal receipt will include the details of the deal: } ``` -If a deal fails, the failing commPs will be reported: +If offered aggregate is invalid, details on failing commPs are reported: ```json { From 3e624b7c555b6c427fdf4e364eb04413a18c78f4 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 11:12:02 +0200 Subject: [PATCH 04/31] chore: move bits around per other suggestion s --- w3-aggregation.md | 62 +++++++++++++++++++---------------------------- 1 file changed, 25 insertions(+), 37 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 5a2636e..292d11a 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -50,21 +50,11 @@ A Component of the [authority] that performs UCAN validation A Storefront is the entry point for user/application data into the web3. It will act on behalf of users to move data around into different storage points. One of the key storage presences may be Filecoin Storage Providers. -## Authorization +### Authorization -A Filecoin broker authority (e.g. Spade) MUST give capabilities to Storefront principals (e.g. web3.storage) to invoke capabilities to get stored content into Filecoin deals. +Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, if it would allow web3.storage to rotate keys without having too coordinate that with broker. -```mermaid -sequenceDiagram - participant Authority as 🌐

did:web:spade.storage - participant Storefront as 🌐

did:web:web3.storage - - Authority->>Storefront: delegate `aggregate/*` -``` - -Note that this can be done out of band with presentation of a delegation signature. - -## Deal creation +### Storefront offers broker an aggregate When a Storefront has enough content to fulfill an aggregate (each broker might have different requirements), a Filecoin deal for an aggregate SHALL be requested by a `aggregate/submit` invocations. Deal negotiations with Filecoin Storage Providers will be handled out off band. Therefore, Broker should generate a receipt to acknowledge received request. This receipt MUST contain a followup task in the (`.fx.join` field) that is run when submitted request is processed which MAY succeed (if aggregate was accepted) or fail (e.g. if aggregated was determined to be invalid). Result of the subsequent task CAN be looked up using it's receipt. @@ -79,16 +69,24 @@ sequenceDiagram Note left of Authority: Land deal in Filecoin ``` -## Deal information +### Broker queues the offer + +TODO + +### Broker reviews and handles the offer + +TODO -Storefront users MAY want to check details about deals from the content they previously stored. These deals will change over time as they get renewed. Therefore, Storefront should invoke `aggregate/deals` capability to gather information about given aggregate identifier. +### Storefront can query state of the aggregate deals + +Storefront users MAY want to check details about deals from the content they previously stored. These deals will change over time as they get renewed. Therefore, Storefront should invoke `aggregate/get` capability to gather information about given aggregate identifier. ```mermaid sequenceDiagram participant Storefront as 🌐

did:web:web3.storage participant Authority as 🌐

did:web:spade.storage - Storefront->>Authority: invoke `aggregate/deals` + Storefront->>Authority: invoke `aggregate/get` ``` ## Capabilities @@ -127,28 +125,27 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for d "with": "did:web:web3.storage", "can": "aggregate/submit", "nb": { - "link": "bagy...aggregate", + "offer": { + "link": "bagy...aggregate", + "size": 110101, + "commP": "commP...", + "src": ["https://w3s.link/ipfs/bagy...aggregate"] + } } }], - "prf": [ - "iss": "did:web:spade.storage", - "aud": "did:web:web3.storage", - "att": [{ "with": "did:web:spade.storage", "can": "aggregate/*" }], - "exp": null, - "sig": "..." - ], + "prf": [], "sig": "..." } ``` -This capability is invoked to submit a request to a broker service when an aggregate is ready for a Filecoin deal. The `nb.link` is the CAR CID that refers to a "Ferry" aggregate, a collection of dag-cbor blocks with format: +This capability is invoked to submit a request to a broker service when an aggregate is ready for a Filecoin deal. The `nb.offer.link` is the CAR CID that refers to a "Ferry" aggregate, a collection of dag-cbor blocks with format: ```json { "link": "bag...", "size": 110101, "commP": "commP...", - "url": "https://.../bag(...).car" + "src": ["https://.../bag(...).car"] } ``` @@ -163,11 +160,7 @@ A receipt will be generated to acknowledge the received request. This receipt MU } }, "fx": { - "fork": [ - "bafy...dealinvocation1", - "bafy...dealinvocation2", - "bafy...dealinvocation3" - ] + "join": { "/": "bafy...dequeue" } }, "meta": {}, "iss": "did:web:spade.storage", @@ -178,7 +171,6 @@ A receipt will be generated to acknowledge the received request. This receipt MU Open questions: - can we get a `commP` of the aggregate with `commP` of every CAR that is part of it? -- perhaps the `nb` should also include the `commP` of the aggregate? ### `aggregate/get` @@ -194,7 +186,7 @@ A Storefront principal can invoke a capability to get state of the accepted aggr "with": "did:web:web3.storage", "can": "aggregate/get", "nb": { - "link": "bagy...aggregate", + "commP": "commP...", } }], "prf": [ @@ -207,10 +199,6 @@ A Storefront principal can invoke a capability to get state of the accepted aggr } ``` -Open questions: - -- should the `nb` include only the `commP` of the aggregate as only field? - ### `deal/offer` When a broker receives an `aggregate/submit` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with multiple fork tasks to be performed asynchronously. From 7248631cffa7ae44b8ff26bd7762d3c27c48f70a Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 11:14:22 +0200 Subject: [PATCH 05/31] chore: apply suggestions from code review for deal out Co-authored-by: Irakli Gozalishvili --- w3-aggregation.md | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 292d11a..f051247 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -221,23 +221,15 @@ When a broker receives an `aggregate/submit` invocation from a Storefront Princi Once this invocation is executed, a receipt is generated with the result of the operation. -A successful deal receipt will include the details of the deal: +Accepted aggregate receipt will provide aggregate status info: ```json { - "ran": "bafy...invocation", + "ran": "bafy...review", "out": { "ok": { - "dealId": 142334312, - "storageProvider": "f07...", - "status": "Active", - "pieceCid": "bag...", - "dataCid": "bafy...", - "dataModelSelector": "Links/...", - "activation": "2023-04-13T01:58:00+00:00", - "expiration": "2024-09-05T01:58:00+00:00", - "created": "2023-04-11T17:57:30.522198+00:00", - "updated": "2024-04-12T03:42:26.928993+00:00" + "status": "accepted", + "link": "bafy...aggregate" } }, "fx": { @@ -256,7 +248,8 @@ If offered aggregate is invalid, details on failing commPs are reported: "ran": "bafy...invocation", "out": { "error": { - "parts": [{ + "status": "denied", + "cause": [{ "commP": "commP", "reason": "reasonCode", }], From 8286f15157d92d4255069b3c49654bb1ba8a1be4 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 12:29:46 +0200 Subject: [PATCH 06/31] chore: restructure per feedback --- w3-aggregation.md | 152 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 108 insertions(+), 44 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index f051247..30ff7d2 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -5,10 +5,14 @@ ## Editors - [Vasco Santos], [Protocol Labs] +- [Irakli Gozalishvili], [Protocol Labs] +- [Alan Shaw], [Protocol Labs] ## Authors - [Vasco Santos], [Protocol Labs] +- [Irakli Gozalishvili], [Protocol Labs] +- [Alan Shaw], [Protocol Labs] # Abstract @@ -48,34 +52,55 @@ A Component of the [authority] that performs UCAN validation ## Overview -A Storefront is the entry point for user/application data into the web3. It will act on behalf of users to move data around into different storage points. One of the key storage presences may be Filecoin Storage Providers. +A Storefront is the entry point for user/application data into web3. It will act on behalf of users to move data around into different storage points. One of the key storage presences may be Filecoin Storage Providers. ### Authorization -Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, if it would allow web3.storage to rotate keys without having too coordinate that with broker. +Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, it allows web3.storage to rotate keys without having too coordinate that with the broker. ### Storefront offers broker an aggregate -When a Storefront has enough content to fulfill an aggregate (each broker might have different requirements), a Filecoin deal for an aggregate SHALL be requested by a `aggregate/submit` invocations. Deal negotiations with Filecoin Storage Providers will be handled out off band. Therefore, Broker should generate a receipt to acknowledge received request. This receipt MUST contain a followup task in the (`.fx.join` field) that is run when submitted request is processed which MAY succeed (if aggregate was accepted) or fail (e.g. if aggregated was determined to be invalid). Result of the subsequent task CAN be looked up using it's receipt. +When a Storefront has enough content to fulfill an aggregate (each broker MAY have different requirements), a Filecoin deal for an aggregate SHALL be requested by a `aggregate/offer` invocations. Deal negotiations with Filecoin Storage Providers will be handled out off band. Broker SHOULD generate a receipt to acknowledge the received request. ```mermaid sequenceDiagram participant Storefront as 🌐

did:web:web3.storage participant Authority as 🌐

did:web:spade.storage - Storefront->>Authority: invoke `aggregate/submit` - Note left of Authority: Prepare for deal - Authority-->>Storefront: receipt - Note left of Authority: Land deal in Filecoin + Storefront->>Authority: invoke `aggregate/offer` + Note left of Authority: Request offer to be queued ``` ### Broker queues the offer -TODO +Once broker successfuly gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. + +This receipt MUST contain a follow up task in the (`.fx.join` field) that is run when submitted request is processed. It MAY succeed (if aggregate was accepted) or fail (if aggregated was determined to be invalid). The result of the subsequent task CAN be looked up using its receipt. + +```mermaid +sequenceDiagram + participant Authority as 🌐

did:web:spade.storage + participant Storefront as 🌐

did:web:web3.storage + + Storefront->>Authority: invoke `aggregate/offer` + Note left of Storefront: Queueing offer + Authority-->>Storefront: receipt issued +``` ### Broker reviews and handles the offer -TODO +When a broker pops the offer from the queue, the offer details MUST be retrievable. With the offer details, the broker MAY interact with available Filecoin Storage Providers, in order to establish a previously determined number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handlded or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. + +```mermaid +sequenceDiagram + participant Authority as 🌐

did:web:spade.storage + participant Storefront as 🌐

did:web:web3.storage + + Note left of Storefront: Review and handle offer async + Authority-->>Storefront: receipt issued +``` + +Once offer is accepted broker takes care of arranging and renewing deals. ### Storefront can query state of the aggregate deals @@ -93,27 +118,11 @@ participant Storefront as 🌐

did:web:web3.storage This section describes the capabilities that form the w3 aggregation protocol, along with the details relevant for invoking capabilities with a service provider. -In this document, we will be looking at `spade.storage` as an implementer of the `aggregate` and `deals/*` protocol. - -### `aggregate/*` +In this document, we will be looking at `spade.storage` as an implementer of the `aggregate/*` and `offer/*` protocol. -A Filecoin broker Authority MAY delegate capabilities to any Storefront principal. - -> `did:web:spade.storage` delegates `aggregate/*` capabilities to `did:web:web3.storage` - -```json -{ - "iss": "did:web:spade.storage", - "aud": "did:web:web3.storage", - "att": [{"with": "did:web:web3.storage", "can": "aggregate/*" }], - "exp": null, - "sig": "..." -} -``` +### `aggregate/offer` -### `aggregate/submit` - -A Storefront principal can invoke a capabilty to submit an aggregate ready for deals. +A Storefront principal can invoke a capabilty to submit an aggregate ready for offers. > `did:web:web3.storage` invokes capability from `did:web:spade.storage` @@ -123,7 +132,7 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for d "aud": "did:web:spade.storage", "att": [{ "with": "did:web:web3.storage", - "can": "aggregate/submit", + "can": "aggregate/offer", "nb": { "offer": { "link": "bagy...aggregate", @@ -170,11 +179,11 @@ A receipt will be generated to acknowledge the received request. This receipt MU Open questions: -- can we get a `commP` of the aggregate with `commP` of every CAR that is part of it? +- can we get `commP` of the aggregate with `commP` of every CAR that is part of it? ### `aggregate/get` -A Storefront principal can invoke a capability to get state of the accepted aggregate. +A Storefront principal can invoke a capability to get state of an accepted aggregate. > `did:web:web3.storage` invokes capability from `did:web:spade.storage` @@ -189,19 +198,14 @@ A Storefront principal can invoke a capability to get state of the accepted aggr "commP": "commP...", } }], - "prf": [ - "iss": "did:web:spade.storage", - "aud": "did:web:web3.storage", - "att": [{ "with": "did:web:spade.storage", "can": "aggregate/*" }], - "sig": "..." - ], + "prf": [], "sig": "..." } ``` -### `deal/offer` +### `offer/review` -When a broker receives an `aggregate/submit` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with multiple fork tasks to be performed asynchronously. +When a broker receives an `aggregate/offer` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with join task to be performed asynchronously. ```json { @@ -209,7 +213,7 @@ When a broker receives an `aggregate/submit` invocation from a Storefront Princi "aud": "did:web:web3.storage", "att": [{ "with": "did:web:spade.storage", - "can": "deal/offer", + "can": "offer/review", "nb": { "commP": "commP", } @@ -219,9 +223,7 @@ When a broker receives an `aggregate/submit` invocation from a Storefront Princi } ``` -Once this invocation is executed, a receipt is generated with the result of the operation. - -Accepted aggregate receipt will provide aggregate status info: +Once this invocation is executed, a receipt is generated with the status of the task updated. Accepted aggregate receipt will provide aggregate status info: ```json { @@ -241,7 +243,7 @@ Accepted aggregate receipt will provide aggregate status info: } ``` -If offered aggregate is invalid, details on failing commPs are reported: +If offered aggregate is invalid, details on failing commPs are also reported: ```json { @@ -264,6 +266,66 @@ If offered aggregate is invalid, details on failing commPs are reported: } ``` +### Schema + +```ipldsch +type Aggregate = + { "status": "queued", link: Link } | + { "status": "accepted", link: Link } | + { "status": "denied", link: Link } + +type AggregateCapability enum { + AggregateOffer "aggregate/offer" + AggregateGet "aggregate/get" +} representation inline { + discriminantKey "can" +} + +type AggregateGet struct { + with StorefrontDID + nb SucceedAggregateRef +} + +type SucceedAggregateRef struct { + commP string +} + +type AggregateRef struct { + link &AggregateCAR +} + +type AggregateOffer struct { + with StorefrontDID + nb AggregateOfferDetail +} + +type AggregateOfferDetail struct { + offer: Offer +} + +type Offer { + link &AggregateCAR + src [URL] + size number + commP string +} + +type StorefrontDID string +type URL string +type AggregateCAR any +type OfferCapability union { + OfferReview "offer/review" +} representation inline { + discriminantKey "can" +} + +type OfferReview struct { + with BrokerDID nb AggregateRef +} + +type BrokerDID string +``` + [`did:web`]: https://w3c-ccg.github.io/did-method-web/ [UCAN]: https://github.com/ucan-wg/spec/ [principal]: https://github.com/ucan-wg/spec/#321-principals @@ -271,3 +333,5 @@ If offered aggregate is invalid, details on failing commPs are reported: [Protocol Labs]:https://protocol.ai/ [Vasco Santos]:https://github.com/vasco-santos +[Irakli Gozalishvili]:https://github.com/Gozala +[Alan Shaw]:https://github.com/alanshaw From d5cb7122d5233d3ae27687bcfe239df0452db5d1 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 12:40:40 +0200 Subject: [PATCH 07/31] fix: spell issues --- w3-aggregation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 30ff7d2..8cc072f 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -73,7 +73,7 @@ sequenceDiagram ### Broker queues the offer -Once broker successfuly gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. +Once broker successfully gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. This receipt MUST contain a follow up task in the (`.fx.join` field) that is run when submitted request is processed. It MAY succeed (if aggregate was accepted) or fail (if aggregated was determined to be invalid). The result of the subsequent task CAN be looked up using its receipt. @@ -89,7 +89,7 @@ sequenceDiagram ### Broker reviews and handles the offer -When a broker pops the offer from the queue, the offer details MUST be retrievable. With the offer details, the broker MAY interact with available Filecoin Storage Providers, in order to establish a previously determined number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handlded or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. +When a broker pops the offer from the queue, the offer details MUST be retrievable. With the offer details, the broker MAY interact with available Filecoin Storage Providers, in order to establish a previously determined number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. ```mermaid sequenceDiagram From 1de9b86d60353a076a36028d157044765a14ffac Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 16:53:09 +0200 Subject: [PATCH 08/31] chore: apply suggestions from code review Co-authored-by: Alan Shaw --- w3-aggregation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 8cc072f..50a661d 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -56,7 +56,7 @@ A Storefront is the entry point for user/application data into web3. It will act ### Authorization -Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, it allows web3.storage to rotate keys without having too coordinate that with the broker. +Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, it allows web3.storage to rotate keys without having to coordinate with the broker. ### Storefront offers broker an aggregate @@ -250,7 +250,7 @@ If offered aggregate is invalid, details on failing commPs are also reported: "ran": "bafy...invocation", "out": { "error": { - "status": "denied", + "status": "rejected", "cause": [{ "commP": "commP", "reason": "reasonCode", @@ -272,7 +272,7 @@ If offered aggregate is invalid, details on failing commPs are also reported: type Aggregate = { "status": "queued", link: Link } | { "status": "accepted", link: Link } | - { "status": "denied", link: Link } + { "status": "rejected", link: Link } type AggregateCapability enum { AggregateOffer "aggregate/offer" From 58026c50109de18e4b48a5563b3ced69a78b4606 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 16:59:43 +0200 Subject: [PATCH 09/31] chore: address review comments --- w3-aggregation.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 50a661d..1340d8f 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -69,6 +69,7 @@ sequenceDiagram Storefront->>Authority: invoke `aggregate/offer` Note left of Authority: Request offer to be queued + Authority-->>Storefront: receipt issued ``` ### Broker queues the offer @@ -77,26 +78,16 @@ Once broker successfully gets an offer, the offer gets queued for review. A rece This receipt MUST contain a follow up task in the (`.fx.join` field) that is run when submitted request is processed. It MAY succeed (if aggregate was accepted) or fail (if aggregated was determined to be invalid). The result of the subsequent task CAN be looked up using its receipt. -```mermaid -sequenceDiagram - participant Authority as 🌐

did:web:spade.storage - participant Storefront as 🌐

did:web:web3.storage - - Storefront->>Authority: invoke `aggregate/offer` - Note left of Storefront: Queueing offer - Authority-->>Storefront: receipt issued -``` - ### Broker reviews and handles the offer When a broker pops the offer from the queue, the offer details MUST be retrievable. With the offer details, the broker MAY interact with available Filecoin Storage Providers, in order to establish a previously determined number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. ```mermaid sequenceDiagram - participant Authority as 🌐

did:web:spade.storage participant Storefront as 🌐

did:web:web3.storage + participant Authority as 🌐

did:web:spade.storage - Note left of Storefront: Review and handle offer async + Note right of Storefront: Review and handle offer async Authority-->>Storefront: receipt issued ``` @@ -158,6 +149,8 @@ This capability is invoked to submit a request to a broker service when an aggre } ``` +In addition, the calculated `commP` of the entire aggregate and the total size of every CAR within it is provided for convenience and consistency with the dag-cbor blocks. + A receipt will be generated to acknowledge the received request. This receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate) ```json From dae9ff404015b8eecfe07e4c9384937444b9494d Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 17:29:00 +0200 Subject: [PATCH 10/31] chore: add receipt for aggregate get --- w3-aggregation.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/w3-aggregation.md b/w3-aggregation.md index 1340d8f..ab6c13f 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -196,6 +196,38 @@ A Storefront principal can invoke a capability to get state of an accepted aggre } ``` +Once this invocation is executed, a receipt is generated with the aggregate information: + +```json +{ + "ran": "bafy...get", + "out": { + "ok": { + "deals": [ + { + "dealId": 111, + "storageProvider": "f07...", + "status": "Active", + "pieceCid": "bag...", + "dataCid": "bafy...", + "dataModelSelector": "Links/...", + "activation": "2023-04-13T01:58:00+00:00", + "expiration": "2024-09-05T01:58:00+00:00", + "created": "2023-04-11T17:57:30.522198+00:00", + "updated": "2024-04-12T03:42:26.928993+00:00" + } + ] + }, + }, + "fx": { + "fork": [] + }, + "meta": {}, + "iss": "did:web:spade.storage", + "prf": [] +} +``` + ### `offer/review` When a broker receives an `aggregate/offer` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with join task to be performed asynchronously. From 91e928080484202966556f87d020404fb31035fe Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 28 Mar 2023 17:35:26 +0200 Subject: [PATCH 11/31] chore: rephrrase dequeue --- w3-aggregation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index ab6c13f..755426c 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -80,7 +80,7 @@ This receipt MUST contain a follow up task in the (`.fx.join` field) that is run ### Broker reviews and handles the offer -When a broker pops the offer from the queue, the offer details MUST be retrievable. With the offer details, the broker MAY interact with available Filecoin Storage Providers, in order to establish a previously determined number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. +When a broker dequeues the offer from the queue, the offer details MUST be retrievable. With the offer details, the broker will interact with available Filecoin Storage Providers, in order to establish a previously determined number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. ```mermaid sequenceDiagram From 7695dc0cfdf737c4d7c7b5fc0966755b66a3a60a Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Fri, 31 Mar 2023 10:55:15 +0200 Subject: [PATCH 12/31] fix: drop commp of aggregate --- w3-aggregation.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 755426c..c2d2ddf 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -128,7 +128,6 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for o "offer": { "link": "bagy...aggregate", "size": 110101, - "commP": "commP...", "src": ["https://w3s.link/ipfs/bagy...aggregate"] } } @@ -149,7 +148,7 @@ This capability is invoked to submit a request to a broker service when an aggre } ``` -In addition, the calculated `commP` of the entire aggregate and the total size of every CAR within it is provided for convenience and consistency with the dag-cbor blocks. +A `src` with URLs to fetch the aggregate offer is provided. In addition, the total `size` of every CAR within it is provided, both for convenience and consistency with the dag-cbor blocks. A receipt will be generated to acknowledge the received request. This receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate) @@ -172,8 +171,6 @@ A receipt will be generated to acknowledge the received request. This receipt MU Open questions: -- can we get `commP` of the aggregate with `commP` of every CAR that is part of it? - ### `aggregate/get` A Storefront principal can invoke a capability to get state of an accepted aggregate. @@ -188,7 +185,7 @@ A Storefront principal can invoke a capability to get state of an accepted aggre "with": "did:web:web3.storage", "can": "aggregate/get", "nb": { - "commP": "commP...", + "link": "bagy...aggregate", } }], "prf": [], @@ -240,7 +237,7 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip "with": "did:web:spade.storage", "can": "offer/review", "nb": { - "commP": "commP", + "link": "bagy...aggregate", } }], "prf": [], @@ -276,6 +273,7 @@ If offered aggregate is invalid, details on failing commPs are also reported: "out": { "error": { "status": "rejected", + "link": "bafy...aggregate", "cause": [{ "commP": "commP", "reason": "reasonCode", @@ -312,7 +310,7 @@ type AggregateGet struct { } type SucceedAggregateRef struct { - commP string + link &AggregateCAR } type AggregateRef struct { @@ -332,7 +330,6 @@ type Offer { link &AggregateCAR src [URL] size number - commP string } type StorefrontDID string From 965ceedd59b0843058eb990d35331e21ef13ef4c Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Fri, 31 Mar 2023 11:12:31 +0200 Subject: [PATCH 13/31] fix: use single dag cbor block with info of cars to add to aggregate instead of car file --- w3-aggregation.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index c2d2ddf..5539cfd 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -76,7 +76,7 @@ sequenceDiagram Once broker successfully gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. -This receipt MUST contain a follow up task in the (`.fx.join` field) that is run when submitted request is processed. It MAY succeed (if aggregate was accepted) or fail (if aggregated was determined to be invalid). The result of the subsequent task CAN be looked up using its receipt. +This receipt MUST contain a follow up task in the (`.fx.join` field) that is run when submitted request is processed. It MAY succeed (if aggregate was accepted) or fail (if aggregated was determined to be invalid). The result of the subsequent task CAN be looked up using its receipt. ### Broker reviews and handles the offer @@ -126,9 +126,9 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for o "can": "aggregate/offer", "nb": { "offer": { - "link": "bagy...aggregate", + "link": "bafy...aggregate", "size": 110101, - "src": ["https://w3s.link/ipfs/bagy...aggregate"] + "src": ["https://w3s.link/ipfs/bafy...aggregate"] } } }], @@ -137,15 +137,17 @@ A Storefront principal can invoke a capabilty to submit an aggregate ready for o } ``` -This capability is invoked to submit a request to a broker service when an aggregate is ready for a Filecoin deal. The `nb.offer.link` is the CAR CID that refers to a "Ferry" aggregate, a collection of dag-cbor blocks with format: +This capability is invoked to submit a request to a broker service when an aggregate is ready for a Filecoin deal. The `nb.offer.link` is the DAG-CBOR CID that refers to a "Ferry" aggregate. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregate. Its format is: ```json -{ - "link": "bag...", - "size": 110101, - "commP": "commP...", - "src": ["https://.../bag(...).car"] -} +[ + { + "link": "bag...", + "size": 110101, + "commP": "commP...", + "src": ["https://.../bag(...).car"] + } +] ``` A `src` with URLs to fetch the aggregate offer is provided. In addition, the total `size` of every CAR within it is provided, both for convenience and consistency with the dag-cbor blocks. @@ -169,8 +171,6 @@ A receipt will be generated to acknowledge the received request. This receipt MU } ``` -Open questions: - ### `aggregate/get` A Storefront principal can invoke a capability to get state of an accepted aggregate. @@ -185,7 +185,7 @@ A Storefront principal can invoke a capability to get state of an accepted aggre "with": "did:web:web3.storage", "can": "aggregate/get", "nb": { - "link": "bagy...aggregate", + "link": "bafy...aggregate", } }], "prf": [], @@ -237,7 +237,7 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip "with": "did:web:spade.storage", "can": "offer/review", "nb": { - "link": "bagy...aggregate", + "link": "bafy...aggregate", } }], "prf": [], From 49daef571e0479f0557c2e0e5c0e03fad5d45bc8 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 11 Apr 2023 17:26:29 +0200 Subject: [PATCH 14/31] chore: apply suggestions from code review Co-authored-by: Irakli Gozalishvili --- w3-aggregation.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 5539cfd..d254492 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -56,11 +56,11 @@ A Storefront is the entry point for user/application data into web3. It will act ### Authorization -Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, it allows web3.storage to rotate keys without having to coordinate with the broker. +Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, it allows web3.storage to rotate keys and/or re-delegate access without having to coordinate with the broker. ### Storefront offers broker an aggregate -When a Storefront has enough content to fulfill an aggregate (each broker MAY have different requirements), a Filecoin deal for an aggregate SHALL be requested by a `aggregate/offer` invocations. Deal negotiations with Filecoin Storage Providers will be handled out off band. Broker SHOULD generate a receipt to acknowledge the received request. +When a Storefront has enough content to fulfill an aggregate (each broker MAY have different requirements), a Filecoin deal for an aggregate MAY be requested by an `aggregate/offer` invocation. Deal negotiations with Filecoin Storage Providers SHOULD be handled out off band. Broker MUST acknowledge a request by issuing a signed receipt. ```mermaid sequenceDiagram @@ -76,7 +76,9 @@ sequenceDiagram Once broker successfully gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. -This receipt MUST contain a follow up task in the (`.fx.join` field) that is run when submitted request is processed. It MAY succeed (if aggregate was accepted) or fail (if aggregated was determined to be invalid). The result of the subsequent task CAN be looked up using its receipt. +This receipt MUST have link to a followup task (using `.fx.join` field) that either succeeds (if aggregate was accepted) or fails (if aggregated was determined to be invalid) so that it's receipt COULD be looked up using it. + +> Note: Aggregator MAY have several intermediate steps and states it transitions though, however those intentionally are not captured by this protocol, because storefront will take no action until success / failure condition is met. ### Broker reviews and handles the offer From 208e831478e7fdffb597e744b48ce66120e31857 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 11 Apr 2023 18:36:41 +0200 Subject: [PATCH 15/31] chore: apply suggestions from code review with schema Co-authored-by: Irakli Gozalishvili --- w3-aggregation.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/w3-aggregation.md b/w3-aggregation.md index d254492..12a7f78 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -117,6 +117,26 @@ In this document, we will be looking at `spade.storage` as an implementer of the A Storefront principal can invoke a capabilty to submit an aggregate ready for offers. +```iplsch +type AggregateOffer union { + Aggregate "aggregate/offer" +} representation inline { + discriminantKey "can" +} + +type Offer struct { + with StorefrontDID + offer { CARLink: OfferDetails } +} + +type struct OfferDetails { + size Int + commP Bytes + src [URL] +} + +type CARLink string + > `did:web:web3.storage` invokes capability from `did:web:spade.storage` ```json From 02134504b78a8daca74e85167be89e975afb41df Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 11 Apr 2023 18:44:09 +0200 Subject: [PATCH 16/31] chore: apply suggestions from code review Co-authored-by: Irakli Gozalishvili --- w3-aggregation.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 12a7f78..532c10b 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -314,10 +314,11 @@ If offered aggregate is invalid, details on failing commPs are also reported: ### Schema ```ipldsch -type Aggregate = - { "status": "queued", link: Link } | - { "status": "accepted", link: Link } | - { "status": "rejected", link: Link } +type Aggregate union { + | Link "queued" + | Link "status" + | Link "rejected" +} representation keyed type AggregateCapability enum { AggregateOffer "aggregate/offer" From c7016e0a47e077067f30c29a60cf5305be4f7a14 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 11 Apr 2023 18:45:03 +0200 Subject: [PATCH 17/31] chore: address review comments --- w3-aggregation.md | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 532c10b..33d13b5 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -74,7 +74,7 @@ sequenceDiagram ### Broker queues the offer -Once broker successfully gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. +Once broker successfully gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. It is worth mentioning that if an offer is from an aggregate that is already `queued` or `complete` it is ignored. This receipt MUST have link to a followup task (using `.fx.join` field) that either succeeds (if aggregate was accepted) or fails (if aggregated was determined to be invalid) so that it's receipt COULD be looked up using it. @@ -82,7 +82,7 @@ This receipt MUST have link to a followup task (using `.fx.join` field) that eit ### Broker reviews and handles the offer -When a broker dequeues the offer from the queue, the offer details MUST be retrievable. With the offer details, the broker will interact with available Filecoin Storage Providers, in order to establish a previously determined number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. +When a broker dequeues the offer from the queue, its details are retrived from the provided source URL. With the offer details, the broker will interact with available Filecoin Storage Providers, in order to establish a previously determined (out of band) number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. ```mermaid sequenceDiagram @@ -93,7 +93,7 @@ sequenceDiagram Authority-->>Storefront: receipt issued ``` -Once offer is accepted broker takes care of arranging and renewing deals. +Once offer gets into `accepted` state, broker takes care of renewing deals. ### Storefront can query state of the aggregate deals @@ -164,10 +164,10 @@ This capability is invoked to submit a request to a broker service when an aggre ```json [ { - "link": "bag...", + "link": "bafy...", "size": 110101, "commP": "commP...", - "src": ["https://.../bag(...).car"] + "src": ["https://.../bafy(...)"] } ] ``` @@ -222,9 +222,8 @@ Once this invocation is executed, a receipt is generated with the aggregate info "ran": "bafy...get", "out": { "ok": { - "deals": [ - { - "dealId": 111, + "deals": { + "111": { "storageProvider": "f07...", "status": "Active", "pieceCid": "bag...", @@ -235,7 +234,7 @@ Once this invocation is executed, a receipt is generated with the aggregate info "created": "2023-04-11T17:57:30.522198+00:00", "updated": "2024-04-12T03:42:26.928993+00:00" } - ] + } }, }, "fx": { @@ -267,14 +266,13 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip } ``` -Once this invocation is executed, a receipt is generated with the status of the task updated. Accepted aggregate receipt will provide aggregate status info: +Once this invocation is executed, a receipt is generated with the status of the task updated. Accepted aggregate receipt will provide aggregate offer operation info: ```json { "ran": "bafy...review", "out": { "ok": { - "status": "accepted", "link": "bafy...aggregate" } }, @@ -294,7 +292,6 @@ If offered aggregate is invalid, details on failing commPs are also reported: "ran": "bafy...invocation", "out": { "error": { - "status": "rejected", "link": "bafy...aggregate", "cause": [{ "commP": "commP", From d4e8912c472f9be55ef6e9550d2b571d835f8a19 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 11 Apr 2023 18:48:58 +0200 Subject: [PATCH 18/31] chore: appease lint --- w3-aggregation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 33d13b5..89cac4c 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -82,7 +82,7 @@ This receipt MUST have link to a followup task (using `.fx.join` field) that eit ### Broker reviews and handles the offer -When a broker dequeues the offer from the queue, its details are retrived from the provided source URL. With the offer details, the broker will interact with available Filecoin Storage Providers, in order to establish a previously determined (out of band) number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. +When a broker dequeues the offer from the queue, its details are retrieved from the provided source URL. With the offer details, the broker will interact with available Filecoin Storage Providers, in order to establish a previously determined (out of band) number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. ```mermaid sequenceDiagram @@ -195,7 +195,7 @@ A receipt will be generated to acknowledge the received request. This receipt MU ### `aggregate/get` -A Storefront principal can invoke a capability to get state of an accepted aggregate. +A Storefront principal can invoke a capability to get state of a previously accepted aggregate. > `did:web:web3.storage` invokes capability from `did:web:spade.storage` From 904e2d3efae2b089ed997aec26bdfadde82c3eea Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Tue, 11 Apr 2023 18:51:07 +0200 Subject: [PATCH 19/31] chore: rename review to arrange --- w3-aggregation.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 89cac4c..d7e24dd 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -246,7 +246,7 @@ Once this invocation is executed, a receipt is generated with the aggregate info } ``` -### `offer/review` +### `offer/arrange` When a broker receives an `aggregate/offer` invocation from a Storefront Principal, an [Effect](https://github.com/ucan-wg/invocation/#7-effect) for this submission is created with join task to be performed asynchronously. @@ -256,7 +256,7 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip "aud": "did:web:web3.storage", "att": [{ "with": "did:web:spade.storage", - "can": "offer/review", + "can": "offer/arrange", "nb": { "link": "bafy...aggregate", } @@ -270,7 +270,7 @@ Once this invocation is executed, a receipt is generated with the status of the ```json { - "ran": "bafy...review", + "ran": "bafy...arrange", "out": { "ok": { "link": "bafy...aggregate" @@ -356,12 +356,12 @@ type StorefrontDID string type URL string type AggregateCAR any type OfferCapability union { - OfferReview "offer/review" + OfferArrange "offer/arrange" } representation inline { discriminantKey "can" } -type OfferReview struct { +type OfferArrange struct { with BrokerDID nb AggregateRef } From 7dee5329a5940d8e263246b8defdb9dedb2cef5b Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 11 Apr 2023 23:10:07 -0700 Subject: [PATCH 20/31] chore: close code block --- w3-aggregation.md | 1 + 1 file changed, 1 insertion(+) diff --git a/w3-aggregation.md b/w3-aggregation.md index d7e24dd..e4a563b 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -136,6 +136,7 @@ type struct OfferDetails { } type CARLink string +``` > `did:web:web3.storage` invokes capability from `did:web:spade.storage` From 5a5038526835a3eff7063008b232c1ea0e42b800 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Wed, 12 Apr 2023 12:07:54 +0200 Subject: [PATCH 21/31] chore: rename some of aggregate mentions to offer for better clarity --- w3-aggregation.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index e4a563b..1d77b36 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -149,9 +149,9 @@ type CARLink string "can": "aggregate/offer", "nb": { "offer": { - "link": "bafy...aggregate", + "link": "bafy...offer", "size": 110101, - "src": ["https://w3s.link/ipfs/bafy...aggregate"] + "src": ["https://w3s.link/ipfs/bafy...offer"] } } }], @@ -160,7 +160,7 @@ type CARLink string } ``` -This capability is invoked to submit a request to a broker service when an aggregate is ready for a Filecoin deal. The `nb.offer.link` is the DAG-CBOR CID that refers to a "Ferry" aggregate. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregate. Its format is: +This capability is invoked to submit a request to a broker service when an offer is ready for a Filecoin deal. The `nb.offer.link` is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. Its format is: ```json [ @@ -173,9 +173,9 @@ This capability is invoked to submit a request to a broker service when an aggre ] ``` -A `src` with URLs to fetch the aggregate offer is provided. In addition, the total `size` of every CAR within it is provided, both for convenience and consistency with the dag-cbor blocks. +A `src` with URLs to fetch the offer is provided. In addition, the total `size` of every CAR within it is provided, both for convenience and consistency with the dag-cbor blocks. -A receipt will be generated to acknowledge the received request. This receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate) +A receipt will be generated to acknowledge the received request. This receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted offer is processed and either succeeds (implying that offer was accepted and deals will be arranged) or fail (with `error` describing a problem with an offer) ```json { @@ -196,7 +196,7 @@ A receipt will be generated to acknowledge the received request. This receipt MU ### `aggregate/get` -A Storefront principal can invoke a capability to get state of a previously accepted aggregate. +A Storefront principal can invoke a capability to get state of a previously accepted offer. > `did:web:web3.storage` invokes capability from `did:web:spade.storage` @@ -208,7 +208,7 @@ A Storefront principal can invoke a capability to get state of a previously acce "with": "did:web:web3.storage", "can": "aggregate/get", "nb": { - "link": "bafy...aggregate", + "link": "bafy...offer", } }], "prf": [], @@ -216,7 +216,7 @@ A Storefront principal can invoke a capability to get state of a previously acce } ``` -Once this invocation is executed, a receipt is generated with the aggregate information: +Once this invocation is executed, a receipt is generated with the resulting aggregate information: ```json { @@ -259,7 +259,7 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip "with": "did:web:spade.storage", "can": "offer/arrange", "nb": { - "link": "bafy...aggregate", + "link": "bafy...offer", } }], "prf": [], @@ -274,7 +274,7 @@ Once this invocation is executed, a receipt is generated with the status of the "ran": "bafy...arrange", "out": { "ok": { - "link": "bafy...aggregate" + "link": "bafy...offer" } }, "fx": { @@ -293,7 +293,7 @@ If offered aggregate is invalid, details on failing commPs are also reported: "ran": "bafy...invocation", "out": { "error": { - "link": "bafy...aggregate", + "link": "bafy...offer", "cause": [{ "commP": "commP", "reason": "reasonCode", From 49f5704955fcd3eab4a4ef3d394b1f9da9fe5d4c Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Wed, 12 Apr 2023 12:14:16 +0200 Subject: [PATCH 22/31] chore: add other feedback --- w3-aggregation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 1d77b36..a86c970 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -160,7 +160,7 @@ type CARLink string } ``` -This capability is invoked to submit a request to a broker service when an offer is ready for a Filecoin deal. The `nb.offer.link` is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. Its format is: +Invoking `aggregate/offer` capability submits an offer to a broker service to arrange a Filecoin deals. The `nb.offer` representing a "Ferry" aggregate offer that is ready for a Filecoin deal. The `link` field is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. Its format is: ```json [ @@ -173,9 +173,9 @@ This capability is invoked to submit a request to a broker service when an offer ] ``` -A `src` with URLs to fetch the offer is provided. In addition, the total `size` of every CAR within it is provided, both for convenience and consistency with the dag-cbor blocks. +The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field of each piece MUST be set to the byte size of the combined CAR files. -A receipt will be generated to acknowledge the received request. This receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted offer is processed and either succeeds (implying that offer was accepted and deals will be arranged) or fail (with `error` describing a problem with an offer) +Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate). ```json { From ec12c091359ed7c186aa97cfb1aba961e714cbc5 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Wed, 12 Apr 2023 12:23:30 +0200 Subject: [PATCH 23/31] chore: add link to offer arrange from aggregate offer --- w3-aggregation.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/w3-aggregation.md b/w3-aggregation.md index a86c970..d3b182b 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -194,6 +194,8 @@ Broker MUST issue a signed receipt to acknowledge the received request. Issued r } ``` +See [`offer/arrange`](#offerarrange) section to see the subsequent task. + ### `aggregate/get` A Storefront principal can invoke a capability to get state of a previously accepted offer. From 4adee826bbe51b03cc6a296dc4e69b1ac84432fb Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Thu, 13 Apr 2023 17:18:38 +0200 Subject: [PATCH 24/31] chore: other renames from car to cbor --- w3-aggregation.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index d3b182b..d674a03 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -131,7 +131,7 @@ type Offer struct { type struct OfferDetails { size Int - commP Bytes + link String src [URL] } @@ -333,11 +333,11 @@ type AggregateGet struct { } type SucceedAggregateRef struct { - link &AggregateCAR + link &AggregateCBOR } type AggregateRef struct { - link &AggregateCAR + link &AggregateCBOR } type AggregateOffer struct { @@ -350,14 +350,14 @@ type AggregateOfferDetail struct { } type Offer { - link &AggregateCAR + link &AggregateCBOR src [URL] size number } type StorefrontDID string type URL string -type AggregateCAR any +type AggregateCBOR any type OfferCapability union { OfferArrange "offer/arrange" } representation inline { From 9b80833d8cd308b131fd00ed9436d4063f40da02 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Fri, 14 Apr 2023 18:38:41 +0200 Subject: [PATCH 25/31] fix: add cbor block in invocation CAR file to avoid io to execute capability --- w3-aggregation.md | 61 +++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index d674a03..0bb0a95 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -118,24 +118,25 @@ In this document, we will be looking at `spade.storage` as an implementer of the A Storefront principal can invoke a capabilty to submit an aggregate ready for offers. ```iplsch -type AggregateOffer union { - Aggregate "aggregate/offer" -} representation inline { - discriminantKey "can" +type AggregateOffer struct { + with StorefrontDID + nb AggregateOfferDetail } -type Offer struct { - with StorefrontDID - offer { CARLink: OfferDetails } +type AggregateOfferDetail struct { + offer: OfferCBOR } +type OfferCBOR any + type struct OfferDetails { size Int - link String + archive Archive src [URL] + proof Proof } -type CARLink string +type Offer [OfferDetails] ``` > `did:web:web3.storage` invokes capability from `did:web:spade.storage` @@ -145,14 +146,10 @@ type CARLink string "iss": "did:web:web3.storage", "aud": "did:web:spade.storage", "att": [{ - "with": "did:web:web3.storage", + "with": "did:web:spade.storage", "can": "aggregate/offer", "nb": { - "offer": { - "link": "bafy...offer", - "size": 110101, - "src": ["https://w3s.link/ipfs/bafy...offer"] - } + "offer": { "/": "bafy...many-cars" } /* dag-cbor CID */ } }], "prf": [], @@ -160,20 +157,23 @@ type CARLink string } ``` -Invoking `aggregate/offer` capability submits an offer to a broker service to arrange a Filecoin deals. The `nb.offer` representing a "Ferry" aggregate offer that is ready for a Filecoin deal. The `link` field is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. Its format is: +Invoking `aggregate/offer` capability submits an offer to a broker service to arrange a Filecoin deals. The `nb.offer` represents a "Ferry" aggregate offer that is ready for a Filecoin deal. Its value is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. This block MUST be included in CAR file that transports the invocation. Its format is: ```json -[ - { - "link": "bafy...", +/* decoded offers block */ +[ { + "archive": { "/": "bag...file0" }, /* CAR CID */ "size": 110101, - "commP": "commP...", - "src": ["https://.../bafy(...)"] - } + "proof": { "/": "bag...file0" }, /* COMMP CID */ + "src": ["https://w3s.link/ipfs/bag...file0"] + }, + { + /* ... */ + } ] ``` -The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field of each piece MUST be set to the byte size of the combined CAR files. +Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `archive` field has the CAR cid, while the `proof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field MUST be set to the byte size of the CAR file. Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate). @@ -346,15 +346,20 @@ type AggregateOffer struct { } type AggregateOfferDetail struct { - offer: Offer + offer: OfferCBOR } -type Offer { - link &AggregateCBOR - src [URL] - size number +type OfferCBOR any + +type struct OfferDetails { + size Int + archive Archive + src [URL] + proof Proof } +type Offer [OfferDetails] + type StorefrontDID string type URL string type AggregateCBOR any From 0d2f4645d342b0eedf13cad93b9d923d4f992593 Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Fri, 14 Apr 2023 18:39:45 +0200 Subject: [PATCH 26/31] fix: spell issues --- w3-aggregation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 0bb0a95..e4fd19e 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -173,7 +173,7 @@ Invoking `aggregate/offer` capability submits an offer to a broker service to ar ] ``` -Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `archive` field has the CAR cid, while the `proof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field MUST be set to the byte size of the CAR file. +Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `archive` field has the CAR CID, while the `proof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field MUST be set to the byte size of the CAR file. Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate). From babefc0e1a2475bc11cbd2b2c3a6fcb32e32263c Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Fri, 21 Apr 2023 15:22:18 +0200 Subject: [PATCH 27/31] chore: new iteration based on thing discussions --- w3-aggregation.md | 65 +++++++++++++++++++++++++++-------------------- 1 file changed, 37 insertions(+), 28 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index e4fd19e..e128ae9 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -76,13 +76,13 @@ sequenceDiagram Once broker successfully gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. It is worth mentioning that if an offer is from an aggregate that is already `queued` or `complete` it is ignored. -This receipt MUST have link to a followup task (using `.fx.join` field) that either succeeds (if aggregate was accepted) or fails (if aggregated was determined to be invalid) so that it's receipt COULD be looked up using it. +This receipt MUST have link to a followup task (using `.fx.join` field) that either succeeds (if aggregate was added into a deal) or fails (if aggregate was determined to be invalid) so that it's receipt COULD be looked up using it. -> Note: Aggregator MAY have several intermediate steps and states it transitions though, however those intentionally are not captured by this protocol, because storefront will take no action until success / failure condition is met. +> Note: Aggregator MAY have several intermediate steps and states it transitions through, however those intentionally are not captured by this protocol, because storefront will take no action until success / failure condition is met. ### Broker reviews and handles the offer -When a broker dequeues the offer from the queue, its details are retrieved from the provided source URL. With the offer details, the broker will interact with available Filecoin Storage Providers, in order to establish a previously determined (out of band) number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. +After a broker dequeues the offer from the queue, it will interact with available Filecoin Storage Providers, in order to establish a previously determined (out of band) number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. ```mermaid sequenceDiagram @@ -95,6 +95,8 @@ sequenceDiagram Once offer gets into `accepted` state, broker takes care of renewing deals. +It is worth pointing out that Broker might request out of bound signature to Storefront to validate the terms of a deal. + ### Storefront can query state of the aggregate deals Storefront users MAY want to check details about deals from the content they previously stored. These deals will change over time as they get renewed. Therefore, Storefront should invoke `aggregate/get` capability to gather information about given aggregate identifier. @@ -124,16 +126,18 @@ type AggregateOffer struct { } type AggregateOfferDetail struct { - offer: OfferCBOR + offer OfferCBOR + commitmentProof Proof } type OfferCBOR any +type Proof any type struct OfferDetails { - size Int - archive Archive - src [URL] - proof Proof + size Int + link Link + src [URL] + commitmentProof Proof } type Offer [OfferDetails] @@ -149,7 +153,8 @@ type Offer [OfferDetails] "with": "did:web:spade.storage", "can": "aggregate/offer", "nb": { - "offer": { "/": "bafy...many-cars" } /* dag-cbor CID */ + "offer": { "/": "bafy...many-cars" }, /* dag-cbor CID */ + "commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */ } }], "prf": [], @@ -160,11 +165,11 @@ type Offer [OfferDetails] Invoking `aggregate/offer` capability submits an offer to a broker service to arrange a Filecoin deals. The `nb.offer` represents a "Ferry" aggregate offer that is ready for a Filecoin deal. Its value is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. This block MUST be included in CAR file that transports the invocation. Its format is: ```json -/* decoded offers block */ +/* decoded offers block as OfferDetails type */ [ { - "archive": { "/": "bag...file0" }, /* CAR CID */ + "link": { "/": "bag...file0" }, /* CAR CID */ "size": 110101, - "proof": { "/": "bag...file0" }, /* COMMP CID */ + "commitmentProof": { "/": "commitment...car0" }, /* COMMP CID */ "src": ["https://w3s.link/ipfs/bag...file0"] }, { @@ -173,7 +178,7 @@ Invoking `aggregate/offer` capability submits an offer to a broker service to ar ] ``` -Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `archive` field has the CAR CID, while the `proof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field MUST be set to the byte size of the CAR file. +Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `link` field has the CAR CID, while the `commitmentProof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field MUST be set to the byte size of the CAR file. Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate). @@ -210,7 +215,7 @@ A Storefront principal can invoke a capability to get state of a previously acce "with": "did:web:web3.storage", "can": "aggregate/get", "nb": { - "link": "bafy...offer", + "commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */ } }], "prf": [], @@ -261,7 +266,7 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip "with": "did:web:spade.storage", "can": "offer/arrange", "nb": { - "link": "bafy...offer", + "commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */ } }], "prf": [], @@ -269,14 +274,14 @@ When a broker receives an `aggregate/offer` invocation from a Storefront Princip } ``` -Once this invocation is executed, a receipt is generated with the status of the task updated. Accepted aggregate receipt will provide aggregate offer operation info: +Once this invocation is executed, a receipt is generated with the result of the task. Accepted aggregate receipt looks like: ```json { "ran": "bafy...arrange", "out": { "ok": { - "link": "bafy...offer" + "commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */ } }, "fx": { @@ -288,16 +293,16 @@ Once this invocation is executed, a receipt is generated with the status of the } ``` -If offered aggregate is invalid, details on failing commPs are also reported: +If offered aggregate is invalid, details on failing commitmentProofs are also reported: ```json { "ran": "bafy...invocation", "out": { "error": { - "link": "bafy...offer", + "commitmentProof": { "/": "commitment...cars-proof" } /* commitment proof */ "cause": [{ - "commP": "commP", + "commitmentProof": { "/": "commitment...car0" }, "reason": "reasonCode", }], }, @@ -333,11 +338,11 @@ type AggregateGet struct { } type SucceedAggregateRef struct { - link &AggregateCBOR + commitmentProof Proof } type AggregateRef struct { - link &AggregateCBOR + commitmentProof Proof } type AggregateOffer struct { @@ -346,23 +351,27 @@ type AggregateOffer struct { } type AggregateOfferDetail struct { - offer: OfferCBOR + offer OfferCBOR + commitmentProof Proof } type OfferCBOR any +type Proof any type struct OfferDetails { - size Int - archive Archive - src [URL] - proof Proof + size Int + link Link + src [URL] + commitmentProof Proof } type Offer [OfferDetails] type StorefrontDID string type URL string -type AggregateCBOR any +type OfferCBOR any +type Proof any + type OfferCapability union { OfferArrange "offer/arrange" } representation inline { From 555d647b6aa34e85be81cd6302c168a6fb124c1a Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Fri, 21 Apr 2023 15:23:19 +0200 Subject: [PATCH 28/31] fix: spell issues --- .github/workflows/words-to-ignore.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index 86d67c3..de513cc 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -100,4 +100,5 @@ capabilty commPs dag-cbor NameDescriptionStorefrontPrincipal -w3upAuthorityPrincipal \ No newline at end of file +w3upAuthorityPrincipal +commitmentProofs \ No newline at end of file From c2169a45f912aa55ffa22b1c0c45ff13c5ff772e Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Thu, 4 May 2023 08:57:24 +0200 Subject: [PATCH 29/31] chore: apply suggestions from code review Co-authored-by: Alan Shaw --- w3-aggregation.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index e128ae9..45a1aca 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -56,11 +56,11 @@ A Storefront is the entry point for user/application data into web3. It will act ### Authorization -Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-bound exchange of information or through a well defined API. In other words, broker can authorize invocations from `did:web:web3.storage` by validating signature from did. This way, it allows web3.storage to rotate keys and/or re-delegate access without having to coordinate with the broker. +Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-band exchange of information or through a well defined API. In other words, a broker can authorize invocations from `did:web:web3.storage` by validating the signature is from the DID. This way, it allows web3.storage to rotate keys and/or re-delegate access without having to coordinate with the broker. ### Storefront offers broker an aggregate -When a Storefront has enough content to fulfill an aggregate (each broker MAY have different requirements), a Filecoin deal for an aggregate MAY be requested by an `aggregate/offer` invocation. Deal negotiations with Filecoin Storage Providers SHOULD be handled out off band. Broker MUST acknowledge a request by issuing a signed receipt. +When a Storefront has enough content to fulfill an aggregate (each broker MAY have different requirements), a Filecoin deal for an aggregate MAY be requested by an `aggregate/offer` invocation. Deal negotiations with Filecoin Storage Providers SHOULD be handled out-of-band. A broker MUST acknowledge a request by issuing a signed receipt. ```mermaid sequenceDiagram @@ -72,17 +72,17 @@ sequenceDiagram Authority-->>Storefront: receipt issued ``` -### Broker queues the offer +### Broker queues the aggregate -Once broker successfully gets an offer, the offer gets queued for review. A receipt is created to proof the transition of `aggregate/offer` state from `null` into `queued`. It is worth mentioning that if an offer is from an aggregate that is already `queued` or `complete` it is ignored. +Once a broker successfully receives the offer of an aggregate, the aggregate gets queued for review. A receipt is created to proof the transition of the offered aggregate state from `null` into `queued`. It is worth mentioning that if an offer is for an aggregate that is already `queued` or `complete` it is ignored. -This receipt MUST have link to a followup task (using `.fx.join` field) that either succeeds (if aggregate was added into a deal) or fails (if aggregate was determined to be invalid) so that it's receipt COULD be looked up using it. +This receipt MUST have link to a followup task (using `.fx.join` field) that either succeeds (if the aggregate was added into a deal) or fails (if the aggregate was determined to be invalid) so that it's receipt COULD be looked up using it. > Note: Aggregator MAY have several intermediate steps and states it transitions through, however those intentionally are not captured by this protocol, because storefront will take no action until success / failure condition is met. -### Broker reviews and handles the offer +### Broker reviews and handles the aggregate -After a broker dequeues the offer from the queue, it will interact with available Filecoin Storage Providers, in order to establish a previously determined (out of band) number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of `aggregate/offer` state from `queued` into `accepted` or `denied`. +After a broker dequeues the aggregate, it will interact with available Filecoin Storage Providers, in order to establish a previously determined (out of band) number of deals. Depending on storage providers availability, as well as the content present in the offer, the aggregate MAY be handled or not. A receipt is created to proof the transition of the aggregate state from `queued` into `accepted` or `rejected`. ```mermaid sequenceDiagram @@ -93,9 +93,9 @@ sequenceDiagram Authority-->>Storefront: receipt issued ``` -Once offer gets into `accepted` state, broker takes care of renewing deals. +If the aggregate reaches the `accepted` state, the broker takes care of renewing deals. -It is worth pointing out that Broker might request out of bound signature to Storefront to validate the terms of a deal. +The broker MAY request an out of bound signature from the Storefront to validate the terms of a deal. ### Storefront can query state of the aggregate deals @@ -117,7 +117,7 @@ In this document, we will be looking at `spade.storage` as an implementer of the ### `aggregate/offer` -A Storefront principal can invoke a capabilty to submit an aggregate ready for offers. +A Storefront principal can invoke a capabilty to offer an aggregate that is ready to be included in Filecoin deal(s). ```iplsch type AggregateOffer struct { @@ -162,11 +162,12 @@ type Offer [OfferDetails] } ``` -Invoking `aggregate/offer` capability submits an offer to a broker service to arrange a Filecoin deals. The `nb.offer` represents a "Ferry" aggregate offer that is ready for a Filecoin deal. Its value is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. This block MUST be included in CAR file that transports the invocation. Its format is: +Invoking `aggregate/offer` capability submits an aggregate to a broker service for inclusion in one or more Filecoin deals. The `nb.offer` field represents a "Ferry" aggregate offer that is ready for a Filecoin deal. Its value is the DAG-CBOR CID that refers to a "Ferry" offer. It encodes a dag-cbor block with an array of entries representing all the CAR files to include in the aggregated deal. This block MUST be included in the CAR file that transports the invocation. Its format is: ```json -/* decoded offers block as OfferDetails type */ -[ { +/* offers block as OfferDetails type, encoded as DAG-JSON (for readability) */ +[ + { "link": { "/": "bag...file0" }, /* CAR CID */ "size": 110101, "commitmentProof": { "/": "commitment...car0" }, /* COMMP CID */ @@ -180,7 +181,7 @@ Invoking `aggregate/offer` capability submits an offer to a broker service to ar Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `link` field has the CAR CID, while the `commitmentProof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field MUST be set to the byte size of the CAR file. -Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with an aggregate). +Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with the aggregate). ```json { From 055603e5997033e654e512bd35802c793f7cee3e Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Thu, 4 May 2023 09:03:07 +0200 Subject: [PATCH 30/31] chore: make src optional --- w3-aggregation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index 45a1aca..df23787 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -179,7 +179,7 @@ Invoking `aggregate/offer` capability submits an aggregate to a broker service f ] ``` -Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `link` field has the CAR CID, while the `commitmentProof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. The `size` field MUST be set to the byte size of the CAR file. +Each entry of the decoded offers block, has all the necessary information for a Storage Provider to fetch and store a CAR file. The `link` field has the CAR CID, while the `commitmentProof` field has the required `proof` bytes by Storage Providers (for example, `commP`). The `size` field MUST be set to the byte size of the CAR file. The `src` field of each piece MUST be set to a (alphabetically sorted) list of URLs from which it can be fetched. Note that `src` field is optional and can be provided in a different part of the flow such as when deal is signed or through a previously agreed API. Broker MUST issue a signed receipt to acknowledge the received request. Issued receipt MUST contain an [effect](https://github.com/ucan-wg/invocation/#7-effect) with a subsequent task (`.fx.join` field) that is run when submitted aggregate is processed and either succeeds (implying that aggregate was accepted and deals will be arranged) or fail (with `error` describing a problem with the aggregate). @@ -362,7 +362,7 @@ type Proof any type struct OfferDetails { size Int link Link - src [URL] + src? [URL] commitmentProof Proof } From 8c561b01146683537a0ea4bc3ac38550fb6331fd Mon Sep 17 00:00:00 2001 From: Vasco Santos Date: Wed, 10 May 2023 10:49:36 +0200 Subject: [PATCH 31/31] chore: apply suggestions from code review Co-authored-by: Alan Shaw --- w3-aggregation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/w3-aggregation.md b/w3-aggregation.md index df23787..8b182e2 100644 --- a/w3-aggregation.md +++ b/w3-aggregation.md @@ -31,7 +31,7 @@ There are several roles in the authorization flow: | Name | Description | | ----------- | ----------- | | Storefront | [Principal] identified by [`did:web`] identifier, representing a storage aggregator like w3up | -| Authority | [Principal] that represents service provider that executes invoked capabilities | +| Authority | [Principal] that represents the service provider that executes invoked capabilities | | Verifier | Component of the [authority] that performs UCAN validation | ### Storefront @@ -56,7 +56,7 @@ A Storefront is the entry point for user/application data into web3. It will act ### Authorization -Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-band exchange of information or through a well defined API. In other words, a broker can authorize invocations from `did:web:web3.storage` by validating the signature is from the DID. This way, it allows web3.storage to rotate keys and/or re-delegate access without having to coordinate with the broker. +Broker MUST have an authorization mechanism for allowed Storefront principals (e.g. web3.storage). Either by out-of-band exchange of information or through a well defined API. For example, a broker can authorize invocations from `did:web:web3.storage` by validating the signature is from the DID. This way, it allows web3.storage to rotate keys and/or re-delegate access without having to coordinate with the broker. ### Storefront offers broker an aggregate