From 100ffb4f99e4e8ef8d3aa809e284388d94956be3 Mon Sep 17 00:00:00 2001 From: anorth <445306+anorth@users.noreply.github.com> Date: Mon, 15 Aug 2022 09:17:03 +1200 Subject: [PATCH 1/7] First draft of FIP for decoupling FIL+ from markets --- FIPS/fip-filplus.md | 470 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 470 insertions(+) create mode 100644 FIPS/fip-filplus.md diff --git a/FIPS/fip-filplus.md b/FIPS/fip-filplus.md new file mode 100644 index 000000000..0d4bea354 --- /dev/null +++ b/FIPS/fip-filplus.md @@ -0,0 +1,470 @@ +--- +fip: +title: De-couple verified registry from markets +author: Alex North (@anorth) +discussions-to: https://github.com/filecoin-project/FIPs/discussions/313 +status: Draft type: Technical category : Core created: 2022-08-12 requires: FIP-0034 +--- + +TODO: +- store allocation ID in deal state. +- Decide whether to include the super-sector representations now, or defer to later +- Find a better (compressed) representation of the piece ranges in a claim +- Discuss how market delegate means no external API changes +- Discuss mechanisms for conditional allocations, negatively priced deals + +## Simple Summary + +TODO + +## Abstract + +TODO + +## Change Motivation + +TODO + +## Specification +**Terminology** +- *Allowance*: data cap provided to a verifier by the root, or to a client by a verifier. +- *Allocation*: data cap allocated by a client to a specific piece of data. +- *Term*: period of time for which a sector/deal/allocation is active or valid. +- *Claim*: a provider's assertion they are storing all or part of an allocation + +### Data cap allocations + +The Filecoin Plus verified registry actor is extended to record allocations of data cap to specific +pieces of data to be stored by a provider. An allocation is independent of any deal with the +built-in storage market (or any future market). + +``` +struct Allocation { + // The verified client which allocated the data cap. + Client: Address + // The provider (miner actor) which may claim the allocation. + Provider: Address + // Identifier of the data to be committed. + Data: CID + // The (padded) size of data. + Size: uint64 + // The minimum duration which the provider must commit to storing the piece to avoid + // early-termination penalties (epochs). + TermMinimum: uint64 + // The maximum period for which a provider can earn quality-adjusted power + // for the piece (epochs). + TermMaximum: uint64 + // The latest epoch by which a provider must commit data before the allocation expires. + Expiration: Epoch + // Whether the provider must commit the entire piece in a single sector. + // This is initially un-used, but supports allocations larger than a full sector in the future. + AllOrNothing: bool +} +``` + +An allocation specifies a range of terms for which the provider may commit the data, between some +minimum and maximum. An allocation's minimum term must be at least as large as its maximum term. + +Due to the implementation of term enforcement (details below), clients should leave a large buffer +between the minimum and maximum term to make the allocation practical for a provider. + +**Parameters** + +- The minimum allowed value for `TermMinimum` is 6 months. +- The maximum allowed value for `TermMaximum` is 5 years (can be increased by a future FIP). + +Allocations are stored in the verified registry actor's state, grouped by client address. Nesting by +client promotes more efficient lookups at scale, and supports per-client iteration for revocation. + +``` +struct State { + // ... Existing verified registry state ... + + // Allocations indexed by client, then by ID. + Allocations: HAMT[Address]AMT[AllocationID]Allocation + + // Sequence number for allocation IDs. + NextAllocationId: AllocationID // uint64 +} +``` + +#### Operations + +An allocation is created by a verified client by transferring data cap tokens (see below) +to the verified registry. The `Allocation` metadata must be provided as transfer metadata. The +verified registry checks the same preconditions as the existing `UseBytes` method. The size of the +allocation must match the number of tokens received. + +An allocation may be revoked after its expiration epoch has passed. When revoked, the data cap +tokens are transferred back to the client. The verified registry provides a new method to process +revocations. + +``` +struct RevokeExpiredAllocationsParams { + AllocationIDs: []AllocationID // empty for "all", or specify a set +} + +// Removes all expired allocations for a client. +// The data cap for cleaned-up allocations is returned to the client. +// Note that un-expired allocations cannot be revoked. +fn RevokeAllocations(params: RevokeExpiredAllocationsParams) + +``` + +Allocations are also removed when claimed by a provider (see below). + +### Data cap claims + +The verified registry actor is further extended to record each storage provider's commitment to a +data cap allocation, called a claim. A claim represents a provider's obligation to store a piece of +data, and corresponding benefit of incentivized storage power. + +``` +struct Claim { + // The provider storing the data (from allocation). + Provider: Address + // The client which allocated the data cap (from allocation). + Client: Address + // Identifier of the data committed (from allocation). + Data: CID + // The (padded) size of data (from allocation). + Size: uint64 + // The minimum period which the provider must commit to storing the piece (from allocation). + TermMinimum: uint64 + // The maximum period for which the provider can earn QA-power for the piece (from allocation). + TermMaximum: uint64 + // The epoch at which the (first chunk of the) piece was committed. + TermStart: uint64 + // The sector(s) in which the data is committed. + // This representation permits a claim to span pieces of data in multiple sectors. + // Ranges are over the allocation data, and are non-overlapping. (0, 0) means all. + // This structure initially un-used (just a single entry) but supports claims larger + // than a single sector. + Sectors [] { + Sector SectorID, + RangeStart uint64, + RangeSize uint64, + } +} +``` + +Claims are stored in the verified registry actor state, grouped by provider address. A claim's ID is +inherited from the allocation that created it. + +``` +struct State { + // ... Existing verified registry state ... + + // Claims by provider ID, then Claim ID. + Claims HAMT[Address]AMT[ClaimID]Claim +} +``` + +A data cap allocation may identify data that is larger than a single sector. +Such an allocation may be satisfied by multiple claims, committed to different sectors, +each for a distinct, non-overlapping range of the allocation's data. +Each sector hosting a range of an allocation gains power individually. +All ranges from a single allocation must be claimed by the same provider. +If a sector hosting one range of an allocation is lost, +the provider will pay the usual termination penalty but can re-seal the range into a new sector +to regain the same power. +This maintains the incentive to host a complete replica of the allocation. +An allocation must be fully satisifed by a _single_ claim if `AllOrNothing` is true. + +An allocation is successfully claimed if at least one range of the allocation +is committed before the allocation expires. +Other ranges may be committed after the expiration epoch. +This provides symmetry with resumption after faults. + +This larger-than-sector allocation capability will a more scalable representation within +the verified registry actor. +Full realisation of this capability will require changes to the +storage miner actor that are beyond the scope of this proposal. + + + +#### Operations +An allocation is claimed when a storage miner actor proves the corresponding piece of data +is committed to a sector. + +``` +struct SectorAllocationClaim { + // The allocation being claimed. + AllocationID: AllocationID + // The piece data committed (either the allocations data, or a part of it). + PieceCID: CID + // The (padded) size of the data committed (up to the allocation's data size). + PieceSize: uint64 + // The offset into the allocation's data the piece committed. + PieceOffset: uint64, + // A proof that the committed PieceCID and PieceSize is included in the allocation's + // data at PieceOffset. + // Only when the PieceCID and PieceSize do not match the allocation's Data and Size. + InclusionProof: []byte + // The sector into which the piece has been committed. + SectorID: uint64, + +} +struct ClaimAllocationsParams { + Sectors: []SectorAllocationClaim +} +struct SectorClaimResult { + // The IDs of allocation claims that succeeded. + SuccessfulClaims: []AllocationID +} +// Called by storage miner actor to claim allocations for data provably committed to storage. +// For each allocation claim, the registry checks that the provided piece CID +// and size match that of the allocation. +fn ClaimAllocations(params: ClaimAllocationsParams) -> ClaimAllocationsResult +``` + +FIXME: distinguish sub-pieces in SectorClaimResult, that share allocation ID + +The record of a claim may be removed by the provider after it has been completed. + +The maximum term for a claim may be increased by the client which originally made the allocation, +up to the network policy maximum. + +A verified client can extend the term for a claim _beyond_ the initial maximum term by +spending new data cap. +The term maximum can be extended up to the network policy maximum beyond the current epoch. +The client extending the claim need not be the one that made the original allocation. +This is similar to issuing a new allocation for the same data, but avoids overheads of re-sealing. + +``` +TODO: spec method to adjust claim terms +``` + + + +### Data cap token +The `Clients` map of addreses to balances is removed from the verified registry state. +The balances of data cap tokens held by verified clients are extracted to a data cap token actor. +This actor represents un-allocated data cap as a fungible token. +Token balances are represented as BigIntegers with 18 decimal places of precision, +a standard which is likely to be adopted by future token standards. +Token quantities are constrained to be whole multiples of `10^18`, +thus matching the existing whole unit semantics. + +The data cap token is generally non-transferable, except to or from the verified registry actor. +The verified registry actor can invoke privileged methods to create and destroy tokens. + +A token standard is expected to support delegation of control of token balances to specified +third-party actors. +The built-in storage market actor is pre-authorized as a delegate for all verified clients. +Thus, the built-in market actor can allocate data cap on clients' behalf. +This mechanism allows this proposal to be impelemented without changing any clients' or +providers' workflows. +It also will support user-programmed market actors dealing in datacap in the future. +The data cap token actor is intended to present a public interface conforming to standards for +fungible token behaviour and an extensible calling convention. +Neither of those standards are pre-requisite for this proposal, however. +The built-in actors will be retroactively updated to support those standards when they settle. + +#### Operations +The `AddVerifiedClient` method is changed to invoke the data cap token actor to +mint new tokens to the client's balance. + +The `UseBytes` method is removed. +To create an allocation, a verified client transfers data cap tokens to the verified registry actor. +The transfer must be accompanied by one or more `Allocation` records which specify +the allocation(s) to create. +The registry may reject the transfer if the allocations are invalid. + +The `RestoreBytes` method is also removed. +Instead, the `RevokeAllocations` method on the verified registry actor transfers any released +data cap tokens back to the client's balance for re-use. + +### Simplified quality-adjusted power +A data cap allocation/claim is valid for a range of commitment terms, between a client-specified +minimum and maximum. + +The actual term for a claim begins at the epoch the data is committed into a sector, +proven either with proof-of-replication or replica-update. +There is no distinct "deal start epoch". +A sector containing a data cap claim is immediately eligible for a power multiplier of 10 +for that fraction of the sector's capacity occupied by the incentivized data. + +The actual term for a claim ends the epoch that the sector containing the data expires, +or that data is replaced. +The term cannot end during a sector's lifetime, except by explicit replacement of data. +If data is replaced, the sector immediately losses the power multiplier for that fraction +of the sector's capacity. +(Replacing non-zero data in a sector is not possible today). + +Aligning the claim's term with the sector's expiration exactly in this way removes the +"spreading-out" of QA power over a sector's life in the current method of computation. +Providers gain 10x power for each byte of verified data exactly while that data is proven. +The miner actor computes a sector’s power as `SectorSize + (9 * VerifiedSize)`. + +#### Pledge +A sector's pledge requirement is proportional to its share of power, just as today. +When the minimum term for a verified piece expires, the pledge requirement does not change. +The provider's commitment to storing the verified piece extends until +the sector's scheduled expiration. +If the sector is terminated early, the termination penalty includes the boosted power. + +If a verified piece is transferred to new sector (see below), +the initial pledge for the new sector is calculated using the network conditions at the time. +The pledge requirement for the old sector is not decreased, +but any subsequent penalty is calculated using the current (reduced) power. + +### Storage miner actor +The built-in miner actor is trusted to enforce the verified and the continual storage of the data. + +The miner actor requires that the sector into which a claimed piece is committed must have a +**remaining lifespan that is sufficient to fulfil the minimum term**. +Thus, when a sector terminates at its scheduled expiration epoch, the claim's term has been met. + +Similarly, the miner actor requires that sector into which the piece is committed must have a +**remaining lifespan no longer than that which would reach the maximum term**. +That is, a sector's commitment expiration epoch must fall between the minimum and maximum terms +of any verified claim it holds. + +The longest practical _minimum_ term for a verified data cap allocation is +the maximum initial sector commitment duration. + +#### Failure and repair +There is **no change to termination penalty** calculations. +If a sector is terminated early (continual faults, or manual termination), +the provider will pay a penalty related to the expected reward earned by the sector (up to some limit). + +The verified registry can support re-commitment of the same data +(even though the built-in storage market actor does not support resumption of a terminated deal). +If a claim’s maximum term has not been reached when a sector terminates, +the provider may commit the same piece into a new sector and regain the quality-adjusted power. +The new sector’s committed lifespan is still constrained to fit within the claim’s term. + +#### Sector extension +A provider cannot extend the scheduled expiration for a sector past the maximum term +of any piece of verified data that it holds. +The provider must forgo the verified power boost when processing such a sector extension. +This does not reduce the sector’s pledge requirement, +but does reduce penalties to be paid on any future fault to reflect the new, lower power. + +This facility to drop a claim can only be exercised: +- for claims that have passed their minimum commitment, +- for non-faulty sectors, and +- in the final 30 days of the sector's currently committed lifespan + +These restrictions prevent a provider using extension as a means of escaping their commitment to +store the claimed piece for the sector’s full originally-committed lifetime. + +### Built-in storage market as a delegate +The workflows described above are independent of any market actor, +and take place through interactions by the client and provider with +the verified registry actor directly. + +However, in order to maintain current client and provider workflows, +the built-in market actor can act as a delegate. + +The built-in storage market actor is pre-authorized as a data cap token delegate for all clients. +When a deal proposal is published with `VerifiedDeal=true`, the market actor transfers +the corresponding data cap tokens from the client to the verified registry actor. +This transfer is accompanied by an `Allocation` record populated with data from the deal. +The built-in market specifies an allocation's minimum term to be equal to the deal duration, +and maximum term to be one year greater than the deal duration. + +``` +let dealAllocation = Allocation { + Client: deal.Client + Provider: deal.Provider + Data: deal.PieceCID + Size: deal.PieceSize + TermMinimum: deal.EndEpoch - deal.StartEpoch + TermMaximum: deal.EndEpoch - deal.StartEpoch + EPOCHS_IN_YEAR + Expiration: deal.StartEpoch + AllOrNothing: true + } +``` + +The market stores the allocation ID in deal state. + +``` +struct DealState { + // Existing state + SectorStartEpoch: int64, + LastUpdatedEpoch: int64, + SlashEpoch: int64, + // New + AllocationID: uint64 +} +``` + +When a verified deal expires without being committed, the market revokes the allocation. + +When a verified deal is activated by the provider, the market actor returns +the allocation ID and piece metadata to the storage miner actor. +The miner actor then invokes `ClaimAllocations` and, if successful, +computes quality adjusted power for the sector according to the piece size and QA power multiplier. + + + +### Migration +TODO! + +### Outline of main algorithms +Describe the common flows. + + +## Design Rationale + + + +Remaining work to fully remove trust in market actor (API changes) + +Future capabilities +- Internal transfer between sectors (requires new miner APIs) +- Transferable claims to new provider (new APIs) + +See https://github.com/filecoin-project/FIPs/discussions/298. + +## Backwards Compatibility + +TODO + +## Test Cases + +TODO + +## Security Considerations + +TODO + +## Incentive Considerations + +TODO + +## Product Considerations + + +TODO: +- make third-party markets possible (after subsequent changes) +- allocations larger than sectors (and towards deals similar), scalable representation +- potential for direct route to FIL+ allocations (realised in next set of changes) + + +The pattern of proxying verified registry interactions through the built-in market actor will +initially raise the gas cost of deals, but this is a transitional state: + +This proposal is a stepping stone toward greater potential gas efficiency of deals. +- For verified data where the client is offering no payment above the verified rewards, + no deal with a market will be necessary at all. + A client need only make an allocation directly with the verified registry. + This allocation is much cheaper to make and maintain than a deal. +- The proposed representations make possible data cap allocations larger than a single sector. + As well as improvements in utility, such allocations will be far more gas efficient than + negotiating a separate deal for each sector-sized unit. +- For deals where additional payment or terms are necessary, + this decoupling opens the potential to develop alternative storage markets, + including ones much more efficient than the built-in one. + We can expect deal costs to decrease over time. + +## Implementation + +Implementation is in progress on the `decouple-fil+` branch of the built-in actors repository: +https://github.com/filecoin-project/builtin-actors/tree/decouple-fil+. + + +## Copyright +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 52e31bb51edd18dfdbe56dd5bfaaf5475449cff8 Mon Sep 17 00:00:00 2001 From: anorth <445306+anorth@users.noreply.github.com> Date: Tue, 16 Aug 2022 14:59:27 +1200 Subject: [PATCH 2/7] Add ExtendClaimTerms and sketch of migration --- FIPS/{fip-filplus.md => fip-0045.md} | 150 +++++++++++++++++---------- README.md | 1 + 2 files changed, 95 insertions(+), 56 deletions(-) rename FIPS/{fip-filplus.md => fip-0045.md} (81%) diff --git a/FIPS/fip-filplus.md b/FIPS/fip-0045.md similarity index 81% rename from FIPS/fip-filplus.md rename to FIPS/fip-0045.md index 0d4bea354..b09a3a8e4 100644 --- a/FIPS/fip-filplus.md +++ b/FIPS/fip-0045.md @@ -1,5 +1,5 @@ --- -fip: +fip: 0045 title: De-couple verified registry from markets author: Alex North (@anorth) discussions-to: https://github.com/filecoin-project/FIPs/discussions/313 @@ -7,11 +7,15 @@ status: Draft type: Technical category : Core created: 2022-08-12 requires: FIP- --- TODO: -- store allocation ID in deal state. -- Decide whether to include the super-sector representations now, or defer to later +- Method for extending a sector, SP provides allocation ids (because market doesn't retain them after deal expiration) +- Spec methods for sector migration on miner, market, and verifreg actors +- Confirm whether to include the super-sector representations now, or defer to later - Find a better (compressed) representation of the piece ranges in a claim - Discuss how market delegate means no external API changes - Discuss mechanisms for conditional allocations, negatively priced deals +- Market method for fetching allocation ids for deals (needed?) +- Spec hook/method for a different client extending the claim by spending data cap (needed?). + ## Simple Summary @@ -56,9 +60,9 @@ struct Allocation { TermMaximum: uint64 // The latest epoch by which a provider must commit data before the allocation expires. Expiration: Epoch - // Whether the provider must commit the entire piece in a single sector. + // Whether the provider can commit the data in multiple pieces. // This is initially un-used, but supports allocations larger than a full sector in the future. - AllOrNothing: bool + AllowRanges: bool } ``` @@ -73,17 +77,18 @@ between the minimum and maximum term to make the allocation practical for a prov - The minimum allowed value for `TermMinimum` is 6 months. - The maximum allowed value for `TermMaximum` is 5 years (can be increased by a future FIP). -Allocations are stored in the verified registry actor's state, grouped by client address. Nesting by -client promotes more efficient lookups at scale, and supports per-client iteration for revocation. +Allocations are stored in the verified registry actor's state, grouped by client address. +Nesting by client promotes more efficient lookups at scale, +and supports per-client iteration for revocation. ``` struct State { // ... Existing verified registry state ... // Allocations indexed by client, then by ID. - Allocations: HAMT[Address]AMT[AllocationID]Allocation + Allocations: HAMT[Address]AMT[AllocationID]Allocation - // Sequence number for allocation IDs. + // Sequence number for allocation IDs. NextAllocationId: AllocationID // uint64 } ``` @@ -91,24 +96,25 @@ struct State { #### Operations An allocation is created by a verified client by transferring data cap tokens (see below) -to the verified registry. The `Allocation` metadata must be provided as transfer metadata. The -verified registry checks the same preconditions as the existing `UseBytes` method. The size of the -allocation must match the number of tokens received. +to the verified registry. +The `Allocation` metadata must be provided as transfer metadata. +The verified registry checks the same preconditions as the existing `UseBytes` method. +The size of the allocation must match the number of tokens received. -An allocation may be revoked after its expiration epoch has passed. When revoked, the data cap -tokens are transferred back to the client. The verified registry provides a new method to process -revocations. +An allocation may be revoked after its expiration epoch has passed. +When revoked, the data cap tokens are transferred back to the client. +The verified registry provides a new method to process revocations, replacing `RestoreBytes`. ``` struct RevokeExpiredAllocationsParams { - AllocationIDs: []AllocationID // empty for "all", or specify a set + Client: Address // client to clean up (need not be the caller) + AllocationIDs: []AllocationID // empty for "all", or specify a set } // Removes all expired allocations for a client. // The data cap for cleaned-up allocations is returned to the client. // Note that un-expired allocations cannot be revoked. fn RevokeAllocations(params: RevokeExpiredAllocationsParams) - ``` Allocations are also removed when claimed by a provider (see below). @@ -122,9 +128,9 @@ data, and corresponding benefit of incentivized storage power. ``` struct Claim { // The provider storing the data (from allocation). - Provider: Address - // The client which allocated the data cap (from allocation). - Client: Address + Provider: Address + // The client which allocated the data cap (from allocation). + Client: Address // Identifier of the data committed (from allocation). Data: CID // The (padded) size of data (from allocation). @@ -133,18 +139,18 @@ struct Claim { TermMinimum: uint64 // The maximum period for which the provider can earn QA-power for the piece (from allocation). TermMaximum: uint64 - // The epoch at which the (first chunk of the) piece was committed. - TermStart: uint64 + // The epoch at which the (first range of the) piece was committed. + TermStart: uint64 // The sector(s) in which the data is committed. // This representation permits a claim to span pieces of data in multiple sectors. // Ranges are over the allocation data, and are non-overlapping. (0, 0) means all. // This structure initially un-used (just a single entry) but supports claims larger // than a single sector. - Sectors [] { - Sector SectorID, - RangeStart uint64, - RangeSize uint64, - } + Sectors [] { + Sector SectorID, + RangeStart uint64, + RangeSize uint64, + } } ``` @@ -155,8 +161,8 @@ inherited from the allocation that created it. struct State { // ... Existing verified registry state ... - // Claims by provider ID, then Claim ID. - Claims HAMT[Address]AMT[ClaimID]Claim + // Claims by provider ID, then Claim ID. + Claims HAMT[Address]AMT[ClaimID]Claim } ``` @@ -169,14 +175,14 @@ If a sector hosting one range of an allocation is lost, the provider will pay the usual termination penalty but can re-seal the range into a new sector to regain the same power. This maintains the incentive to host a complete replica of the allocation. -An allocation must be fully satisifed by a _single_ claim if `AllOrNothing` is true. +An allocation must be fully satisifed by a _single_ claim if `AllowRanges` is false. An allocation is successfully claimed if at least one range of the allocation is committed before the allocation expires. Other ranges may be committed after the expiration epoch. This provides symmetry with resumption after faults. -This larger-than-sector allocation capability will a more scalable representation within +This larger-than-sector allocation capability allow will a more scalable representation within the verified registry actor. Full realisation of this capability will require changes to the storage miner actor that are beyond the scope of this proposal. @@ -190,27 +196,27 @@ is committed to a sector. ``` struct SectorAllocationClaim { // The allocation being claimed. - AllocationID: AllocationID - // The piece data committed (either the allocations data, or a part of it). - PieceCID: CID - // The (padded) size of the data committed (up to the allocation's data size). - PieceSize: uint64 - // The offset into the allocation's data the piece committed. - PieceOffset: uint64, - // A proof that the committed PieceCID and PieceSize is included in the allocation's - // data at PieceOffset. - // Only when the PieceCID and PieceSize do not match the allocation's Data and Size. - InclusionProof: []byte + AllocationID: AllocationID + // The piece data committed (either the allocations data, or a part of it). + PieceCID: CID + // The (padded) size of the data committed (up to the allocation's data size). + PieceSize: uint64 + // The offset into the allocation's data the piece committed. + PieceOffset: uint64, + // A proof that the committed PieceCID and PieceSize is included in the allocation's + // data at PieceOffset. + // Only when the PieceCID and PieceSize do not match the allocation's Data and Size. + InclusionProof: []byte // The sector into which the piece has been committed. - SectorID: uint64, + SectorID: uint64, } struct ClaimAllocationsParams { - Sectors: []SectorAllocationClaim + Sectors: []SectorAllocationClaim } struct SectorClaimResult { - // The IDs of allocation claims that succeeded. - SuccessfulClaims: []AllocationID + // Indices of allocation claims that succeeded. + Successful: []int64 } // Called by storage miner actor to claim allocations for data provably committed to storage. // For each allocation claim, the registry checks that the provided piece CID @@ -218,8 +224,6 @@ struct SectorClaimResult { fn ClaimAllocations(params: ClaimAllocationsParams) -> ClaimAllocationsResult ``` -FIXME: distinguish sub-pieces in SectorClaimResult, that share allocation ID - The record of a claim may be removed by the provider after it has been completed. The maximum term for a claim may be increased by the client which originally made the allocation, @@ -232,10 +236,25 @@ The client extending the claim need not be the one that made the original alloca This is similar to issuing a new allocation for the same data, but avoids overheads of re-sealing. ``` -TODO: spec method to adjust claim terms -``` - +struct ClaimTerm { + Provider: Address // needed to find claim in state + ClaimID: ClaimID + TermMaximum: int64 // duration in epochs +} +struct ExtendClaimTermsParams{ + Claims: []ClaimTerm +} +struct ExtendClaimTermsResult { + // Indices of extensions that succeeded. + Successful: []int64 +} +// Called by allocation client to extend the maximum term of their allocations, +// without consuming data cap. +// The new TermMaximum must be no less than the current one, +// and no greater than the maximum initial term. +fn ExtendClaimTerms(params: ExtendClaimTermsParams) -> ExtendClaimTermsResult +``` ### Data cap token The `Clients` map of addreses to balances is removed from the verified registry state. @@ -283,7 +302,8 @@ The actual term for a claim begins at the epoch the data is committed into a sec proven either with proof-of-replication or replica-update. There is no distinct "deal start epoch". A sector containing a data cap claim is immediately eligible for a power multiplier of 10 -for that fraction of the sector's capacity occupied by the incentivized data. +for that fraction of the sector's capacity occupied by the incentivized data +(subject to the existing requirement to initially prove the sector in Window PoSt). The actual term for a claim ends the epoch that the sector containing the data expires, or that data is replaced. @@ -374,7 +394,7 @@ let dealAllocation = Allocation { TermMinimum: deal.EndEpoch - deal.StartEpoch TermMaximum: deal.EndEpoch - deal.StartEpoch + EPOCHS_IN_YEAR Expiration: deal.StartEpoch - AllOrNothing: true + AllowRanges: false } ``` @@ -391,18 +411,36 @@ struct DealState { } ``` -When a verified deal expires without being committed, the market revokes the allocation. +When a verified deal expires without being committed, the data cap allocation remains. +The market does not revoke the allocation during cron handling. +Revocation is a manual operation that the client must perform. When a verified deal is activated by the provider, the market actor returns the allocation ID and piece metadata to the storage miner actor. The miner actor then invokes `ClaimAllocations` and, if successful, computes quality adjusted power for the sector according to the piece size and QA power multiplier. - - ### Migration TODO! +The power for existing sectors is not changed by this proposal. + +A provider may opt-in to migrate a sector to the new QA power calculation method. +A new method on the storage miner actor: +- Fetches information about each not-yet-expired verified deal from the market +- Calls the verified registry to create a claim for each such deal. + The claim's minimum term is the deal's duration, and maximum term is the sector's current expiration epoch +- Re-calculates power according to the amount of space committed to active claims. + +This will generally result in an increase in power for the sector. +Note that power may be lost for deals that have already expired, +but which weight was previously amortized over the sector lifespan. + +After migration, the client can extend the migrated claims' terms up to the network policy maximum. +After claims are extended, the provider can extend the sector commitment duration to match. +Note that a migrated sector cannot be extended unless the claim terms are extended, +lest it violate a claim's maximum term. + ### Outline of main algorithms Describe the common flows. diff --git a/README.md b/README.md index ad46933bb..398c57656 100644 --- a/README.md +++ b/README.md @@ -82,3 +82,4 @@ This improvement protocol helps achieve that objective for all members of the Fi |[0040](https://github.com/filecoin-project/FIPs/blob/master/FRCs/frc-0040.md) | Boost - Filecoin Storage Deals Market Protocol | FRC |@dirkmc, @nonsense, @jacobheun, @brendalee |Draft | |[0041](https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0041.md) | Forward Compatibility for PreCommit and ReplicaUpdate | FIP |@Kubuxu |Draft | |[0042](https://github.com/filecoin-project/FIPs/blob/master/FRCs/frc-0042.md) | Calling Convention for Hashed Method Name | FRC |@Kubuxu, @anorth |Draft | +|[0045](https://github.com/filecoin-project/FIPs/blob/master/FRCs/fip-0045.md) | De-couple verified registry from markets | FIP |@anorth |Draft | From e7d47d2bc014e8208320bf9fcc7490befabb8d50 Mon Sep 17 00:00:00 2001 From: anorth <445306+anorth@users.noreply.github.com> Date: Mon, 22 Aug 2022 11:18:13 +1200 Subject: [PATCH 3/7] Remove larger-than-sector allocations, and other cleanups --- FIPS/fip-0045.md | 88 +++++++++++++++++++----------------------------- 1 file changed, 34 insertions(+), 54 deletions(-) diff --git a/FIPS/fip-0045.md b/FIPS/fip-0045.md index b09a3a8e4..f29e79071 100644 --- a/FIPS/fip-0045.md +++ b/FIPS/fip-0045.md @@ -8,9 +8,9 @@ status: Draft type: Technical category : Core created: 2022-08-12 requires: FIP- TODO: - Method for extending a sector, SP provides allocation ids (because market doesn't retain them after deal expiration) +- Confirm policy for built-in market actor's default term maximum +- Confirm policy for sector migration any extension to term maximum beyond the sector's lifespan - Spec methods for sector migration on miner, market, and verifreg actors -- Confirm whether to include the super-sector representations now, or defer to later -- Find a better (compressed) representation of the piece ranges in a claim - Discuss how market delegate means no external API changes - Discuss mechanisms for conditional allocations, negatively priced deals - Market method for fetching allocation ids for deals (needed?) @@ -31,10 +31,9 @@ TODO ## Specification **Terminology** -- *Allowance*: data cap provided to a verifier by the root, or to a client by a verifier. - *Allocation*: data cap allocated by a client to a specific piece of data. -- *Term*: period of time for which a sector/deal/allocation is active or valid. - *Claim*: a provider's assertion they are storing all or part of an allocation +- *Term*: period of time for which a data cap allocation or claim is valid or active. ### Data cap allocations @@ -67,7 +66,7 @@ struct Allocation { ``` An allocation specifies a range of terms for which the provider may commit the data, between some -minimum and maximum. An allocation's minimum term must be at least as large as its maximum term. +minimum and maximum. An allocation's maximum term must be at least as large as its minimum term. Due to the implementation of term enforcement (details below), clients should leave a large buffer between the minimum and maximum term to make the allocation practical for a provider. @@ -141,16 +140,8 @@ struct Claim { TermMaximum: uint64 // The epoch at which the (first range of the) piece was committed. TermStart: uint64 - // The sector(s) in which the data is committed. - // This representation permits a claim to span pieces of data in multiple sectors. - // Ranges are over the allocation data, and are non-overlapping. (0, 0) means all. - // This structure initially un-used (just a single entry) but supports claims larger - // than a single sector. - Sectors [] { - Sector SectorID, - RangeStart uint64, - RangeSize uint64, - } + // ID of the provider's sector in which the data is committed. + Sector uint64 } ``` @@ -166,47 +157,20 @@ struct State { } ``` -A data cap allocation may identify data that is larger than a single sector. -Such an allocation may be satisfied by multiple claims, committed to different sectors, -each for a distinct, non-overlapping range of the allocation's data. -Each sector hosting a range of an allocation gains power individually. -All ranges from a single allocation must be claimed by the same provider. -If a sector hosting one range of an allocation is lost, -the provider will pay the usual termination penalty but can re-seal the range into a new sector -to regain the same power. -This maintains the incentive to host a complete replica of the allocation. -An allocation must be fully satisifed by a _single_ claim if `AllowRanges` is false. - -An allocation is successfully claimed if at least one range of the allocation -is committed before the allocation expires. -Other ranges may be committed after the expiration epoch. -This provides symmetry with resumption after faults. - -This larger-than-sector allocation capability allow will a more scalable representation within -the verified registry actor. -Full realisation of this capability will require changes to the -storage miner actor that are beyond the scope of this proposal. - - - #### Operations An allocation is claimed when a storage miner actor proves the corresponding piece of data is committed to a sector. ``` struct SectorAllocationClaim { + // Client of the allocation being claimed. + Client: Address // The allocation being claimed. AllocationID: AllocationID // The piece data committed (either the allocations data, or a part of it). PieceCID: CID // The (padded) size of the data committed (up to the allocation's data size). PieceSize: uint64 - // The offset into the allocation's data the piece committed. - PieceOffset: uint64, - // A proof that the committed PieceCID and PieceSize is included in the allocation's - // data at PieceOffset. - // Only when the PieceCID and PieceSize do not match the allocation's Data and Size. - InclusionProof: []byte // The sector into which the piece has been committed. SectorID: uint64, @@ -215,7 +179,7 @@ struct ClaimAllocationsParams { Sectors: []SectorAllocationClaim } struct SectorClaimResult { - // Indices of allocation claims that succeeded. + // Indices of allocation claims in the parameters that succeeded. Successful: []int64 } // Called by storage miner actor to claim allocations for data provably committed to storage. @@ -224,7 +188,19 @@ struct SectorClaimResult { fn ClaimAllocations(params: ClaimAllocationsParams) -> ClaimAllocationsResult ``` -The record of a claim may be removed by the provider after it has been completed. +The record of a claim may be removed by the provider after the term maximum has elapsed. + +``` +struct RemoveExpiredClaimsParams { + Provider: Address // provider to clean up (need not be the caller) + ClaimIDs: []ClaimID // empty for "all", or specify a set +} + +// Removes all (or specified) expired claims for a provider. +// Claims which have not yet passed their term maximum cannot be removed. +fn RemoveExpiredClaims(params: RemoveExpiredClaimsParams) +``` + The maximum term for a claim may be increased by the client which originally made the allocation, up to the network policy maximum. @@ -261,11 +237,13 @@ The `Clients` map of addreses to balances is removed from the verified registry The balances of data cap tokens held by verified clients are extracted to a data cap token actor. This actor represents un-allocated data cap as a fungible token. Token balances are represented as BigIntegers with 18 decimal places of precision, -a standard which is likely to be adopted by future token standards. -Token quantities are constrained to be whole multiples of `10^18`, -thus matching the existing whole unit semantics. +a representation which is likely to be adopted by future token standards. +Token quantities are constrained to be whole multiples of `10^18`. +One whole data cap token corresponds to one unit of data cap to be allocated, +which today is good for one byte of data. The data cap token is generally non-transferable, except to or from the verified registry actor. +The token actor enforces these transfer restrictions. The verified registry actor can invoke privileged methods to create and destroy tokens. A token standard is expected to support delegation of control of token balances to specified @@ -274,7 +252,7 @@ The built-in storage market actor is pre-authorized as a delegate for all verifi Thus, the built-in market actor can allocate data cap on clients' behalf. This mechanism allows this proposal to be impelemented without changing any clients' or providers' workflows. -It also will support user-programmed market actors dealing in datacap in the future. +It also will support user-programmed market actors dealing in data cap in the future. The data cap token actor is intended to present a public interface conforming to standards for fungible token behaviour and an extensible calling convention. Neither of those standards are pre-requisite for this proposal, however. @@ -318,16 +296,17 @@ Providers gain 10x power for each byte of verified data exactly while that data The miner actor computes a sector’s power as `SectorSize + (9 * VerifiedSize)`. #### Pledge -A sector's pledge requirement is proportional to its share of power, just as today. +A sector's pledge requirement is proportional to its share of power, but never decreases, just as today. When the minimum term for a verified piece expires, the pledge requirement does not change. The provider's commitment to storing the verified piece extends until the sector's scheduled expiration. -If the sector is terminated early, the termination penalty includes the boosted power. +If the sector is terminated early, the termination penalty is based on its boosted power. If a verified piece is transferred to new sector (see below), -the initial pledge for the new sector is calculated using the network conditions at the time. +the new initial pledge for the new sector is calculated using the network conditions at the time, +including the sector's new power after adding the verified piece. The pledge requirement for the old sector is not decreased, -but any subsequent penalty is calculated using the current (reduced) power. +but any subsequent penalty is calculated using its current (i.e. reduced) power. ### Storage miner actor The built-in miner actor is trusted to enforce the verified and the continual storage of the data. @@ -430,6 +409,7 @@ A new method on the storage miner actor: - Fetches information about each not-yet-expired verified deal from the market - Calls the verified registry to create a claim for each such deal. The claim's minimum term is the deal's duration, and maximum term is the sector's current expiration epoch + (TODO: plus some buffer?) - Re-calculates power according to the amount of space committed to active claims. This will generally result in an increase in power for the sector. From ff51c56e7789bb0eb58264d8f2e02a1458a0a7fa Mon Sep 17 00:00:00 2001 From: anorth <445306+anorth@users.noreply.github.com> Date: Wed, 24 Aug 2022 11:23:59 +1200 Subject: [PATCH 4/7] Add front matter. Reference FRC-0046. Spec new extension method. --- FIPS/fip-0045.md | 180 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 143 insertions(+), 37 deletions(-) diff --git a/FIPS/fip-0045.md b/FIPS/fip-0045.md index f29e79071..519dae094 100644 --- a/FIPS/fip-0045.md +++ b/FIPS/fip-0045.md @@ -3,31 +3,82 @@ fip: 0045 title: De-couple verified registry from markets author: Alex North (@anorth) discussions-to: https://github.com/filecoin-project/FIPs/discussions/313 -status: Draft type: Technical category : Core created: 2022-08-12 requires: FIP-0034 +status: Draft +type: Technical +category : Core +created: 2022-08-12 +requires: FIP-0034 --- TODO: -- Method for extending a sector, SP provides allocation ids (because market doesn't retain them after deal expiration) -- Confirm policy for built-in market actor's default term maximum -- Confirm policy for sector migration any extension to term maximum beyond the sector's lifespan +- Spec receiver hook on Account, Multisig actors - Spec methods for sector migration on miner, market, and verifreg actors +- Spec change to SectorOnChainInfo to distinguish migrated sectors +- Discuss changes to deal packing - Discuss how market delegate means no external API changes - Discuss mechanisms for conditional allocations, negatively priced deals +- Confirm policy for built-in market actor's default term maximum +- Confirm policy for sector migration any extension to term maximum beyond the sector's lifespan - Market method for fetching allocation ids for deals (needed?) - Spec hook/method for a different client extending the claim by spending data cap (needed?). ## Simple Summary - -TODO +Represents Filecoin Plus data cap allocations separately from the built-in market actor. +Adds explicit term (duration) limits to data cap allocations. +Simplifies the quality-adjusted power calculation to remove spreading out over a sector's lifetime. +Supports extending the terms of data cap allocations independently of market deals. +Represents un-allocated data cap as a fungible token. +These changes provide a foundation for user-programmed market actors to broker Filecoin Plus verified deals in the future. ## Abstract - -TODO +The current implementation of Filecoin Plus verified deals is deeply coupled to the built-in market actor, +which is treated as trusted code. +The built-in market is inefficient, limited in functionality and can only improve very slowly through network upgrades. +The architectural coupling prevents any user-programmed actors from replicating, improving, or varying deal-related functionality, +especially with any relation to Filecoin Plus. + +This proposal decouples the representation of Filecoin Plus data cap allocations from the built-in market actor's deals, +and the process of claiming it from the built-in market's processes. +The built-in market actor remains a delegate, acting on behalf of verified clients, +but only in a way that will be accessible to user-programmed market actors too. +This removes some of the necessary trust vested in the built-in market actor's code. + +Along the way, this proposal implements the ability to extend data cap terms beyond the market's deal terms. +It also simplifies the quality adjusted power calculation so as to support +the future ability to update the data in a sector regardless of its previous content. +This proposal gets half way toward full support for user-programmed market actors. ## Change Motivation - -TODO +The core motivation behind this proposal is to enable the development of markets and market-like actors +on the FVM, once user-programability is supported. +All negotiation between data clients and storage providers is currently limited by +the built-in market actor's limited abilities, +and improvements to it are thus limited to FIPs and network upgrades. +Features like deal extension, multi-sector deals, re-negotiation, deal transfer, capacity deals, +auction markets, repair markets, insurance, derivatives etc. should all be possible in user-programmed actors. +None should require a network-wide upgrade to realise. + +The built-in market actor currently occupies an exclusive position as the sole broker of +Filecoin Plus data cap, tightly coupled with the verified registry and storage miner actors. +In order to enable the development of useful storage-related applications and features, +we need to break the tight coupling between the built-in actors, expose composable on-chain primitives, +and make the capabilities of the built-in market actor available to user-programmable contracts. +This proposal addresses the capability to broker of Filecoin Plus data cap. + +Breaking coupling between the built-in market actor, verified registry actor, and miner actors +will eventually remove the storage market from consensus-critical risk surface, +and move subsequent market development into the realm of smart-contract upgrades, +rather than network-wide consensus protocol upgrades. +Minimising coupling is a key principle in software and systems engineering. +Decoupling the verified data rewards from storage deals provides great opportunities for +simplifying Filecoin’s core mechanics in order to provide a better platform for development. + +One benefit we can immediately realise is simplifying the calculation of quality-adjusted power. +The current mechanism, which is coupled to deal terms, imposes significant limits on the flexibility of sector storage, +essentially rendering it write-once. +Decoupling these neatly resolves the calculation difficulties to give full flexibility, +and is much simpler than the Filecoin Plus premium proposal that also attempted this. ## Specification **Terminology** @@ -172,15 +223,19 @@ struct SectorAllocationClaim { // The (padded) size of the data committed (up to the allocation's data size). PieceSize: uint64 // The sector into which the piece has been committed. - SectorID: uint64, + SectorID: uint64 + // The epoch during which the sector is scheduled to expire. + SectorExpiration: int64 } struct ClaimAllocationsParams { Sectors: []SectorAllocationClaim } struct SectorClaimResult { - // Indices of allocation claims in the parameters that succeeded. - Successful: []int64 + // Count of successful claims. + SuccessCount: uint64 + // Index and failure code for each unsuccessful claim. + Failures: []{index: uint64, code: ExitCode} } // Called by storage miner actor to claim allocations for data provably committed to storage. // For each allocation claim, the registry checks that the provided piece CID @@ -221,8 +276,10 @@ struct ExtendClaimTermsParams{ Claims: []ClaimTerm } struct ExtendClaimTermsResult { - // Indices of extensions that succeeded. - Successful: []int64 + // Count of successful extensions. + SuccessCount: uint64 + // Index and failure code for each unsuccessful extension. + Failures: []{index: uint64, code: ExitCode} } // Called by allocation client to extend the maximum term of their allocations, @@ -236,27 +293,27 @@ fn ExtendClaimTerms(params: ExtendClaimTermsParams) -> ExtendClaimTermsResult The `Clients` map of addreses to balances is removed from the verified registry state. The balances of data cap tokens held by verified clients are extracted to a data cap token actor. This actor represents un-allocated data cap as a fungible token. +This token actor is intended to implement the FRC-0046 token standard in the future, +but specification of the full, public API is deferred to a subsequent proposal. +The data cap token actor should be retroactively updated to support the standard later. + Token balances are represented as BigIntegers with 18 decimal places of precision, a representation which is likely to be adopted by future token standards. Token quantities are constrained to be whole multiples of `10^18`. One whole data cap token corresponds to one unit of data cap to be allocated, which today is good for one byte of data. - The data cap token is generally non-transferable, except to or from the verified registry actor. The token actor enforces these transfer restrictions. The verified registry actor can invoke privileged methods to create and destroy tokens. -A token standard is expected to support delegation of control of token balances to specified -third-party actors. -The built-in storage market actor is pre-authorized as a delegate for all verified clients. +FRC-0046 supports delegation of control of token balances to specified third-party actors as "operators". +The built-in storage market actor is pre-authorized as an operator for all verified clients. Thus, the built-in market actor can allocate data cap on clients' behalf. -This mechanism allows this proposal to be impelemented without changing any clients' or -providers' workflows. -It also will support user-programmed market actors dealing in data cap in the future. -The data cap token actor is intended to present a public interface conforming to standards for -fungible token behaviour and an extensible calling convention. -Neither of those standards are pre-requisite for this proposal, however. -The built-in actors will be retroactively updated to support those standards when they settle. +This delegation mechanism will support user-programmed market actors dealing in data cap in the future. + +This token delegation mechanism allows this proposal to be implemented without changing any +clients' or providers' workflows for existing functionality. +Parties will continue to interact with the built-in market actor and verified registry actor directly. #### Operations The `AddVerifiedClient` method is changed to invoke the data cap token actor to @@ -274,7 +331,7 @@ data cap tokens back to the client's balance for re-use. ### Simplified quality-adjusted power A data cap allocation/claim is valid for a range of commitment terms, between a client-specified -minimum and maximum. +minimum and maximum, so long as the allocation is claimed before the allocation expires. The actual term for a claim begins at the epoch the data is committed into a sector, proven either with proof-of-replication or replica-update. @@ -347,7 +404,43 @@ This facility to drop a claim can only be exercised: - in the final 30 days of the sector's currently committed lifespan These restrictions prevent a provider using extension as a means of escaping their commitment to -store the claimed piece for the sector’s full originally-committed lifetime. +store the claimed piece for the sector’s full originally-committed lifetime. + +The storage provider operator must provide the IDs of the allocations claimed by the sector when extending. + +``` +// The verified data claims currently supported by a sector, to be maintained or dropped +// while processing extension. +// All claims currently supported by the sector must be provided. +struct SectorClaims { + Sector: uint64, + MaintainClaims: []ClaimID + DropClaims: []ClaimID +} +struct ExpirationExtension2 { + Deadline: uint64, + Partition: uint64, + Sectors: BitField, // IDs of non-FIL+ sectors (must have zero verified deal weight) + SectorsWithClaims: []SectorClaims + NewExpiration: int64, // epoch +} +struct ExtendSectorExpirationParams2 { + Extensions: []ExpirationExtension2, +} +struct ExtendSectorExpirationResult2 { + // Count of successful extensions. + SuccessCount: uint64 + // Index and failure code for each unsuccessful extension. + Failures: []{index: uint64, code: ExitCode} +} + +// Increases the expiration epoch for some sectors, optionally dropping expired claims. +// For each sector, the provided claim IDs must correspond to claims that refer back to the +// sector, and the sum of their size must equal the sector's verified deal weight. +// The new expiration must be prior to the earliest term maximum for any maintained claims. +fn ExtendSectorExpiration2(params: ExtendSectorExpirationParams2) -> ExtendSectorExpirationResult2 + +``` ### Built-in storage market as a delegate The workflows described above are independent of any market actor, @@ -377,16 +470,24 @@ let dealAllocation = Allocation { } ``` -The market stores the allocation ID in deal state. +The market stores allocation IDs for not-yet-activated deals in a new mapping, +and the claim ID for activated deals in deal state. ``` +struct State { + // Remainder unchanged... + + // Allocation IDs for deals that are not yet activated. + PendingDealAllocationIDs CID // HAMT[DealID]AllocationID +} + struct DealState { // Existing state SectorStartEpoch: int64, LastUpdatedEpoch: int64, SlashEpoch: int64, // New - AllocationID: uint64 + ClaimID: uint64 } ``` @@ -405,17 +506,22 @@ TODO! The power for existing sectors is not changed by this proposal. A provider may opt-in to migrate a sector to the new QA power calculation method. +Each active deal is converted to a claim as if an allocation had been created when the deal was made. +The claim's minimum term is the deal's duration, and maximum term is the sector's current expiration epoch +(TODO: plus some buffer?). + +Expired deals cannot be migrated, even though the old QA power calculation method has spread out +their power and reward over the sector's full lifetime. +This will generally result in an increase in power for a sector, +but may result in a decrease in case of expired deals. +This is why the migration is opt-in. + A new method on the storage miner actor: - Fetches information about each not-yet-expired verified deal from the market - Calls the verified registry to create a claim for each such deal. - The claim's minimum term is the deal's duration, and maximum term is the sector's current expiration epoch - (TODO: plus some buffer?) - Re-calculates power according to the amount of space committed to active claims. - -This will generally result in an increase in power for the sector. -Note that power may be lost for deals that have already expired, -but which weight was previously amortized over the sector lifespan. - +- Re-caclulates pledge requirement according to the new power and network conditions (but never decreases) + After migration, the client can extend the migrated claims' terms up to the network policy maximum. After claims are extended, the provider can extend the sector commitment duration to match. Note that a migrated sector cannot be extended unless the claim terms are extended, From d9ee44502d20caa26a4bdd158de2977eb041b73b Mon Sep 17 00:00:00 2001 From: anorth <445306+anorth@users.noreply.github.com> Date: Wed, 24 Aug 2022 13:29:07 +1200 Subject: [PATCH 5/7] Add rationale, product considerations, compatibility --- FIPS/fip-0045.md | 175 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 128 insertions(+), 47 deletions(-) diff --git a/FIPS/fip-0045.md b/FIPS/fip-0045.md index 519dae094..856f8dace 100644 --- a/FIPS/fip-0045.md +++ b/FIPS/fip-0045.md @@ -1,7 +1,7 @@ --- fip: 0045 title: De-couple verified registry from markets -author: Alex North (@anorth) +author: Alex North (@anorth), Zenground0 (@zenground0) discussions-to: https://github.com/filecoin-project/FIPs/discussions/313 status: Draft type: Technical @@ -11,14 +11,9 @@ requires: FIP-0034 --- TODO: -- Spec receiver hook on Account, Multisig actors - Spec methods for sector migration on miner, market, and verifreg actors - Spec change to SectorOnChainInfo to distinguish migrated sectors -- Discuss changes to deal packing -- Discuss how market delegate means no external API changes -- Discuss mechanisms for conditional allocations, negatively priced deals - Confirm policy for built-in market actor's default term maximum -- Confirm policy for sector migration any extension to term maximum beyond the sector's lifespan - Market method for fetching allocation ids for deals (needed?) - Spec hook/method for a different client extending the claim by spending data cap (needed?). @@ -123,7 +118,6 @@ Due to the implementation of term enforcement (details below), clients should le between the minimum and maximum term to make the allocation practical for a provider. **Parameters** - - The minimum allowed value for `TermMinimum` is 6 months. - The maximum allowed value for `TermMaximum` is 5 years (can be increased by a future FIP). @@ -136,7 +130,7 @@ struct State { // ... Existing verified registry state ... // Allocations indexed by client, then by ID. - Allocations: HAMT[Address]AMT[AllocationID]Allocation + Allocations: HAMT[Address]HAMT[AllocationID]Allocation // Sequence number for allocation IDs. NextAllocationId: AllocationID // uint64 @@ -204,7 +198,7 @@ struct State { // ... Existing verified registry state ... // Claims by provider ID, then Claim ID. - Claims HAMT[Address]AMT[ClaimID]Claim + Claims HAMT[Address]HAMT[ClaimID]Claim } ``` @@ -315,6 +309,10 @@ This token delegation mechanism allows this proposal to be implemented without c clients' or providers' workflows for existing functionality. Parties will continue to interact with the built-in market actor and verified registry actor directly. +#### Token receiver hook in account and multisig actors +The built-in account and multisig actors are changed to implement a receiver hook method, as specified by FRC-0046. +This hook always succeeds, without inspecting the payload. + #### Operations The `AddVerifiedClient` method is changed to invoke the data cap token actor to mint new tokens to the client's balance. @@ -455,7 +453,10 @@ When a deal proposal is published with `VerifiedDeal=true`, the market actor tra the corresponding data cap tokens from the client to the verified registry actor. This transfer is accompanied by an `Allocation` record populated with data from the deal. The built-in market specifies an allocation's minimum term to be equal to the deal duration, -and maximum term to be one year greater than the deal duration. +and maximum term to be some amount greater than the deal duration. + +**Parameters** +- The built-in market calculates a term maximum of 1 year greater than a deal's specified duration. ``` let dealAllocation = Allocation { @@ -501,14 +502,13 @@ The miner actor then invokes `ClaimAllocations` and, if successful, computes quality adjusted power for the sector according to the piece size and QA power multiplier. ### Migration -TODO! - The power for existing sectors is not changed by this proposal. -A provider may opt-in to migrate a sector to the new QA power calculation method. +A provider may opt in to migrate a sector to the new QA power calculation method by invoking a new method. Each active deal is converted to a claim as if an allocation had been created when the deal was made. -The claim's minimum term is the deal's duration, and maximum term is the sector's current expiration epoch -(TODO: plus some buffer?). +The claim's minimum term is the deal's duration, and maximum term is the greater of either +the sector's current expiration epoch, or 540 days beyond the current epoch +(since the provider could extend the sector to such lifetime prior to migrating). Expired deals cannot be migrated, even though the old QA power calculation method has spread out their power and reward over the sector's full lifetime. @@ -527,68 +527,149 @@ After claims are extended, the provider can extend the sector commitment duratio Note that a migrated sector cannot be extended unless the claim terms are extended, lest it violate a claim's maximum term. -### Outline of main algorithms -Describe the common flows. - +TODO: specify migration method and changes to SectorOnChainInfo ## Design Rationale +The design rationale is mostly provided by the proposal's motivation. +See also https://github.com/filecoin-project/FIPs/discussions/298. - +The rationale behind the new flows is to remove system trust in the built-in market actors. +This proposal removes trust of the verified registry in the built-in market, +by moving network-critical state into the verified registry actor state directly. -Remaining work to fully remove trust in market actor (API changes) - -Future capabilities -- Internal transfer between sectors (requires new miner APIs) -- Transferable claims to new provider (new APIs) - -See https://github.com/filecoin-project/FIPs/discussions/298. +Data cap is presented as a fungible token in order to support greater programmability +in the future, such as through automated notary actors. +Many tools and libraries will emerge to support fungible tokens generally, +and conforming to the same standard will allow data cap to enjoy their transparent support. ## Backwards Compatibility - -TODO +This proposal changes the behaviour and internal APIs of several built-in actors, and thus requires a network upgrade. +The proposal also changes state schema of several built-in actors, which requires a state migration at that upgrade. +Off-chain tools that directly inspect actor state may need to change to handle the post-migration schema. + +This proposal leaves all existing externally-invoked actor methods intact and functional. +This is acheived through the built-in market actor acting as a delegate for data cap, +without needing to be trusted by the verified registry. +Deal clients and providers are not expected to need to change their operations. + +Sectors containing verified data claims that are either committed after this upgrade, +and sectors which are migrated to verified data claims, +can only have their lifetimes extended via the new `ExtendSectorExpiration2` method specified by this proposal. +Storage providers must invoke this new method in order to extend such sectors. +Sectors with no verified claims, including existing sectors which are not migrated, +can continue to use the existing `ExtendSectorExpiration` method. ## Test Cases - -TODO +To be provided with implementation, prior to acceptance. ## Security Considerations TODO +- Mention circuit breaker possibility + ## Incentive Considerations TODO +- SP preference for longer term maximums, shorter terms may be harder to fill + ## Product Considerations - +Most of the product considerations of this proposal are directed towards the future product possibilities +toward which this proposal works. -TODO: -- make third-party markets possible (after subsequent changes) -- allocations larger than sectors (and towards deals similar), scalable representation -- potential for direct route to FIL+ allocations (realised in next set of changes) +TODO: Future capabilities +- Internal transfer between sectors (requires new miner APIs) +### Preparing for new market actors +This proposal breaks the trust relationship between the verified registry actor (a core network component), +and the built-in market actor. +This is a step towards removing all network trust in market actors, at which point we can +support user-programmed market-like actors on the FVM. +Such markets could implement a wide range of marketplace models and rules, and be significantly more efficient. +The remaining step toward this goal is to remove the miner actor's trust of the built-in market actor. + +After this trust is fully removed, the built-in market actor could transition to an alternative governance model, +decoupled from network upgrades. + +### Bypassing markets +After the built-in market actor is decoupled from the miner actor, +a client will be able to allocate data cap to a piece of data directly, +without necessarily using the built-in or any market actor. +Such an allocation would represent the simplest, unconditional deal for verified data possible. +This direct route would likely be far more gas-efficient than the built-in market actor. + +### Markets as delegates +The "market as datacap delegate" pattern enables market actors to implement arbitrary deal logic, +and make the allocation of FIL+ data cap conditional upon the provider's acceptance of deal terms. +For example, a market could specify a negative deal price (i.e. the provider pays the client), +and allocte the data cap only after the provider has escrowed the necessary payment. +Or a market could require simultaneous acceptance of a retrieval assurance policy as part of deal terms, +again withholding the datacap allocation until on-chain conditions are met. +Filecoin Plus verified clients will be able to choose markets that implement policies they want, +which may extend far beyond what's reasonable to implement in the built-in market actor. + +### Multi-sector allocations +The verified registry actor and its allocations and claims are not coupled to sector size. +A future iteration to this schema could support allocations which are larger than a single sector, +and may be claimed by a collection of data commitments spread across multiple sectors. + +Such an allocation would be far more gas efficient than the multiple per-sector deals +required for a large data set today. + +### Support for FIP-0036 +Take alone, FIP-0036 could introduce an unfortunate disadvantage to storage providers with many verified deals, +because such deals cannot be extended by the built-in market in order to enjoy higher duration multipliers. + +This proposal introduces two possible mechanisms to extend these existing deals. +If a storage provider migrates a sector with verified deals to the proposed verified allocations, then either: +- the deal client can extend the claim maximum term up to 5 years since it was committed, at no additional data cap +- a different verified client can extend any claim up to 5 years from the current epoch, by spending additional data cap + +These mechanisms could restore equality of access to duration multipliers for existing sectors, +but depend on client assent and intervention. + +### Gas costs The pattern of proxying verified registry interactions through the built-in market actor will -initially raise the gas cost of deals, but this is a transitional state: +initially raise the gas cost of publishing deals, and of committing a sector with verified claims, +but this is a transitional state. +This proposal is a stepping stone toward much greater potential for gas efficiency after +the final decoupling from the miner actor. -This proposal is a stepping stone toward greater potential gas efficiency of deals. -- For verified data where the client is offering no payment above the verified rewards, +- For simple, unconditional verified data allocations, with no additional payment, no deal with a market will be necessary at all. A client need only make an allocation directly with the verified registry. - This allocation is much cheaper to make and maintain than a deal. -- The proposed representations make possible data cap allocations larger than a single sector. - As well as improvements in utility, such allocations will be far more gas efficient than + This direct allocation will be much cheaper to make and maintain than a deal. +- Upgrades to the claim schema could support data cap allocations larger than a single sector. + As well as improvements in utility, such allocations would be far more gas efficient than negotiating a separate deal for each sector-sized unit. -- For deals where additional payment or terms are necessary, - this decoupling opens the potential to develop alternative storage markets, - including ones much more efficient than the built-in one. - We can expect deal costs to decrease over time. +- Decoupling FIL+ from markets opens the potential to develop alternative storage markets, + including those far more efficient than the built-in one. + +By opening the field of competition for market-like actors, we can expect deal costs to decrease over time, +while the network enforces just enough state to administer the quality-adjusted power of verified data. + +### Deal packing +Storage providers choose how to arrange deals into sectors, subject to the constraints on size and start epoch. +They may optimise for various goals, such as filling space, or space-time, or latency to commit a sector. +This proposal introduces a new constraint of an allocation's maximum term, +and removes a possible optimisation target of space-time. + +Each allocation constrains the sector's commitment duration to lie between its minimum and maximum term. +Two data cap allocations may only be sealed into the same sector if there is some overlap between +the ranges specified by their minimum and maximum term. +Storage providers' deal packing algorithms must be updated to respect these constraints. + +Prior to this change, colocating deals with very different durations is merely an inefficient use of space-time, +which diluted and delayed a provider's power and rewards. +After this change, some allocation will be incompatible, but every compatible collection of allocations +will then grant maximal per-byte quality-adjusted power for the sector's full lifetime. + ## Implementation - Implementation is in progress on the `decouple-fil+` branch of the built-in actors repository: https://github.com/filecoin-project/builtin-actors/tree/decouple-fil+. - ## Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 9c4e88dc999c2d1b16198e8bd71b1024d896afc9 Mon Sep 17 00:00:00 2001 From: anorth <445306+anorth@users.noreply.github.com> Date: Fri, 26 Aug 2022 11:19:55 +1200 Subject: [PATCH 6/7] Review responses --- FIPS/fip-0045.md | 148 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 66 deletions(-) diff --git a/FIPS/fip-0045.md b/FIPS/fip-0045.md index 856f8dace..8f30591b6 100644 --- a/FIPS/fip-0045.md +++ b/FIPS/fip-0045.md @@ -14,16 +14,16 @@ TODO: - Spec methods for sector migration on miner, market, and verifreg actors - Spec change to SectorOnChainInfo to distinguish migrated sectors - Confirm policy for built-in market actor's default term maximum -- Market method for fetching allocation ids for deals (needed?) -- Spec hook/method for a different client extending the claim by spending data cap (needed?). +- Market method for fetching allocation/claim ids for deals (needed?) +- Spec hook/method for a different client extending the claim by spending DataCap (needed?). ## Simple Summary -Represents Filecoin Plus data cap allocations separately from the built-in market actor. -Adds explicit term (duration) limits to data cap allocations. -Simplifies the quality-adjusted power calculation to remove spreading out over a sector's lifetime. -Supports extending the terms of data cap allocations independently of market deals. -Represents un-allocated data cap as a fungible token. +Represents [Filecoin Plus](https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0003.md) DataCap allocations separately from the built-in market actor. +Adds explicit term (duration) limits to DataCap allocations. +Simplifies the quality-adjusted power calculation to remove power spreading out over a sector's lifetime. +Supports extending the terms of DataCap allocations independently of market storage deals. +Represents DataCap as a fungible token. These changes provide a foundation for user-programmed market actors to broker Filecoin Plus verified deals in the future. ## Abstract @@ -33,13 +33,13 @@ The built-in market is inefficient, limited in functionality and can only improv The architectural coupling prevents any user-programmed actors from replicating, improving, or varying deal-related functionality, especially with any relation to Filecoin Plus. -This proposal decouples the representation of Filecoin Plus data cap allocations from the built-in market actor's deals, +This proposal decouples the representation of Filecoin Plus DataCap allocations from the built-in market actor's deals, and the process of claiming it from the built-in market's processes. The built-in market actor remains a delegate, acting on behalf of verified clients, but only in a way that will be accessible to user-programmed market actors too. This removes some of the necessary trust vested in the built-in market actor's code. -Along the way, this proposal implements the ability to extend data cap terms beyond the market's deal terms. +Along the way, this proposal implements the ability to extend DataCap terms beyond the market's deal terms. It also simplifies the quality adjusted power calculation so as to support the future ability to update the data in a sector regardless of its previous content. This proposal gets half way toward full support for user-programmed market actors. @@ -55,11 +55,11 @@ auction markets, repair markets, insurance, derivatives etc. should all be possi None should require a network-wide upgrade to realise. The built-in market actor currently occupies an exclusive position as the sole broker of -Filecoin Plus data cap, tightly coupled with the verified registry and storage miner actors. +Filecoin Plus DataCap, tightly coupled with the verified registry and storage miner actors. In order to enable the development of useful storage-related applications and features, we need to break the tight coupling between the built-in actors, expose composable on-chain primitives, and make the capabilities of the built-in market actor available to user-programmable contracts. -This proposal addresses the capability to broker of Filecoin Plus data cap. +This proposal addresses the capability to broker of Filecoin Plus DataCap. Breaking coupling between the built-in market actor, verified registry actor, and miner actors will eventually remove the storage market from consensus-critical risk surface, @@ -77,19 +77,19 @@ and is much simpler than the Filecoin Plus premium proposal that also attempted ## Specification **Terminology** -- *Allocation*: data cap allocated by a client to a specific piece of data. +- *Allocation*: DataCap allocated by a client to a specific piece of data. - *Claim*: a provider's assertion they are storing all or part of an allocation -- *Term*: period of time for which a data cap allocation or claim is valid or active. +- *Term*: period of time for which a DataCap allocation or claim is valid or active. ### Data cap allocations -The Filecoin Plus verified registry actor is extended to record allocations of data cap to specific +The Filecoin Plus verified registry actor is extended to record allocations of DataCap to specific pieces of data to be stored by a provider. An allocation is independent of any deal with the built-in storage market (or any future market). ``` struct Allocation { - // The verified client which allocated the data cap. + // The verified client which allocated the DataCap. Client: Address // The provider (miner actor) which may claim the allocation. Provider: Address @@ -139,41 +139,45 @@ struct State { #### Operations -An allocation is created by a verified client by transferring data cap tokens (see below) +An allocation is created by a verified client by transferring DataCap tokens (see below) to the verified registry. The `Allocation` metadata must be provided as transfer metadata. The verified registry checks the same preconditions as the existing `UseBytes` method. The size of the allocation must match the number of tokens received. -An allocation may be revoked after its expiration epoch has passed. -When revoked, the data cap tokens are transferred back to the client. -The verified registry provides a new method to process revocations, replacing `RestoreBytes`. +An allocation may be removed after its expiration epoch has passed (by anyone). +When removed, the DataCap tokens are transferred back to the client. +The verified registry provides a new method to process removals, replacing `RestoreBytes`. ``` -struct RevokeExpiredAllocationsParams { +struct RemoveExpiredAllocationsParams { Client: Address // client to clean up (need not be the caller) AllocationIDs: []AllocationID // empty for "all", or specify a set } // Removes all expired allocations for a client. -// The data cap for cleaned-up allocations is returned to the client. -// Note that un-expired allocations cannot be revoked. -fn RevokeAllocations(params: RevokeExpiredAllocationsParams) +// The DataCap for cleaned-up allocations is returned to the client. +// Note that un-expired allocations cannot be removed. +fn RemoveExpiredAllocations(params: RemoveExpiredAllocationsParams) ``` Allocations are also removed when claimed by a provider (see below). +The `RemoveVerifiedClientDataCap` method invokes a privileged method of the DataCap token actor +to burn DataCap tokens held on a client's balance. +Only the verified registry actor is permitted to invoke this method. + ### Data cap claims The verified registry actor is further extended to record each storage provider's commitment to a -data cap allocation, called a claim. A claim represents a provider's obligation to store a piece of +DataCap allocation, called a claim. A claim represents a provider's obligation to store a piece of data, and corresponding benefit of incentivized storage power. ``` struct Claim { // The provider storing the data (from allocation). Provider: Address - // The client which allocated the data cap (from allocation). + // The client which allocated the DataCap (from allocation). Client: Address // Identifier of the data committed (from allocation). Data: CID @@ -255,7 +259,7 @@ The maximum term for a claim may be increased by the client which originally mad up to the network policy maximum. A verified client can extend the term for a claim _beyond_ the initial maximum term by -spending new data cap. +spending new DataCap. The term maximum can be extended up to the network policy maximum beyond the current epoch. The client extending the claim need not be the one that made the original allocation. This is similar to issuing a new allocation for the same data, but avoids overheads of re-sealing. @@ -277,33 +281,33 @@ struct ExtendClaimTermsResult { } // Called by allocation client to extend the maximum term of their allocations, -// without consuming data cap. +// without consuming DataCap. // The new TermMaximum must be no less than the current one, // and no greater than the maximum initial term. fn ExtendClaimTerms(params: ExtendClaimTermsParams) -> ExtendClaimTermsResult ``` ### Data cap token -The `Clients` map of addreses to balances is removed from the verified registry state. -The balances of data cap tokens held by verified clients are extracted to a data cap token actor. -This actor represents un-allocated data cap as a fungible token. +The `VerifiedClients` map of addreses to balances is removed from the verified registry state. +The balances of DataCap tokens held by verified clients are extracted to a DataCap token actor. +This actor represents un-allocated DataCap as a fungible token. This token actor is intended to implement the FRC-0046 token standard in the future, but specification of the full, public API is deferred to a subsequent proposal. -The data cap token actor should be retroactively updated to support the standard later. +The DataCap token actor should be retroactively updated to support the standard later. Token balances are represented as BigIntegers with 18 decimal places of precision, a representation which is likely to be adopted by future token standards. Token quantities are constrained to be whole multiples of `10^18`. -One whole data cap token corresponds to one unit of data cap to be allocated, +One whole DataCap token corresponds to one unit of DataCap to be allocated, which today is good for one byte of data. -The data cap token is generally non-transferable, except to or from the verified registry actor. +The DataCap token is generally non-transferable, except to or from the verified registry actor. The token actor enforces these transfer restrictions. The verified registry actor can invoke privileged methods to create and destroy tokens. FRC-0046 supports delegation of control of token balances to specified third-party actors as "operators". The built-in storage market actor is pre-authorized as an operator for all verified clients. -Thus, the built-in market actor can allocate data cap on clients' behalf. -This delegation mechanism will support user-programmed market actors dealing in data cap in the future. +Thus, the built-in market actor can allocate DataCap on clients' behalf. +This delegation mechanism will support user-programmed market actors dealing in DataCap in the future. This token delegation mechanism allows this proposal to be implemented without changing any clients' or providers' workflows for existing functionality. @@ -314,34 +318,34 @@ The built-in account and multisig actors are changed to implement a receiver hoo This hook always succeeds, without inspecting the payload. #### Operations -The `AddVerifiedClient` method is changed to invoke the data cap token actor to +The `AddVerifiedClient` method is changed to invoke the DataCap token actor to mint new tokens to the client's balance. The `UseBytes` method is removed. -To create an allocation, a verified client transfers data cap tokens to the verified registry actor. +To create an allocation, a verified client transfers DataCap tokens to the verified registry actor. The transfer must be accompanied by one or more `Allocation` records which specify the allocation(s) to create. The registry may reject the transfer if the allocations are invalid. The `RestoreBytes` method is also removed. Instead, the `RevokeAllocations` method on the verified registry actor transfers any released -data cap tokens back to the client's balance for re-use. +DataCap tokens back to the client's balance for re-use. ### Simplified quality-adjusted power -A data cap allocation/claim is valid for a range of commitment terms, between a client-specified +A DataCap allocation/claim is valid for a range of commitment terms, between a client-specified minimum and maximum, so long as the allocation is claimed before the allocation expires. The actual term for a claim begins at the epoch the data is committed into a sector, proven either with proof-of-replication or replica-update. There is no distinct "deal start epoch". -A sector containing a data cap claim is immediately eligible for a power multiplier of 10 +A sector containing a DataCap claim is immediately eligible for a power multiplier of 10 for that fraction of the sector's capacity occupied by the incentivized data (subject to the existing requirement to initially prove the sector in Window PoSt). The actual term for a claim ends the epoch that the sector containing the data expires, or that data is replaced. The term cannot end during a sector's lifetime, except by explicit replacement of data. -If data is replaced, the sector immediately losses the power multiplier for that fraction +If data is replaced, the sector immediately loses the power multiplier for that fraction of the sector's capacity. (Replacing non-zero data in a sector is not possible today). @@ -375,8 +379,9 @@ Similarly, the miner actor requires that sector into which the piece is committe That is, a sector's commitment expiration epoch must fall between the minimum and maximum terms of any verified claim it holds. -The longest practical _minimum_ term for a verified data cap allocation is -the maximum initial sector commitment duration. +The longest practical _minimum_ term for a verified DataCap allocation is the larger of +the maximum initial sector commitment duration, and maximum sector extension duration +(which today are identical). #### Failure and repair There is **no change to termination penalty** calculations. @@ -390,14 +395,16 @@ the provider may commit the same piece into a new sector and regain the quality- The new sector’s committed lifespan is still constrained to fit within the claim’s term. #### Sector extension -A provider cannot extend the scheduled expiration for a sector past the maximum term -of any piece of verified data that it holds. -The provider must forgo the verified power boost when processing such a sector extension. -This does not reduce the sector’s pledge requirement, -but does reduce penalties to be paid on any future fault to reflect the new, lower power. - -This facility to drop a claim can only be exercised: -- for claims that have passed their minimum commitment, +A sector's scheduled commitment term cannot be extended beyond the maximum term of any +verified claim attributed to it. +Thus, to extend a sector, a provider must simultaneously drop any verified claims with +terms that would be exceeded by the extended commitment. +Dropping a verified claim reduces the sector's power accordingly, as well as +reducing penalties to be paid on any future fault to reflect the new, lower power. +Dropping a claim does not reduce the sector’s pledge requirement. + +This facility to drop a claim while extending a sector's commitment can only be exercised: +- for claims that have passed their minimum term, - for non-faulty sectors, and - in the final 30 days of the sector's currently committed lifespan @@ -440,6 +447,15 @@ fn ExtendSectorExpiration2(params: ExtendSectorExpirationParams2) -> ExtendSecto ``` +#### Sector termination +This proposal makes no changes to sector termination. +When a sector terminates at the end of its scheduled commitment, no specific action is necessary. +When a sector terminates before its scheduled commitment is complete, +the provider pays penalties according to the sector's power at that time. + +The verified registry actor will continue to hold `Claim` records for terminated sectors +until the provider removes them with `RemoveExpiredClaims`. + ### Built-in storage market as a delegate The workflows described above are independent of any market actor, and take place through interactions by the client and provider with @@ -448,9 +464,9 @@ the verified registry actor directly. However, in order to maintain current client and provider workflows, the built-in market actor can act as a delegate. -The built-in storage market actor is pre-authorized as a data cap token delegate for all clients. +The built-in storage market actor is pre-authorized as a DataCap token delegate for all clients. When a deal proposal is published with `VerifiedDeal=true`, the market actor transfers -the corresponding data cap tokens from the client to the verified registry actor. +the corresponding DataCap tokens from the client to the verified registry actor. This transfer is accompanied by an `Allocation` record populated with data from the deal. The built-in market specifies an allocation's minimum term to be equal to the deal duration, and maximum term to be some amount greater than the deal duration. @@ -492,7 +508,7 @@ struct DealState { } ``` -When a verified deal expires without being committed, the data cap allocation remains. +When a verified deal expires without being committed, the DataCap allocation remains. The market does not revoke the allocation during cron handling. Revocation is a manual operation that the client must perform. @@ -540,7 +556,7 @@ by moving network-critical state into the verified registry actor state directly Data cap is presented as a fungible token in order to support greater programmability in the future, such as through automated notary actors. Many tools and libraries will emerge to support fungible tokens generally, -and conforming to the same standard will allow data cap to enjoy their transparent support. +and conforming to the same standard will allow DataCap to enjoy their transparent support. ## Backwards Compatibility This proposal changes the behaviour and internal APIs of several built-in actors, and thus requires a network upgrade. @@ -548,7 +564,7 @@ The proposal also changes state schema of several built-in actors, which require Off-chain tools that directly inspect actor state may need to change to handle the post-migration schema. This proposal leaves all existing externally-invoked actor methods intact and functional. -This is acheived through the built-in market actor acting as a delegate for data cap, +This is acheived through the built-in market actor acting as a delegate for DataCap, without needing to be trusted by the verified registry. Deal clients and providers are not expected to need to change their operations. @@ -595,18 +611,18 @@ decoupled from network upgrades. ### Bypassing markets After the built-in market actor is decoupled from the miner actor, -a client will be able to allocate data cap to a piece of data directly, +a client will be able to allocate DataCap to a piece of data directly, without necessarily using the built-in or any market actor. Such an allocation would represent the simplest, unconditional deal for verified data possible. This direct route would likely be far more gas-efficient than the built-in market actor. ### Markets as delegates -The "market as datacap delegate" pattern enables market actors to implement arbitrary deal logic, -and make the allocation of FIL+ data cap conditional upon the provider's acceptance of deal terms. +The "market as DataCap delegate" pattern enables market actors to implement arbitrary deal logic, +and make the allocation of FIL+ DataCap conditional upon the provider's acceptance of deal terms. For example, a market could specify a negative deal price (i.e. the provider pays the client), -and allocte the data cap only after the provider has escrowed the necessary payment. +and allocte the DataCap only after the provider has escrowed the necessary payment. Or a market could require simultaneous acceptance of a retrieval assurance policy as part of deal terms, -again withholding the datacap allocation until on-chain conditions are met. +again withholding the DataCap allocation until on-chain conditions are met. Filecoin Plus verified clients will be able to choose markets that implement policies they want, which may extend far beyond what's reasonable to implement in the built-in market actor. @@ -619,13 +635,13 @@ Such an allocation would be far more gas efficient than the multiple per-sector required for a large data set today. ### Support for FIP-0036 -Take alone, FIP-0036 could introduce an unfortunate disadvantage to storage providers with many verified deals, +Taken alone, FIP-0036 could introduce an unfortunate disadvantage to storage providers with many verified deals, because such deals cannot be extended by the built-in market in order to enjoy higher duration multipliers. This proposal introduces two possible mechanisms to extend these existing deals. If a storage provider migrates a sector with verified deals to the proposed verified allocations, then either: -- the deal client can extend the claim maximum term up to 5 years since it was committed, at no additional data cap -- a different verified client can extend any claim up to 5 years from the current epoch, by spending additional data cap +- the deal client can extend the claim maximum term up to 5 years since it was committed, at no additional DataCap +- a different verified client can extend any claim up to 5 years from the current epoch, by spending additional DataCap These mechanisms could restore equality of access to duration multipliers for existing sectors, but depend on client assent and intervention. @@ -641,7 +657,7 @@ the final decoupling from the miner actor. no deal with a market will be necessary at all. A client need only make an allocation directly with the verified registry. This direct allocation will be much cheaper to make and maintain than a deal. -- Upgrades to the claim schema could support data cap allocations larger than a single sector. +- Upgrades to the claim schema could support DataCap allocations larger than a single sector. As well as improvements in utility, such allocations would be far more gas efficient than negotiating a separate deal for each sector-sized unit. - Decoupling FIL+ from markets opens the potential to develop alternative storage markets, @@ -657,7 +673,7 @@ This proposal introduces a new constraint of an allocation's maximum term, and removes a possible optimisation target of space-time. Each allocation constrains the sector's commitment duration to lie between its minimum and maximum term. -Two data cap allocations may only be sealed into the same sector if there is some overlap between +Two DataCap allocations may only be sealed into the same sector if there is some overlap between the ranges specified by their minimum and maximum term. Storage providers' deal packing algorithms must be updated to respect these constraints. From 5e914b065d2fd6a51b9787b488636280a8bddf6f Mon Sep 17 00:00:00 2001 From: anorth <445306+anorth@users.noreply.github.com> Date: Mon, 29 Aug 2022 15:34:31 +1200 Subject: [PATCH 7/7] Migration notes --- FIPS/fip-0045.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/FIPS/fip-0045.md b/FIPS/fip-0045.md index 8f30591b6..4c88093a0 100644 --- a/FIPS/fip-0045.md +++ b/FIPS/fip-0045.md @@ -11,8 +11,10 @@ requires: FIP-0034 --- TODO: +- Re-spec ExtendClaimTerms as a token receiver hook value rather than explicit method - Spec methods for sector migration on miner, market, and verifreg actors - Spec change to SectorOnChainInfo to distinguish migrated sectors +- Figure out best migration plan for pending deals, pre-committed sectors - Confirm policy for built-in market actor's default term maximum - Market method for fetching allocation/claim ids for deals (needed?) - Spec hook/method for a different client extending the claim by spending DataCap (needed?). @@ -209,6 +211,8 @@ struct State { #### Operations An allocation is claimed when a storage miner actor proves the corresponding piece of data is committed to a sector. +Upon converting an allocation to a claim, the verified registry actor burns the associated +data cap tokens from its balance. ``` struct SectorAllocationClaim { @@ -262,7 +266,10 @@ A verified client can extend the term for a claim _beyond_ the initial maximum t spending new DataCap. The term maximum can be extended up to the network policy maximum beyond the current epoch. The client extending the claim need not be the one that made the original allocation. -This is similar to issuing a new allocation for the same data, but avoids overheads of re-sealing. +This is similar to issuing a new allocation/claim for the same data, +but avoids overheads of re-sealing. + +TODO: rework this to be a token receiver hook rather than explicit method ``` struct ClaimTerm { @@ -388,11 +395,12 @@ There is **no change to termination penalty** calculations. If a sector is terminated early (continual faults, or manual termination), the provider will pay a penalty related to the expected reward earned by the sector (up to some limit). -The verified registry can support re-commitment of the same data +The verified registry can support re-commitment of the same data in a new sector (even though the built-in storage market actor does not support resumption of a terminated deal). If a claim’s maximum term has not been reached when a sector terminates, the provider may commit the same piece into a new sector and regain the quality-adjusted power. The new sector’s committed lifespan is still constrained to fit within the claim’s term. +The miner actor must ensure that the old sector is no longer alive. #### Sector extension A sector's scheduled commitment term cannot be extended beyond the maximum term of any @@ -518,6 +526,17 @@ The miner actor then invokes `ClaimAllocations` and, if successful, computes quality adjusted power for the sector according to the piece size and QA power multiplier. ### Migration +#### Network upgrade +A network state migration is required to implement this proposal. +That migration must: +- Implement the new verified registry state schema for allocations and claims. +- Migrate the verified registry client balances map to DataCap token balances, + with appropriate unit conversion. +- Add a new field to SectorOnChainInfo identifying whether its quality-adjusted power is calculated + according to the old spread-out method, or the new direct method +- TODO: handle pending deals: either create allocations or keep not-yet-proven sectors in the old world. + +#### Opt-in per-sector The power for existing sectors is not changed by this proposal. A provider may opt in to migrate a sector to the new QA power calculation method by invoking a new method.