Add a message for New CC Sector with Snap-up Data

[snadrus]

This space is to discuss FIP-snadrus/FIPs/FIPS/fip-draft_new_msg_cc_with_snap-up_data.md, a plan to make trivial the decision to create CC sectors first and snap up data afterward.

Problem:
Sealing pipelines need to bring in a lot of data and add delay between accepting data and having them in sectors.
Snap deals help, but are add too much cost. They also do not help with the problem of keeping sectors alive as they transfer out of sealing machines (and possibly across services)
Solution:
Use non-interactive PoRep (Discussion 603) to avoid the PreCommit message entirely (just take randomness from the chain).
Delay CC messages until data is snapped in. Have a combo message with ProveCommit & Snap to reduce cost.

Benefits:

• Simpler sealing pipelines (our most complex pipeline), especially for Sealing-as-a-Service or fast sealing machines.
• Faster turnaround from data ingest to on-chain availability: This should make for more efficient use of deal-accepters time.
• Potential avenue for storage partners to ship "formatted drives" with a wallet & empty, unregistered sectors.

Drawbacks overcome by this FIP:

• Avoiding registration until data is ready means that sectors can "sit ready" for no FIL cost or penalty.
• Fewer messages (and hopefully unified proofs) should make the snap-up cost nearly the same as a regular data sector.

Drawbacks not overcome:

• In total, more processing will occur than with regular data sectors (as snap-up does and will continue to do). But as this lightens the data-transfer burden of expensive "sealing machines", this can easily be economically beneficial.

[jennijuju]

a plan to make trivial the decision to create CC sectors first and snap up data afterward.

Could you please elaborate more on what is the problem today (whats missing from the existing pipeline) and whats the proposal?

[ZenGround0]

It would be very helpful to elaborate on what problem a single message solves as opposed to calling these two messages in succession, which I expect will unlock the sealing pipeline workflow you want.

[anorth]

I don't understand the proposal. There is a bit more specified in the draft FIP, but could you please explain the proposal in more detail here so we can discuss the problem and alternative solutions?

[suykerbuyk]

@anorth the idea is to streamline data onboarding. The basic idea is to turn CC sectors into the functional equivalent of pre-formatted floppies from the days of yesteryear, without necessarily committing them to the network.

Snapping is MUCH faster and far less compute intensive. In an ideal scenario, one would be accumulating inconsequential CC sectors in a relatively slow but predictable pipeline and at the point of data ingest, "snap" customer data into pre-sealed sectors and commit the transaction in the fewest number of block chain messages as possible, and also at a sector rate that exceeds what a conventional PC1/PC2/C1 pipeline could muster.

By not putting the CC sectors (our pre-formatted floppy disk) on the network, this also opens up all kinds of opportunities for service providers to "sell" CC sectors ready for data ingest while being unbound from PoST timing constraints.

In a previous life (at Seagate), we even openly discussed pre-loading new build disk drives with CC sectors before sending them on to SPs. This simply was not and is not possible without this kind of innovation.

[ZenGround0]

that I don't have to keep online

Is this the piece Im missing? Keeping them online and posting is a lot of overhead cost?

[anorth]

that I don't have to keep online

This sounds like the key. From my understanding of our proof-of-space security, we can't support this, so it's not just a matter of a different API.

At initial proof-of-replication, we need the sealer to commit to a blockchain and then take a randomness challenge from that chain at after a defined delay. This is achieved by PreCommit. That challenge is then used in later sealing stages to permute the data and generate a proof that this was done correctly (unique challenges give unique replicas). My understanding is that we need the proof of this sealing to land on chain within some defined time period (30 days) after the challenge was taken in order to have sufficiently high confidence that the SP really does have the whole sector. We can't allow an arbitrary time period between the challenge and the proof, or SPs could fake some storage.

Non-interactive PoRep (see #603) would get a bit closer to the ideal (removing the pre-commit step), but I think it still requires that the proof takes a challenge from the chain and a similar bound on the delay between that challenge and proof verification. It just doesn't have to commit to that challenge epoch up-front. This would reduce the cost of missing the window to be only the computational cost, and not loss of pre-commit deposit.

After the PoRep is verified on chain, there's another potential opportunity for a sector to go offline. The protocol currently requires daily WindowPoSt to start immediately, but there may be some way to allow an arbitrary period before the first Window PoSt, which would I think then meet the request here. I don't think the Window PoSt is contributing to PoRep security, only the ongoing proof of space (which gives power).

@nicola or @irenegia, do I have this right? Is there any potential scheme where an SP could sit on "cold" CC sectors?

[anorth]

(A large amount of "cold" power that could be activated in an instant does give rise to other risks to consensus, which would need careful evaluation).

[snadrus]

If we keep the 30-day limit, this would be no different than the current risk profile for Snap. It would just be unknown instead of pre-committed.

[irenegia]

ehi @anorth, sorry for the late replay!
From the security point of view, if we consider the case of interactive PoRep (current situation), we do not have an upper bound for the PreCommitChallengeDelay (today set to 150 epoch). This values can not be reduced, but it can make it larger.
And same for MaxProveCommitDuration, which is set to 30 days plus the PreCommitChallengeDelay.

If we consider non-interactive PoRep, then the concepts of PreCommitChallengeDelay and MaxProveCommitDuration disappear. (We will still have the same condition as today for SealRandom. That is, SealRandEpoch ≤ SealEpoch - ChainFinality).

[snadrus]

Updated: This should depend on #603 (approach 1) as well as provide a lower cost via message bundling.