Explicit deal settlement - long-term resolution of built-in market cron risk (FIP-0074)

[anorth]

Edit: FIP-0074

Background

See prior discussion #638 that led to FIP-0060.

Cron-initiated payments

The built-in cron actor calls the built-in storage market actor at every epoch to perform maintenance (mainly the disbursement of payments) for all active deals. Each deal is scheduled for this maintenance on a 30 day period.

There are currently about 45 million active deals (currently ~520 deals per epoch), a number which is growing quickly. The cost of this level of service is unsustainable. It nearly caused disaster when the update period was 24 hours. Pushing it out to 30 days in FIP-0060 only bought some time to design a more permanent solution. There remains significant risk that this processing could grow beyond the ability of the chain to safely execute it. The cost is also largely redundant for the 98% of deals that specify no payment.

At the end of each epoch, the market actor loads all deals that are scheduled for maintenance. For each deal, the function:

1. Removes state for any unactivated deals that have timed out (“expired”)
2. Settles incremental payments for activated deals
3. Removes state for any completed or terminated deals

Direct data onboarding

A concurrent proposal is to support skipping the built-in storage market actor for some data onboarding, which provides a cost reduction for arrangements that don’t need payments. As this capability is implemented and adopted, we might expect the number of built-in market deals to decrease even as data onboarding increases.

If and when adopted, this would mitigate the risk significantly. However, the benefit depends on participants’ behavior and we can’t make any guarantees about the extent to which they will adopt the new methods. The built-in market actor might still be expected to host a large number of payment-bearing deals. So we don’t want to rely on direct data onboarding in either the near or long term as a final solution.

Proposal

Remove payment settlement and state clean-up from cron, and provide a new, explicit user message to settle payments. Continue timing out un-activated deals in cron.

New method for settlement

A new externally-invoked method, MarketActor::SettleDealPayments, may be invoked by a storage provider (or anyone on their behalf) to settle incremental payments for a specified list of deals. SPs can invoke this to calculate and disburse accrued payments. Settling the final payment for a completed deal also removes the deal metadata from state.

This explicit message aligns the gas cost of settling payments with the party deriving the benefit. It also bounds the cost, like most other state transitions, by the block gas limit. Providers might settle payments quite infrequently, reflecting the relatively low utility of doing so, and will prefer periods of low gas demand. Less frequent updates result in a lower system-wide amount of computation performed.

Synchronous clean-up on termination

The existing MarketActor::OnMinerSectorsTerminate method is changed to synchronously make a final payment settlement, charge penalties, and clean up state. It behaves as if SettleDealPayments was invoked immediately after marking deals as terminated. This replaces the current technique of merely recording the termination epoch and then waiting for cron to do the work.

Since sector termination may be invoked by the miner actor’s cron handler, this will sometimes still do work in cron. This may be resolved by future changes to reduce miner actor cron too. Explicit sector termination by the SP will account for the cost of clean up to their gas usage.

No automatic settlement for new deals

When a deal is first published, it is added to the cron queue after its start epoch. If it is not activated prior to that epoch, it is expired and cleaned up in cron. This remains as it works today.

However, when a deal is found to have been newly-activated, it is not re-scheduled for subsequent processing.

Storage providers are not expected to invoke settlement for deals that carry no payment. The metadata for such newly-created but no-payment deals will thus remain in state indefinitely. However, with direct data onboarding, there is little benefit and some cost to involving the built-in market actor in new unpaid deals. We expect relatively few additional unpaid deals to be accumulated after that is possible. Stale blockchain state poses a much lesser risk to blockchain execution than continual processing. If a large amount does accumulate, it could be dealt with independently.

Continue automatic settlement for existing deals

Deals that are already activated prior to this change continue to be settled automatically in cron. Since no new deals are scheduled for automatic settlement, this will result in the cost gradually declining until there are no active deals scheduled. At that point, a future simplification can remove the code for payment settlement and state clean-up from cron.

No migration

The scheme described above involves no state migration. The cron queue will eventually empty itself as existing deals complete and no new ones are scheduled.

Alternatives

Some alternative ideas that seem less favourable than the proposal include:

• Rather than leave existing deals in the queue as scheduled, we could redistribute them to an even longer update period, like 90 days or more. This would reduce the ongoing cron cost to ⅓ , and can be achieved without a migration. There is no significant downside to this, it’s merely deemed not worthwhile.
• Rather than leave existing deals in the queue as scheduled, we could reschedule them all once to an epoch after their expected completion. This would give a large reduction in the ongoing cron cost. However, it would increase the size of the queue significantly (28,800 -> ~1.5m epoch entries, HAMT height 3 -> 4). This too is deemed not worthwhile.
• Rather than rescheduling existing deals when processed, we could stop scheduling them. This would leave most of their metadata in state indefinitely (SPs have no incentive to settle payments on zero-payment deals). This only seems like a good idea for new deals, which will mostly have payments (and thus an incentive to be settled) after direct data onboarding is implemented.
• Similarly, we could migrate to empty the queue, but this would again leave most deal metadata in state indefinitely.

[anorth]

I am drafting a FIP for this proposal at #805.

Implementation is being prototyped in filecoin-project/builtin-actors#1377.

[jennijuju]

qq - how would manual deal payment settlement work when a sector is under faulty status?

[anorth]

The built in market has no knowledge of temporary faults either before or after this changes, so settlement would work as usual. This proposal does not attempt to change the amount payable for any deal.

[kaitlin-beegle]

Wanted to follow up THIS COMMENT, re: future scenarios and existing proposals.

I think Jennifer raises a good point. FWIW, FIPs should be holistic policy documents. They should consider future scenarios and potential impacts/outcomes, but they should not be solely predicated upon them. FIPs should independently introduce good network policy in the present, even as they consider future needs.

I don't think it's a problem that this FIP references the potential impacts of direct data onboarding, since the FIP itself does not solely exist in anticipation of a future where direct data onboarding is the norm.

[anorth]

Context from the draft:

Storage providers are not expected to invoke settlement for deals that carry no payment.
The metadata for such newly-created but no-payment deals will thus remain in state indefinitely.
However, with direct data onboarding, there is little benefit and some cost to
involving the built-in market actor in new unpaid deals.
Relatively few additional unpaid deals might be accumulated after that is possible.
Stale blockchain state poses a much lesser risk to blockchain execution than continual processing.
If a large amount does accumulate, it could be dealt with independently.

and from @jennijuju

I don't necessary think there is anything wrong with whats stated here. However, Im not sure if we should discard a potential negative impact based on the assumption some future FIP will be accepted.

[anorth]

I think @jennijuju's point is that the behaviour introduced (stale deal state) might be undesirable if direct onboarding doesn't happen.

I think the text acknowledges the stale state and suggests two reasons why it's not considered worth addressing:

1. direct onboarding, if used widely, will greatly reduce the size of the impact
2. in any case, stale state is much less bad than growing cron work

We'll probably want to eventually do something about it, but I'm claiming it's sufficiently low importance and long term that it's not worth adding complexity now.

[Stebalien]

I'm confused about:

Synchronous clean-up on termination

versus

The metadata for such newly-created but no-payment deals will thus remain in state indefinitely.

Wouldn't synchronous cleanup remove the no-payment deals as well?

[anorth]

Deals in sectors that are terminated early will be synchronously cleaned up.

Deals that run to completion with never be cleaned up, except by the SP invoking settlement. The market actor is not informed about sectors that reach their committed lifetime and merely expire.

[Stebalien]

Rather than leave existing deals in the queue as scheduled, we could reschedule them all once to an epoch after their expected completion.

It would also be possible to do this per storage provider, forcing the storage provider to do their own cleanup e.g., on before submitting any proofs.

Although I'm not sure if that's worth it, honestly.

[anorth]

I don't think we can block proofs (after DDO, anyway), but yes partitioning by provider would mean we could force clean-up while publishing new deals or similar. It would be complex to do efficiently though – the manual approach means we can rely on the SP to identify which deals are done with off-chain indexes. We'd need a queue or similar structure to identify them on-chain.

[anorth]

I have made a gas estimate for manual settlement by inferring it from the gas consumed by cron. Now included in FIP-0074:

At epoch 3055532 (2023-07-22 08:46:00) the network hosted approximately 37,488,000 deals
(according to Starboard).
Deals are processed every 86,400 epochs, giving 433 per epoch (deals are distributed quite uniformly between epochs).
The built-in market cron invocation in this epoch consumed 25,509,119,291 gas
(note that this subsidy exceeds the entire target budget for a tipset),
which implies an upper bound on per-deal settlement cost of 58,912,000 gas units.
Note that the cron invocation also performs other work, so this bound is conservative.
Note also that this access pattern is random; a future optimisation to market state could group deals by provider,
significantly reducing the state churn cost of settling deals that share a provider.

As a reference, publishing a batch of 18 deals in the same epoch cost 2,117,931,981 gas units (117,662,887 per deal).
This means that performing a single settlement after deal completion will likely cost
a little less than half as much as publishing it.

Payment settlement, in our very inefficient built-in market actor implementation, is expensive. This is why doing it automatically and for free is a threat to network security. A future market implementation that cares about gas costs will likely do it much better.

@raulk suggested gas refunds. I don't think they're likely to be positive in this situation. Also from the FIP:

One possible extension to manual settlement is to provide gas refunds to callers.
Gas used for these calls would still contribute toward the block gas limit (and so be bounded),
but charge only a fraction of the usage to the caller.
This was rejected since it does not align the cost of settlement with the party deriving the benefit,
charging it instead to users of all other blockchain computation by raising their gas price.
With greatly reduced gas cost, providers might settle payments more frequently than is actually valuable to them,
wasting chain capacity, and could even abuse it to push the base fee higher than any natural market level.
Network-level gas refunds would also be impossible to provide to future user-programmed smart markets,
creating an artificial barrier to their development and adoption.

With market cron currently "only" consuming 1x the entire tipset gas budget, this does raise the possibility of holding off on these changes until we can also implement some optimisations to settlement to reduce the cost. If direct data onboarding is widely adopted, the pressure on the built-in market might be reduced significantly in the near term. However, if it doesn't, we could possibly find ourselves in an emergency situation again.

FYI @alexytsu @momack2 @raulk @jennijuju