how can we virtualize Notifiers/Subscriptions?

[warner]

What is the Problem Being Solved?

To get Zoe and all contracts to use virtual objects (#4504, #4511, #4383), we also need a solution for Notifiers and Subscriptions. These are objects with which a publisher can push out updates to some piece of state. Their clients use promises to learn about the updates. When the client wants to be updated (maybe for the first time, maybe triggered by the previous update), they send a message and camp out on the result promise. The Notifier holds on to the resolver function, and invokes it when a new update is published.

Promises do not play nice with virtualized data. It would be awfully convenient for our implementation if we could forbid promises from both virtual and durable objects/collections (#2485)

If we can't put promises in virtual data, then I don't know how we virtualize a Notifier. When the notifier is inactive (pushed out to disk), then when an update is published, the notifier will be deserialized and makeKind invoked to build a new Representative. This Representative wants to pull the resolver function out of its .state, but those can't be serialized. So either we have to keep the resolver function in RAM for all Notifiers (even inactive ones, breaking the virtual-object goal of "consume no RAM for inactive objects"), or we abandon the ability to resolve after being pushed to disk, which breaks Notifiers entirely.

#3787 is a sketch for "virtual promises" that might be useful. It would be a special kind of serializable object that wraps a promise ID (a vpid), and has a method named .resolve. It'd probably be created in conjunction with a real Promise, which could be serialized in the usual way.

Description of the Design

none yet

Security Considerations

Test Plan

[warner]

As @dtribble pointed out, It may help to break the old Notifier and obligate customers to obtain a new one (i.e. include the Notifier in the "visible upgrade trauma"). We could establish a rule that when your getUpdateSince promise rejects, instead of re-requesting one from the old notifier, you need to back to the Vault (or Purse or whatever) and get a whole new Notifier first.

function followPurse(purse) {
  let currentNotifierP;
  function startOrRestart() {
    currentNotifierP = E(purse).getBalanceNotifier();
    currentNotifierP.getUpdateSince().then(gotUpdate, gotError);
  }
  function gotUpdate(({value, updateCount})) {
    console.log(`balance: ${value}`);
    E(currentNotifierP).getUpdateSince(updateCount).then(gotUpdate, gotError);
  }
  function gotError() {
    startOrRestart();
  }
  startOrRestart();
}

Our current plan calls for all outstanding Promises (i.e. all kernel promise table entries whose decider field points at the vat being upgraded) to be rejected during the upgrade process, which would feed the getUpdateSince().catch() rejection path nicely.

The part I don't yet have a plan for is how the durable Purse could hold on to a merely-Virtual Notifier in one version, lose that reference during upgrade (mixing virtual+durable in the state instead of rejecting merely-virtual at set() time? eww. scanning the state and deleting the merely-virtual pieces during upgrade? eww.), learn that the reference was lost (meta-notifier to inform the Purse that a Notifier was lost? eww), and then build a new (again merely-virtual) Notifier in the version-2 world.

If the notifier were a DurableNotifier, which uses virtual promises (#3787) of some sort, that might help.

[warner]

One idea @FUDCo and I had today:

• create a DurableNotifier object whose kind closes over a merely-virtual collection
• each instance of DurableNotifier has an allocated integer in its state
• this integer is used as a key of the merely-virtual collection (BigMap)
• the values of the merely-virtual collection are merely-virtual VirtualNotifiers
• now the durable object rules are satisfied, but durableNotifier.getVirtualNotifier() gives you back a usable notifier
• after upgrade, the version-2 code provides a new DurableNotifier kind definition, which closes over a new merely-virtual collection
  • getVirtualNotifier notices the !mvns.has(index) and creates a new VirtualNotifier
• clients use their VirtualNotifier until one of the getUpdateSince promises rejects, at which point they go back to durableNotifier.getVirtualNotifier() to get a new one

This doesn't require any sort of serialization-visible "frangible" linkage from durable to merely-virtual, since the link is hidden in the closed-over collection.

Downsides:

• deleting one requires manual effort: no automatic GC if the last client drops the VirtualNotifier
  • because userspace doesn't get WeakRef or some sort of "weak value map" to weakly hold the VirtualNotifiers in

[turadg]

May be solved by #3787

[FUDCo]

We've figured out the machinery for this, but there's some implementation remaining in other issues that are being tracked elsewhere. So closing in favor of #4567 and #5185, which are in process.