Clearer presence support; towards remote interfaces and Una

[michaelfig]

In trying to address the underlying issues of #792 and take baby steps towards strongly-typed interfaces and general Una (looking at you @FUDCo), @erights and I came up with the following plan:

1. Clarify terminology: HandledPromises deal with "presences", and marshal deals with "remotables". SwingSet deals with "remote presences" (from some other vat) and other objects (born in this vat).
2. Add export function pureCopy(obj): pureObj which provides a deep "pass-by-copy" independent of obj. The resulting object is both copied to be pure data and automatically hardened, thus cannot be a communications channel. If the new object contains any functions or property getters/setters,
   then pureCopy throws a TypeError.
3. Add export function Remotable(iface = 'Remotable, props = {}, identity = {}) to @agoric/marshal.
   a. Do pureIface = pureCopy(iface) to prevent proxy interference or non-pass-by-copy objects. For now, iface is just a string that is the alleged type name of the returned remotable, and if it is not a string, a "not implemented" error is thrown.
   b. if identity already exists in the WeakMap below, then throw
   c. The identity's prototype is set to an object that contains only toString() and [Symbol.toStringTag] that displays the iface's allegedName when debugging and console.loging the remotable.
   d. Object.defineProperties(identity, getOwnPropertyDescriptors(props))
   e. remotable = harden(identity) and assert that the remotable is compatible with marshal rules for remotables
   f. set the WeakMap entry for the remotable to pureIface to indicate to marshal that the remotable is intended to be pass-by-presence and not pass-by-copy.
   e. The ownProperties of props are defined on the remotable, then it is hardened and returned. If props is an empty object, then the returned remotable will also not have any ownProperties.
4. @agoric/marshal will call iface = getInterfaceOf(obj) to detect remotables (iface is undefined for non-remotables), and add the iface to the @qclass: "slot" type to indicate to the remote side what interface the remotable handles.
5. SwingSet (and @agoric/captp) will consume the slot's iface and use their marshal's Remotable function to reconstitute it on the receiving side. We need to ensure backward/forward compatibility: that the iface portion of the slot does not cause problems with existing code that doesn't expect it, nor with new code when it isn't sent.
6. The ag-solo REPL can detect if it has a remotable by checking that getInterfaceOf(obj) is not undefined, and then stringify the remotable in debug output. It need not be aware of the form of the iface nor the two-level (own properties + prototype) layout of the remotable.

That's the first part of the plan. Along the way to general Una, we can gradually make marshal stricter. We will first print a warning if an empty object to be marshalled wasn't created with Remotable('MyDescription'). After this warning has been eliminated from our (Agoric's) code, we can bump marshal's major version and change to marshalling empty objects as data.

As far as the grand plan: once we retire the old sniffing of objects to see if they are remotables, we can mix data properties with presences. Also, we can make function remotables as first-class citizens with Remotable(iface, props, function() { return foo }), and use tildot to call them.

Separately, we can evolve the iface argument so that it is a proper interface type descriptor that can be enforced by SwingSet to restrict the input and output types of the props.

[erights]

Step 3.a should also reject if remote has already been made into a presence, i.e., if it is already a key in that weakmap.

[erights]

Step 6. I don't think the repl should test whether or not it is a remote. Instead, it should test whether or not it is a presence. It should print a main presence with the same logic with which it prints a remote presence.

[erights]

HandledPromise is available to the user for other purposes. They may very well create other kinds of remotes that aren't presences. This suggests that the three argument Presence is still too dangerous to make generally available. Only a marshalling system, like liveSlots or captp, should use the semantics of the three argument form, and only on remotes that it had just asked HandledPromise to create.

[erights]

Everything else looks exactly right. Thanks!

[FUDCo]

I will reread this tomorrow when my environment doesn't contain various thought-disrupting audio activity (Janice is watching the news in the other room with the TV sound up), but my first cut take is that this seems like a fine start on the immediate problem. And I do endorse a tropism towards strongly typed interfaces. However, I'm unclear how this plan relates to a path towards a more general unum scheme (mind you, I'm not actually certain a general unum scheme is a thing we have a pressing need for, as cool as I think it would be), since a key idea -- to my sensibilities at least -- is that different presences can present different APIs to their local environments depending on those presences' respective roles within the unum they're part of, and managing this heterogeneity is a foundational concern (I have an intuition that whatever we might come up with for interface definitions for objects with multiple facets may be relevant, but that's possibly even more distant from our immediate concerns).

[erights]

Yeah, I'm trying for something much less than that heterogeneity. More like the subset of the Unum vision I did in E. Frankly, I'm confused by the heterogeneity aspect. I don't understand how the differences in "environments" align with vat boundaries.

Perhaps we need a different word than "Unum" for the trajectory we're on. But I love the word "presence" in any case. "Presence" is exactly right.

[michaelfig]

I'll make the other changes you noted. This one deserves more thought:

HandledPromise is available to the user for other purposes. They may very well create other kinds of remotes that aren't presences. This suggests that the three argument Presence is still too dangerous to make generally available. Only a marshalling system, like liveSlots or captp, should use the semantics of the three argument form, and only on remotes that it had just asked HandledPromise to create.

We can export function isPresence(obj): boolean from makeMarshal, and only the two-argument function Presence(iface, props = {}). Then have makeMarshal return:

 return harden({
    serialize,
    unserialize,
    convertRemoteToPresence,
    getInterfaceOf(obj) { ... },
  });

where function convertRemoteToPresence(iface, descs, remote) can be used by the marshalling system.

Can you think about whether getInterfaceOf can be made generally available as a module export instead of the results of makeMarshal? I'm not sure about how global or local the Presences should be.

Bikeshedding on names would also be appreciated.

[michaelfig]

@erights, I updated the above comment.

[michaelfig]

I'm thinking that isPresence is available to any importer of @agoric/marshal, but Presence, getInterfaceOf, and convertRemoteToPresence are only available from the makeMarshal return value.

This prevents cross-contamination of different marshallers with different interface type semantics.

[erights]

I like this direction. Seems right. Let's discuss on Monday.

[warner]

Huh, neat. One thought so far: putting the iface identifer on the @qclass: 'slot' object (the ones that get JSON-serialized into the "body" of the capdata) means there's an opportunity for divergence: the body might say slot: 0, iface: 'foo' in one place, and slot: 0, iface: 'bar' somewhere else. It seems like we should make it impossible to express that. But.. I wouldn't want to put the iface identifier in the slots array (the numerically-indexed list of o+4/p-3 references which get translated through the c-lists by the kernel), even though it would address the divergence problem, because that would expose these types to the kernel.

Or.. maybe we might want the kernel involved in types? We talked a while ago about some sort of IDL registry, where adding a new Vat requires registering the types of all the objects it might expose, so the kernel object table can know the type of every entry, and method invocations that don't match could be detected by the kernel. Better error checking and all.

[michaelfig]

In further conversation with @erights:

• @agoric/marshal should provide a pureCopy export which provides a deep "pass-by-copy" independent of source. An object thus copied+hardened cannot be a communications path.
• Presence(iface, props = {}, identity = {}) will:
  a. do the equivalent of iface = pureCopy(iface)
  b. assert that identity is not frozen, and that it is not already a presence
  c. then modify it with the toStringTag prototype stuff
  d. Object.defineProperties(identity, getOwnPropertyDescriptors(props))
  e. harden(identity),
  f. assert that identity is compliant with presence rules
  g. and associate identity with iface
• Presence-to-iface is a WeakMap that is held by @agoric/marshal. Any consumer of that module can call getInterfaceOf(presence) to see what the interface is (or get undefined if it is not a Presence)
• No need for HandledPromise.isRemote(); getInterfaceOf(presence) subsumes all its useful functionality

[michaelfig]

Huh, neat. One thought so far: putting the iface identifer on the @qclass: 'slot' object (the ones that get JSON-serialized into the "body" of the capdata) means there's an opportunity for divergence

No opportunity. If a Presence is created, its interface is kept in a WeakMap that never changes (see notes above). IOW, the iface (especially when it is not an identifier) is maintained directly by @agoric/marshal. The slot serializer probably will not have a way to attach an iface... it will happen automatically for Presences.

Or.. maybe we might want the kernel involved in types? We talked a while ago about some sort of IDL registry, where adding a new Vat requires registering the types of all the objects it might expose, so the kernel object table can know the type of every entry, and method invocations that don't match could be detected by the kernel. Better error checking and all.

Maybe this information will only be used by liveSlots, but yes, it should be forwarded via the kernel to other consumers of an exported vat object.

[michaelfig]

As far as the grand plan: once we retire the old sniffing of objects to see if they are presences, we can mix data properties with presences. Also, we can make function presences Presence(iface, props, function() { return foo }), and use tildot to call them.

[erights]

Some progress at #2178

[erights]

For many parts of this, we're making progress in other issues and PRs. My sense is that for the remaining issues here that are not represented elsewhere, this can remain in the icebox for now.