Expose structured clone as serialization API

[lorenbrichter]

I haven't quite convinced myself this is a good idea, but I think it is and might be worthy of consideration.

Exposing V8's implementation of the structured clone algorithm as an explicit serialization/deserialization API would be useful in a lot of contexts.

As an example: Cloudflare is implementing their transactional storage API by accepting naked js objects as values and serializing them internally with structured clone. Exposing this API would allow a reimplementation in Deno. It could also be used for low overhead storage and sending data between Deno instances.

https://developers.cloudflare.com/workers/runtime-apis/durable-objects#transactional-storage-api (see put())

Since this would need to be implemented for Workers and/or IndexedDB anyway, it would add very small API surface area, something like:

function Deno.serialize(object: any) : ArrayBuffer|null
function Deno.deserialize(a: ArrayBuffer) : any|null

Obvious downsides:

• Versioning. If this is used for persistence, any changes to the V8 algorithm would put long term storage at risk. Non-trivial but possible to deal with.

[lorenbrichter]

Related to #3557

[ghost]

This specifically goes against what Deno is doing, they don't want to provide access to V8 directly like this.

You can still send data between instances, workers, but I don't think Deno will have IndexedDB.
.postMessage is supposed to use structured cloning already, so why isn't that sufficient?

[crowlKats]

but I don't think Deno will have IndexedDB

we will, just it is a matter of time. see #1699

[lorenbrichter]

This specifically goes against what Deno is doing, they don't want to provide access to V8 directly like this.

You can still send data between instances, workers, but I don't think Deno will have IndexedDB.

I suppose the fact that it's V8's structured cloning is just an implementation detail (it doesn't have to be, it just has to be implemented at the level where you have fast and deep access to js internals). Any sort of fast/binary serialization API would satisfy a lot of use cases, when the only alternative is JSON.stringify.

That said, I see the other side of the argument as well.

.postMessage is supposed to use structured cloning already, so why isn't that sufficient?

This only lets you see the "deserialized" half of the object at both ends. Something that gives you access to the serialized half could be useful too.

[ghost]

I suppose the fact that it's V8's structured cloning is just an implementation detail (it doesn't have to be, it just has to be implemented at the level where you have fast and deep access to js internals)...

That's the root of the problem, providing any access to the "JS internals" results in implementation details one way or another, unless a very carefully setup API is provided. That API likely wouldn't be compatible with other JS runtimes.

Yet I'm not sure where the line should be drawn between making an ECMAScript proposal or just asking that Deno provide such access.

[kitsonk]

This specifically goes against what Deno is doing, they don't want to provide access to V8 directly like this.

It isn't against what Deno is doing.

We need structured cloning working properly before we could consider this though. The challenge is that unless you are doing something with it (like sending it between workers or putting in a db) it doesn't make sense in a runtime isolate. Having it be something that plugins have easy access to though would be more important.

[ghost]

...It isn't against what Deno is doing...

Ah, my apologies, I had rashly assumed that this would fall under

Does not leak V8 concepts into user land.

But by all means, if the need for it exists and it's put under std/ or the Deno namespace.

[ghost]

...we will, just it is a matter of time. see #1699

I was unsure if it would ever end up on Deno's agenda, but it appears that there's already an issue for it.
I'm all for an IndexedDB in Deno, although... it does have a slightly over-complicated setup, but that's just my opinion.

[kitsonk]

Does not leak V8 concepts into user land.

Structured cloning is not a V8 concept... it is an ECMAScript/JavaScript concept, though, like I said, its exposure as a runtime API is "questionable" because it isn't useful from within the isolate. Really useful for "behind the scenes" things like plugins though.

[kitsonk]

Actually, that being said... for some of the web APIs that we provide in Deno, it can be useful there... so something on Deno.core might make sense. We have this in there at the moment:

deno/cli/rt/01_web_util.js

Lines 34 to 36 in 374d433

	/** Clone a value in a similar way to structured cloning. It is similar to a
	* StructureDeserialize(StructuredSerialize(...)). */
	function cloneValue(value) {

Which we have to use in streams and potentially other APIs when transferring/cloning objects.

[ghost]

Structured cloning is not a V8 concept... it is an ECMAScript concept, though, like I said, its exposure as a runtime API is "questionable" because it isn't useful from within the isolate. Really useful for "behind the scenes" things like plugins though.

Yes, precisely what I thought of when I had last commented. Structured cloning is in the ES spec.

Yet the requested API asks for raw access to bytes

serialize: any => ArrayBuffer
deserialize: ArrayBuffer => any

I doubt V8 even uses raw memory to encode them, but if it does, an API like this would forcefully expose implementation details, although it could be used for rapidly creating your own object by simply modifying the correct bytes.

The API exposed would have to be opaque in order to prevent it from leaking implementation, yet if it's opaque then it seems fundamentally useless, as it can't be stored (unless there is a storage mechanism provided by Deno).

[ghost]

node exposes similar interface with use cases found on github

[jeremyBanks]

Most of the dynamic languages that JavaScript/Deno is competing with have some kind of clone function, and there has been a lot of demand to add one to browsers over the years. I've frequently wanted to reach for a built-in function to clone basic data structures, but JSON.parse(JSON.stringify(...)) is often too limited. I wrote a Stack Overflow answer about structured cloning eight years ago, and it still gets several votes a week from other users looking for this capability.

It is unfortunate that none of the browser vendors have moved forward to directly expose structured cloning to users, whatwg/html#793, so there isn't an existing browser interface to follow. But that said, I think that a Deno.structuredClone(...) function as @kitsonk suggested would be a great addition to Deno, and many users would really appreciate it. This wouldn't expose the internals of V8's serialization implementation (as Node's interface does), only the structured cloning behaviour, which is already standardized even if the name is not.

[lucacasonato]

so there isn't an existing browser interface to follow. But that said, I think that a Deno.structuredClone(...) function as @kitsonk suggested would be a great addition to Deno

There actually is. You can use MessageChannel to structured clone objects. We just need to implement it.

[jeremyBanks]

@lucacasonato True, and that will be good to have. However, that's only usable asynchronously, and it would be better if it could also be used from normal synchronous functions.

As mentioned in the Stack Overflow post, browsers do technically expose the behaviour behind a couple of synchronous interfaces too, but they're both indirect and probably not in-scope for Deno:

history.pushState() and history.replaceState() both create a structured clone of their first argument, and assign that value to history.state. You can use this to create a structured clone of any object.

The Notification constructor creates a structured clone of its associated data. It also attempts to display a browser notification to the user, but this will silently fail unless you have requested notification permission. In case you have the permission for other purposes, we'll immediately close the notification we've created.

(Although per Surma's post on the topic, the History API hack actually performs pretty well in Chrome... 🤔)

[jeremyBanks]

@00ff0000red It looks like that repo hasn't been updated in seven years, and the language it's using has become outdated and inconsistent with the specifications it's referencing. I think that looking for consistency/consensus with TC39 or browser vendors would be great, but as far as I can tell there's no sign that any of them are moving this forward. (I'd be happy to be wrong!)

[ghost]

@jeremyBanks Oops, yes, that is a very valid concern... back to what @lucacasonato suggested, what would be so bad about an asynchronous API? A large amount of Deno's API are already async, and with TLA, it's not that bad.

The only concern that I would have regarding using MessageChannel to clone objects is that the object would be serialized+deserialized twice, back and forth. An async Deno.clone that skips the extra serialization/copy would be nice.

[jeremyBanks]

@00ff0000red Having an async API (or both Deno.structuredClone and Deno.structuredCloneSync) wouldn't be the end of the world (❤️ TLA), but it just seems like unnecessary overhead if the underlying operation is always synchronous. As far as I understand (maybe not very far, I don't know much about v8 internals), the actual cloning of the object in memory is always going to block the thread when it happens. It's never going to happen in parallel, so having calls be async just adds overhead and limits use without any benefit.

[wynntee]

Hi, I've just published a module for serializing JavaScript data in browsers, Deno, and Node.js alike. It's meant for exchanging data between browsers and servers, but can also be used for structured cloning.

Do you guys mind having a look?
https://github.com/wynntee/joss

The module is based on a binary serialization scheme that I created (will upload the specs hopefully next week). The scheme covers all the data types and data structures supported by the Structured Clone Algorithm listed in this MDN page, except Blob, File, FileList. ImageBitmap and ImageData. The scheme also covers the "little" things like primitive wrapper objects, sparse arrays, signed zeros, and circular dependencies.

[ghost]

Looks like we've got Deno.core.{serialize, deserialize}, as of PR #9458.

[kitsonk]

I am going to mark this as maybe 2.0. Having them in Deno.core does make them available at runtime, but aren't only intended to be used by the internal of Deno and not part of the public API. That is appropriate for these at this time, and we can consider adding them to the public API in 2.0.

[jeremyBanks]

🎉 Excellent, thank you. 🙃

export const structuredClone = <Value extends StructuredClonable>(
  value: Value,
): Value => {
  // deno-lint-ignore no-explicit-any
  const core = (Deno as any).core;
  return core.deserialize(core.serialize(value));
};

export type StructuredClonable =
  | { [key: string]: StructuredClonable }
  | Array<StructuredClonable>
  | ArrayBuffer
  | ArrayBufferView
  | BigInt
  | bigint
  | Blob
  // deno-lint-ignore ban-types
  | Boolean
  | boolean
  | Date
  | Error
  | EvalError
  | Map<StructuredClonable, StructuredClonable>
  // deno-lint-ignore ban-types
  | Number
  | number
  | RangeError
  | ReferenceError
  | RegExp
  | Set<StructuredClonable>
  // deno-lint-ignore ban-types
  | String
  | string
  | SyntaxError
  | TypeError
  | URIError;

[surma]

FWIW, my PR to the HTML spec to expose structuredClone() saw some recent activity from browser vendors, so I’m somewhat optimistic that we might have this on the web platform soon as well.

[bartlomieju]

Superseded by #11539