Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make VideoFrame/AudioData Transferable and Serializable #286

Merged
merged 5 commits into from
Jul 14, 2021
Merged

Make VideoFrame/AudioData Transferable and Serializable #286

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fe56565
Transfer+serialization steps for VideoFrame
chcunningham Jun 18, 2021
555298f
Transfer+serialization steps for AudioData
chcunningham Jun 19, 2021
01117f8
remove old detached slots
chcunningham Jun 21, 2021
211728d
Improve/centralize transfer and serialization documentation
chcunningham Jul 6, 2021
8d259de
Merge remote-tracking branch 'origin/main' into transferable_raw_media
chcunningham Jul 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 133 additions & 39 deletions index.src.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,7 +452,7 @@
Run these steps:
1. For each |output| in |outputs|:
1. Let |data| be an {{AudioData}}, initialized as follows:
1. Assign `false` to {{AudioData/[[detached]]}}.
1. Assign `false` to {{platform object/[[Detached]]}}.
2. Let |resource| be the [=media resource=] described by |output|.
3. Let |resourceReference| be a reference to |resource|.
4. Assign |resourceReference| to
Expand Down Expand Up @@ -890,7 +890,7 @@
[=Enqueues a control message=] to encode the given |data|.

When invoked, run these steps:
1. If the value of |data|'s {{AudioData/[[detached]]}} internal slot is
1. If the value of |data|'s {{platform object/[[Detached]]}} internal slot is
`true`, throw a {{TypeError}}.
2. If {{AudioEncoder/[[state]]}} is not `"configured"`, throw an
{{InvalidStateError}}.
Expand Down Expand Up @@ -1206,7 +1206,7 @@
[=Enqueues a control message=] to encode the given |frame|.

When invoked, run these steps:
1. If the value of |frame|'s {{VideoFrame/[[detached]]}} internal slot is
1. If the value of |frame|'s {{platform object/[[Detached]]}} internal slot is
`true`, throw a {{TypeError}}.
2. If {{VideoEncoder/[[state]]}} is not `"configured"`, throw an
{{InvalidStateError}}.
Expand Down Expand Up @@ -2150,11 +2150,32 @@
encouraged to destroy such resources quickly to reduce memory pressure and
facilitate resource reuse.

### Transfer and Serialization ### {#raw-media-serialization-and-transfer}

This section is non-normative.

{{AudioData}} and {{VideoFrame}} are both [=transferable objects|transferable=]
and [=serializable objects|serializable=] objects. Their transfer and
serialization steps are defined in [[#audiodata-transfer-serialization]] and
[[#videoframe-transfer-serialization]] respectively.

Transferring an {{AudioData}} or {{VideoFrame}} moves its `[[resource
reference]]` to the destination object and closes (as in {{AudioData/close()}})
the source object. Authors may use this facility
to move an {{AudioData}} or {{VideoFrame}} between realms without copying the
underlying [=media resource=].

Serializing an {{AudioData}} or {{VideoFrame}} effectively clones (as in
{{VideoFrame/clone()}}) the source object, resulting in two objects that
reference the same [=media resource=]. Authors may use this facility to clone
an {{AudioData}} or {{VideoFrame}} to another realm without copying the
underlying [=media resource=].

AudioData Interface {#audiodata-interface}
---------------------------------------------

<xmp class='idl'>
[Exposed=(Window,DedicatedWorker)]
[Exposed=(Window,DedicatedWorker), Serializable, Transferable]
interface AudioData {
constructor(AudioDataInit init);

Expand Down Expand Up @@ -2182,9 +2203,6 @@
</xmp>

### Internal Slots ###{#audiodata-internal-slots}
: <dfn attribute for=AudioData>\[[detached]]</dfn>
:: Boolean indicating whether {{AudioData/close()}} was invoked on this
{{AudioData}}.

: <dfn attribute for=AudioData>[[resource reference]]</dfn>
:: A reference to a [=media resource=] that stores the audio sample data for
Expand All @@ -2210,7 +2228,7 @@
AudioData(init)
</dfn>
1. Let |frame| be a new {{AudioData}} object, initialized as follows:
1. Assign `false` to {{AudioData/[[detached]]}}.
1. Assign `false` to {{platform object/[[Detached]]}}.
2. Assign |init|.{{AudioDataInit/format}} to
{{AudioData/[[format]]}}.
3. Assign |init|.{{AudioDataInit/sampleRate}} to
Expand Down Expand Up @@ -2286,7 +2304,7 @@
destination buffer.

When invoked, run these steps:
1. If the value of |frame|'s {{AudioData/[[detached]]}} internal slot is
1. If the value of |frame|'s {{platform object/[[Detached]]}} internal slot is
`true`, throw an {{InvalidStateError}} {{DOMException}}.
2. Let |copyElementCount| be the result of running the
[=Compute Copy Element Count=] algorithm with |options|.
Expand All @@ -2306,7 +2324,7 @@
:: Creates a new AudioData with a reference to the same [=media resource=].

When invoked, run these steps:
1. If the value of |frame|'s {{AudioData/[[detached]]}} internal slot is
1. If the value of |frame|'s {{platform object/[[Detached]]}} internal slot is
`true`, throw an {{InvalidStateError}} {{DOMException}}.
2. Return the result of running the [=Clone AudioData=] algorithm with
[=this=].
Expand All @@ -2315,9 +2333,7 @@
:: Clears all state and releases the reference to the [=media resource=].
Close is final.

When invoked, run these steps:
1. Assign `true` to the {{AudioData/[[detached]]}} internal slot.
2. Assign `null` to {{AudioData/[[resource reference]]}}.
When invoked, run the [=Close AudioData=] algorithm with [=this=].

### Algorithms ### {#audiodata-algorithms}

Expand Down Expand Up @@ -2347,14 +2363,55 @@
{{AudioData/[[resource reference]]}}.
2. Let |reference| be a new reference to |resource|.
3. Assign |reference| to {{AudioData/[[resource reference]]}}.
4. Assign the values of |data|'s {{AudioData/[[detached]]}},
4. Assign the values of |data|'s {{platform object/[[Detached]]}},
{{AudioData/[[format]]}}, {{AudioData/[[sample rate]]}},
{{AudioData/[[number of frames]]}},
{{AudioData/[[number of channels]]}}, and
{{AudioData/[[timestamp]]}} slots to the corresponding slots in
|clone|.
2. Return |clone|.

: <dfn>Close AudioData</dfn> (with |data|)
:: Run these steps:
1. Assign `true` to |data|'s {{platform object/[[Detached]]}} internal slot.
2. Assign `null` to |data|'s {{AudioData/[[resource reference]]}}.

### Transfer and Serialization ###{#audiodata-transfer-serialization}

: The {{AudioData}} [=transfer steps=] (with |value| and |dataHolder|) are:
:: 1. If |value|'s {{platform object/[[Detached]]}} is `true`, throw a
{{DataCloneError}} {{DOMException}}.
2. For all {{AudioData}} internal slots in |value|, assign the value of
each internal slot to a field in |dataHolder| with the same name as the
internal slot.
3. Run the [=Close AudioData=] algorithm with |value|.

: The {{AudioData}} [=transfer-receiving steps=] (with |dataHolder| and |value|)
are:
:: 1. For all named fields in |dataHolder|, assign the value of each named
field to the {{AudioData}} internal slot in |value| with the same name
as the named field.

: The {{AudioData}} [=serialization steps=] (with |value|, |serialized|, and
|forStorage|) are:
:: 1. If |value|'s {{platform object/[[Detached]]}} is `true`, throw a
{{DataCloneError}} {{DOMException}}.
2. If |forStorage| is `true`, throw a {{TypeError}}.
chcunningham marked this conversation as resolved.
Show resolved Hide resolved
3. Let |resource| be the [=media resource=] referenced by
|value|'s {{AudioData/[[resource reference]]}}.
4. Let |newReference| be a new reference to |resource|.
5. Assign |newReference| to |serialized|.[[resource reference]].
6. For all remaining {{AudioData}} internal slots (excluding
{{AudioData/[[resource reference]]}}) in |value|, assign the value of
each internal slot to a field in |serialized| with the same name as the
internal slot.

: The {{AudioData}} [=deserialization steps=] (with |serialized| and |value|)
are:
:: 1. For all named fields in |serialized|, assign the value of each named
field to the {{AudioData}} internal slot in |value| with the same name
as the named field.

### AudioDataCopyToOptions ### {#audiodata-copy-to-options}

<xmp class='idl'>
Expand Down Expand Up @@ -2555,7 +2612,7 @@
{{CanvasDrawImage}}'s {{CanvasDrawImage/drawImage()}}.

<xmp class='idl'>
[Exposed=(Window,DedicatedWorker)]
[Exposed=(Window,DedicatedWorker), Serializable, Transferable]
interface VideoFrame {
constructor(CanvasImageSource image, optional VideoFrameInit init = {});
constructor(sequence<PlaneInit> planes,
Expand Down Expand Up @@ -2604,10 +2661,6 @@

### Internal Slots ###{#videoframe-internal-slots}

: <dfn attribute for=VideoFrame>\[[detached]]</dfn>
:: A boolean indicating whether {{destroy()}} was invoked and underlying
resources have been released.

: <dfn attribute for=VideoFrame>[[resource reference]]</dfn>
:: A reference to the [=media resource=] that stores the pixel data for
this frame.
Expand Down Expand Up @@ -2905,7 +2958,7 @@
to be used with {{VideoFrame/copyTo()}} with the given options.

When invoked, run these steps:
1. If {{VideoFrame/[[detached]]}} is `true`, return `0`.
1. If {{platform object/[[Detached]]}} is `true`, return `0`.
2. If {{VideoFrame/[[format]]}} is `""`, throw a {{NotSupportedError}}
{{DOMException}}.
3. Let |parsedOptions| be the result of running the [=Parse
Expand All @@ -2923,7 +2976,7 @@
were returned.

When invoked, run these steps:
1. If {{VideoFrame/[[detached]]}} is `true`, return `0`.
1. If {{platform object/[[Detached]]}} is `true`, return `0`.
2. If {{VideoFrame/[[format]]}} is `""`, throw a {{NotSupportedError}}
{{DOMException}}.
3. Let |parsedOptions| be the result of running the [=Parse
Expand Down Expand Up @@ -2975,7 +3028,7 @@
[=media resource=].

When invoked, run these steps:
1. If the value of |frame|’s {{VideoFrame/[[detached]]}} internal slot is
1. If the value of |frame|’s {{platform object/[[Detached]]}} internal slot is
`true`, throw an {{InvalidStateError}} {{DOMException}}.
2. Return the result of running the [=Clone VideoFrame=] algorithm with
[=this=].
Expand All @@ -2984,22 +3037,12 @@
:: Clears all state and releases the reference to the [=media resource=].
Close is final.

When invoked, run these steps:
1. Assign `null` to {{VideoFrame/[[resource reference]]}}.
2. Assign `true` to {{VideoFrame/[[detached]]}}.
3. Assign `""` to {{VideoFrame/format}}.
4. Assign `0` to {{VideoFrame/[[coded width]]}},
{{VideoFrame/[[coded height]]}}, {{VideoFrame/[[visible left]]}},
{{VideoFrame/[[visible top]]}}, {{VideoFrame/[[visible width]]}},
{{VideoFrame/[[visible height]]}}, {{VideoFrame/[[display width]]}},
and {{VideoFrame/[[display height]]}}.
5. Assign `null` to {{VideoFrame/[[duration]]}} and
{{VideoFrame/[[timestamp]]}}.
When invoked, run the [=Close VideoFrame=] algorithm with [=this=].

### Algorithms ###{#videoframe-algorithms}
<dfn>Create a VideoFrame</dfn> (with |output|, |timestamp|, |duration|, |displayAspectWidth|, and |displayAspectHeight|)
1. Let |frame| be a new {{VideoFrame}}, constructed as follows:
1. Assign `false` to {{VideoFrame/[[detached]]}}.
1. Assign `false` to {{platform object/[[Detached]]}}.
2. Let |resource| be the [=media resource=] described by |output|.
3. Let |resourceReference| be a reference to |resource|.
4. Assign |resourceReference| to {{VideoFrame/[[resource reference]]}}.
Expand Down Expand Up @@ -3082,14 +3125,28 @@

: <dfn>Clone VideoFrame</dfn> (with |frame|)
:: 1. Let |clone| be a new {{VideoFrame}} initialized as follows:
1. Assign |frame|.{{VideoFrame/[[resource reference]]}} to
1. Let |resource| be the [=media resource=] referenced by |frame|’s
[[resource reference]].
2. Let |newReference| be a new reference to |resource|.
3. Assign |newReference| to |clone|'s
{{VideoFrame/[[resource reference]]}}.
2. Assign |frame|.{{VideoFrame/format}} to {{VideoFrame/format}}.
3. Assign all remaining attributes of |frame| (
{{VideoFrame/codedWidth}}, {{VideoFrame/codedHeight}}, etc.) to
those of the same name in |clone|.
4. Assign all remaining internal slots of |frame| (excluding
{{VideoFrame/[[resource reference]]}}) to those of the same name
in |clone|.
2. Return |clone|.

: <dfn>Close VideoFrame</dfn> (with |frame|)
:: 1. Assign `null` to |frame|'s {{VideoFrame/[[resource reference]]}}.
2. Assign `true` to |frame|'s {{platform object/[[Detached]]}}.
3. Assign `""` to |frame|'s {{VideoFrame/format}}.
4. Assign `0` to |frame|'s {{VideoFrame/[[coded width]]}},
{{VideoFrame/[[coded height]]}}, {{VideoFrame/[[visible left]]}},
{{VideoFrame/[[visible top]]}}, {{VideoFrame/[[visible width]]}},
{{VideoFrame/[[visible height]]}}, {{VideoFrame/[[display width]]}},
and {{VideoFrame/[[display height]]}}.
5. Assign `null` to |frame|'s {{VideoFrame/[[duration]]}} and
{{VideoFrame/[[timestamp]]}}.

: <dfn for=VideoFrame>Parse VideoFrameCopyToOptions</dfn> (with |options|)
:: 1. Let |parsedRect| be the result of running the [=VideoFrame/Parse CopyTo
Rect=] with |options|.
Expand Down Expand Up @@ -3226,6 +3283,43 @@
[=parsed copyto options/allocationSize=].
9. Return |parsedOptions|.

### Transfer and Serialization ###{#videoframe-transfer-serialization}

: The {{VideoFrame}} [=transfer steps=] (with |value| and |dataHolder|) are:
:: 1. If |value|'s {{platform object/[[Detached]]}} is `true`, throw a
{{DataCloneError}} {{DOMException}}.
2. For all {{VideoFrame}} internal slots in |value|, assign the value of
each internal slot to a field in |dataHolder| with the same name as the
internal slot.
3. Run the [=Close VideoFrame=] algorithm with |value|.

: The {{VideoFrame}} [=transfer-receiving steps=] (with |dataHolder| and
|value|) are:
:: 1. For all named fields in |dataHolder|, assign the value of each named
field to the {{VideoFrame}} internal slot in |value| with the same name
as the named field.

: The {{VideoFrame}} [=serialization steps=] (with |value|, |serialized|, and
|forStorage|) are:
:: 1. If |value|'s {{platform object/[[Detached]]}} is `true`, throw a
{{DataCloneError}} {{DOMException}}.
2. If |forStorage| is `true`, throw a {{TypeError}}.
3. Let |resource| be the [=media resource=] referenced by
|value|'s {{VideoFrame/[[resource reference]]}}.
4. Let |newReference| be a new reference to |resource|.
5. Assign |newReference| to |serialized|.[[resource reference]].
6. For all remaining {{VideoFrame}} internal slots (excluding
{{VideoFrame/[[resource reference]]}}) in |value|, assign the value of
each internal slot to a field in |serialized| with the same name as the
internal slot.

: The {{VideoFrame}} [=deserialization steps=] (with |serialized| and |value|)
are:
:: 1. For all named fields in |serialized|, assign the value of each named
field to the {{VideoFrame}} internal slot in |value| with the same name
as the named field.


VideoFrame CopyTo() Options {#videoframe-copyto-options}
------------------------------------------------------------
Options to specify which {{VideoFrameRect}} to copy and the offset and
Expand Down