Add captureTime, receiveTime and rtpMetadata to VideoFrameMetadata

[guidou]

These fields are useful for WebRTC-based applications.
See issue #601

[guidou]

cc @Djuffin, @padenot , @youennf, @aboba

[aboba]

Does this PR imply any behavior in WebCodecs API?

For example, on encoding is there an expectation that VideoFrame.captureTime is copied to EncodedVideoChunk.captureTime? Or on decoding is EncodedVideoChunk.receiveTime or EncodedVideoChunk.rtpMetadata to be copied to VideoFrame.receiveTime or VideoFrame.rtpMetadata?

If there are no changes in behavior (e.g. if the attributes don't affect the encode or decode process or some other aspect of WebCodecs) then the attributes could be defined in another specification where behavior is affected (e.g. mediacapture-transform?), and added to the VideoFrame Metadata Registry.

[guidou]

This PR as currently written does not imply any behavior in the WebCodecs API, although I would expect the things you mentioned (e.g., forwarding them to/from EncodedVideoChunk) as potentially useful.

The idea for this PR is to provide information to applications so that they can do similar things to what they can do with requestVideoFrameCallback (e.g., better A/V sync and delay measurements). This doesn't require any other behavior changes in WebCodecs (at least for applications using mediacapture-transform + WebRTC).

[guidou]

I think we can specify forwarding to EncodedVideoChunk in a separate PR since this one has value on its own without specifying further changes to WebCodecs.

[Djuffin]

I used to be skeptical about these timestamps since they are not passed through the encoding-decoding cycle, but since we already have entries in VideoFrame Metadata Registry that don't do that, I think it's okay now.

And RTC software like Teams, Mean and Facetime can really use it for A/V sync and latency estimation, even if they have to pass this information via separate channels. So LGTM

[aboba]

I agree that this metadata is useful. The question is whether behavior is well specified, so that interop is possible. For example, there is the question of where the metadata orginates:

1. MAY/SHOULD/MUST the MediaStreamTrackProcessor method provide VideoFrame.captureTimestamp if the MST is obtained from a local capture?
2. MAY/SHOULD/MUST the MediaStreamTrackProcessor method provide VideoFrame.receiveTimestamp and VideoFrame.rtpMetadata if the MST is obtained remotely via WebRTC-PC?

[Djuffin]

I thought for all metadata entries the answer to these questions is MAY.

[aboba]

@Djuffin MAY might be ok for these metadata fields. However, is alignment of VideoFrame.timestamp and EncodedVideoChunk.timestamp optional for WebCodecs implementations?

[youennf]

I thought for all metadata entries the answer to these questions is MAY.

I agree from a WebCodecs POV.
But it is not sufficient from an interop point of view.
Probably each spec defining a MST video source should describe which metadata it generates, just like each spec defines which constraints are supported by a given source.
Putting the definition at the source ensures the same metadata is exposed via MSTP or via VideoFrame constructor (from a video element).

That would mean mediacapture-main and webrtc-pc here.
As of mediacapture-transform VideoTrackGenerator, nothing seems needed though we could add a note stating that metadata are preserved.

[guidou]

FWIW, the requestVideoFrameCallback spec where these fields are originally defined say that captureTime applies to local cameras and remote frames (WebRTC), receiveTime to WebRTC frames, and rtpTimestamp to WebRTC frames. But I agree with @youennf that having each MST source spec indicate the metadata it generates is the best way to organize that.

In any case, we need to have entries for these fields in the VideoFrameMetadata registry.

[chrisn]

Media WG meets today, please add agenda label if you'd like to discuss.

[Djuffin]

@Djuffin MAY might be ok for these metadata fields. However, is VideoFrame.timestamp and EncodedVideoChunk.timestamp optional for WebCodecs implementations?

They're mandatory.

Is there some kind of deep connection here that I miss?

[aboba]

Changes to VideoFrame registry are ok, but should probably not reference RVFC. receiveTime and rtpMetadata could be defined in WebRTC-PC or WebRTC-Extensions and captureTime could be defined in Media Capture & Streams or Media Capture Extensions.

[chrisn]

Minutes from 9 July 2024 Media WG meeting. @aboba summarised the conclusion in #813 (review).

[Djuffin]

Summary of WG discussion:
HTMLVideoElement.requestVideoFrameCallback is not the best spec to reference here, because it doesn't describe how and when these timestamps are set.
Corresponding changes need to be made in MediaStreamTrackProcessor and Media Capture and Streams specs. Something along the lines: "MediaStreamTrackProcessor sets capture timestamps for VideoFrames coming from camera..."

Later this PR should reference these specs.

[aboba]

"And RTC software like Teams, Mean and Facetime can really use it for A/V sync and latency estimation, even if they have to pass this information via separate channels."

[BA] To do A/V sync, captureTime and receiveTime need to be provided for both audio and video.

Also, if they are to be usable for non-RTP transports, they need to be defined in a way that is independent of RTP/RTCP. For example, on the local peer, captureTime represents the capture time of the first byte according to the local wallclock. On a remote peer, captureTime is set by the receiver. For example, the local peer's captureTime can be serialized on the wire and then set on the receiver (e.g. not adjusted to the receiver wallclock). receiveTime is set on the receiver, based on the receiver's wallclock. (receiveTime - captureTime) can then be used to estimate the sender/receiver offset as well as Jitter.

[aboba]

The RVFC text is too RTP centric to be used here. I'd copy over the text and make some changes. For a WebCodecs application, captureTime can be serialized on the wire and set by a WebCodecs application for frames received from a remote peer. Also, NTP timestamp format does not imply a global clock so change “estimated using clock synchronization” to “aligned to the sender wallclock”.

[handellm]

The plan is to

1. Define what the concept "capture time" is in mediacapture-extensions, and define "remote capture time" over at webrtc-extensions (this would refer to the new mediacapture-extensions "capture time" concept plus text copied from rVFC to describe estimation).
2. Refer to these new definitions here (local tracks: look here; remote webrtc tracks: look here) instead of the rVFC reference, but use DOMHighResTimestamp relative to local Performance.timeOrigin.

That would leave captureTime unset otherwise. A webcodecs app could use whichever out-of-band techniques to compute & set a valid local DOMHighResTimestamp. Does it need to be mentioned in the spec?

[aboba]

Comment: RVFC text is also too WebRTC-centric. Can we allow this to be present in a WebCodecs application as well (set by the receiver)?

[handellm]

The plan is to describe this as "set for remote webrtc tracks" and use DOMHighResTimestamp relative to local Performance.timeOrigin. I agree it should be settable by a WebCodecs application.

[aboba]

The RVFC text is ok here but I'd still probaby copy it over rather than referencing it.

[handellm]

I planned to define what a "rtp timestamp" is over at webrtc-extensions and simply define here that it's present for "remote webrtc tracks" together with a reference.