Proposing setCodecPreferences to deal with both send and recv codecs

[henbos]

Based on prior discussions such as #2937 (comment), here's a concrete proposal which I think fixes unidirectionality, requires no changes to JSEP (consistent with #2938 (comment)) and avoids backwards compat issues introduced by #2926.

How did we get here?

JSEP has this note (https://rtcweb-wg.github.io/jsep/#rfc.section.4.2.6):

Note that setCodecPreferences does not directly affect which codec the implementation decides to send. It only affects which codecs the implementation indicates that it prefers to receive, via the offer or answer.

We took this to mean "SCP should only take receive codecs as input", something which JSEP neither says or IMO imply. The note only talks about the direct effect; there's also the indirect effect that the preferences have.

Excluding send codecs was a mistake.

The preferences are relevant even for sendonly transceivers, since we must have something to offer (https://rtcweb-wg.github.io/jsep/#rfc.section.5.2.1):

If codec preferences have been set for the associated transceiver, media formats MUST be generated in the corresponding order, and MUST exclude any codecs not present in the codec preferences.

This does not mean that preferences are irrelevant for the sending use case: the receiver not having called setCodecPreferences is a valid use case, in which case the sender's preferences are the ones we use (https://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1):

• If codec preferences have been set for the associated transceiver, media formats MUST be generated in the corresponding order, regardless of what was offered, and MUST exclude any codecs not present in the codec preferences.
• Otherwise, the media formats on the m= line MUST be generated in the same order as those offered in the current remote description, excluding any currently unsupported formats. Any currently available media formats that are not present in the current remote description MUST be added after all existing formats.

We could simply say we don't care about this use case, but that appears to have some downsides when it comes to unidirectional codecs.

Proposal

"setCodecPreferences accepts both send codecs and receive codecs as input."

The actual preferences you end up using depend on the createOffer filtering we already apply based on transceiver.direction. This makes it valid for...

• A sendonly transceiver to offer sendonly codecs and sendrecv codecs.
• A sendrecv transceiver to offer sendrecv codecs, but not sendonly or recvonly codecs.
• A recvonly transceiver to offer recvonly codecs and sendrecv codecs.

To be discussed

Note that it's possible for the filtering to end up with an empty codec list. There are multiple ways we could deal with this, of which I have no strong preference. Examples:

1. SCP and direction setting could sanity check and throw if we end up with an empty set.
2. createOffer could throw.
3. We don't throw, but if we have nothing to offer we reject the m= section.
4. If we end up with empty set we offer one of the mandated to support sendrecv codecs as a "backup" (VP8 and H264).
5. SCP could require that at least one codec is sendrecv, in which case there is no filtering that could remove it, and changing direction is always safe.

[fippo]

we must have something to offer

We don't. If we have nothing to offer this is similar to "we have nothing to answer" which is described as

None of the offered media formats are supported and, if applicable, allowed by codec preferences.

in https://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1
I think this case is rare enough that going for 3 is a pragmatic solution. And yes, I have a code change for that already that is quite small

[aboba]

@henbos You may be taking JSEP Section 4.2.6 too literally. It is correct in saying that sCP is not for selecting what is sent. That is true regardless of direction. However, saying that sCP only applies to codecs that can be received, regardless of direction, has to contend with RFC 3264 Section 5.1:

"For a sendonly stream, the offer SHOULD indicate those formats the offerer is willing to send for this stream. For a recvonly stream, the offer SHOULD indicate those formats the offerer is willing to receive for this stream. For a sendrecv stream, the offer SHOULD indicate those codecs that the offerer is willing to send and receive with."

Since this is a SHOULD, there is some wiggle room for sendrecv streams. For example, a sendrecv m-line with preference {H.265, H.264} could be allowed if H.265 can be received but not sent. This would allow negotiation to succeed between a receive-only H.265 Offerer and an Answerer who can send and receive both H.265 and H.264, using only a single sendrecv m-line, as long as the Answerer included both H.265 and H.264 in its Answer. Since non-WebRTC endpoints may not support sendonly and recvonly m-lines, this approach has interoperability advantages.

However, concluding that sCP with a sendonly m-line is only about receiving preferences seems like a step too far. Instead I'd suggest that a sendonly transceiver can only offer codecs/profiles that can be sent (send-only or send/recv). In a situation where the browser can send at a different profile/levels than it can receive, only the send profiles can be included when direction = sendonly.

Similarly, a recvonly transceiver can only offer codecs/profiles that can be received (recv-only or send/recv). In a situation where the browser can send at a different profile/levels than it can receive, only the receive profiles are included. This is consistent with JSEP Section 4.2.6.

[dontcallmedom-bot]

This issue was discussed in WebRTC February 2024 meeting – 20 February 2024 (setCodecPreferences to deal with both send and recv codecs #2939)

[jan-ivar]

@henbos what are next steps here?

[aboba]

Related: ietf-wg-avtcore/draft-ietf-avtcore-hevc-webrtc#22

Justin's suggestion.

[aboba]

@jan-ivar H.265 is getting close to being enabled behind a flag, so in the near future we will be able to see how receive-only codecs behave, and what the problems are. At that point we can present the issue to the WG in light of the behavior we see. IMHO it would be desirable to converge on recommended changes prior to enabling H.265 by default.

[henbos]

At TPAC we discussed the different options, including allowing sendrecv transceiver to include recvonly codecs and the potential foot gun there, but there was a strong preference to keep things simple and only include codecs supported in the direction(s) of the transceiver. I.e. sendrecv transceiver can only have sendrecv codecs

Let's close this issue in favor of #3006