Re-query MediaKeys to handle codec profile changes

[SemihGk]

Have you read the FAQ and checked for duplicate open issues?:
Yes
What version of Shaka Player are you using?:
master
Can you reproduce the issue with our latest release version?:
yes
Can you reproduce the issue with the latest code from master?:
yes
Are you using the demo app or your own custom app?:
custom app.
If custom app, can you reproduce the issue using our demo app?:
No, you need to test specific stream.
What browser and OS are you using?:
it is reproducible on Chrome and Tizen.
What are the manifest and license server URIs?:

If you require the custom app, I am happy to share it.
What did you do?
When an encrypted live stream becomes clear, it throws 4011 UNPLAYABLE_PERIOD error.
What did you expect to happen?
It should play the stream smoothly.
What actually happened?
playback is crashed.

This issue is introduced at this commit. When a stream is initiated, all supported types are stored at this array. However, supportedTypes only includes the codec and mime combination of initial segments as it is captured at this line. When the stream becomes clear, the ad segments have different codecs which are not in the supportedTypes array. So, these ad segments are filtered at this line. Although all ad segments are playable, they are filtered out and, eventually throws 4011 error code. I also attached supportedTypes array and an ad segment screenshots.

Please let me know if the issue is still unclear for you. Happy to help on fixing this issue.

Thank you,
Semih.
An ad segment =>

supportedTypes array =>

[SemihGk]

Thank you very much for your quick fix, @vaage . I appreciated. However, there is an edge case and your fix does not cover it unless I am wrong. If a stream starts with clear content and switches to encrypted content later, it will filter out the encrypted contents because the clear segments have different codecs. Do you want me create another ticket for this? Apologies for this. I should have mentioned this earlier.

Thank you,
Semih.

[vaage]

We'll re-open the issue while we figure out if it's a separate issue. This issue was more about a regression during a encrypted to clear transition.

@SemihGk if I understand correctly, you are now asking about the reverse (the clear to encrypted transition). Right now media source does not support codec switching, so the content would still need to have the same codec (that would be filtered outside of DrmEngine). So is the problem that the codec stays the same but the profile changes causing us to reject it?

[SemihGk]

Apologies for the misexplanation, @vaage . When I said different codecs, I meant the transition between mp4a.40.2 and mp4a.40.5 as the screenshots. With your fix, now the stream is able to switch from encrypted (mp4a.40.5) to clear (mp4a.40.2) content (I tested it works fine). However, the reverse case cannot be switched because you only filter encrypted segments at here. Now, imagine the clear content to encrypted content switch case. Clear content codec types will be stored in here. When the stream becomes encrypted, those encrypted segments will be filtered again due to the supportedTypes array will include clear content codec types only. Just a friendly reminder we initiate drmEngine even though stream starts with clear content at here . So, supportedTypes will include the codec types of clear segments.

Please let me know if I missed something. Thank you.

[vaage]

@SemihGk , this case came up in a conversation with @joeyparrish yesterday. He is out of the office until next Friday, but I will sync-up with him again once here is back and we will figure out how we want to approach this issue (we came up with 5 or 6 different cases so it gets messy fast).

I will assign this to myself so that it won't fall between the cracks.

[SemihGk]

I totally understand your concern. I also agree on the switch between clear and encrypted content is a kind of hack-ish workaround on the drm flow anyway. Figuring out the best solution would be better to avoid any temporary workaround. Thank you very much.

[joeyparrish]

Hi @SemihGk, @vaage. To clarify:

The difference between mp4a.40.2 and mp4a.40.5 has nothing to do with clear vs encrypted. These are different profiles of AAC. Either could be clear, and either could be encrypted. Encryption is signalled outside of the codec string.

And since these are two different AAC profiles, this shouldn't count as "codec switching" in MSE. In general, we can switch profiles of the same codec.

The problem is this:

---

When we negotiate with the browser for a CDM, we have to supply a list of MIME types we want to decode. The browser then reports back which ones can be decoded in context of the CDM. We use this information to filter out things the CDM can't decode.

For example, if your content has both VP9 and H.264, you would ask for both. The browser may then report back that the CDM only supports H.264, so you would filter out VP9 as unusable. Failing to do so could result in a playback failure later.

Or, your content may contain multiple profiles of H.264, such as a baseline profile, a main profile, and a UHD profile. You would ask for all of the ones you have in your content, and the browser may report back that the CDM only supports a subset of them, such as baseline & main, so you would filter out the UHD profile. Failing to do so could result in a playback failure later.

Now in your scenario, the main content has mp4a.40.5. If we start playback during a main content period, we ask for a CDM that can decode this. We hear back from the browser that yes, it can decode mp4a.40.5. Later, during an ad break, we see mp4a.40.2 for the first time. Since we never asked for a CDM that could decode that, it's not in the list from the browser. So we're left assuming that the CDM can't decode it. (In fact, in Chrome, the Widevine CDM can handle both just fine.)

---

I see a few options right away:

1. We could assume that the CDM would support any profile we didn't ask about. This could be risky, as it could lead to playback failures if we are wrong.
2. We could hard-code what different CDMs support. This would be brittle and would need to be updated periodically. It would also be hard to get this information for some CDMs. And if there are platform-specific details to what the CDM supports, we would have no good way to discover and represent this in our hard-coded data.
3. We could ask for all profiles instead of just the ones in the manifest. This would give us more info to work with on manifest updates, but it may be difficult to list all valid codec strings. I believe there are 32 bits of info on an h.264 codec string, so brute force is out of the question.
4. We could, on a manifest update, renegotiate with the browser to find out if the CDM supports the new codec strings. This could lead to two problems, though:
   a. If the platform can only support one CDM instance at a time, this negotiation may fail for reasons unrelated to the codec strings. This would lead to a playback failure.
   b. If the platform supports multiple CDM implementations for the same key system (software-backed and hardware-backed), it may report success for the new codec strings, but for a different CDM implementation than the one currently in use. This would lead to a playback failure.
5. We could tear down the media stack and renegotiate from scratch with the new details. This would lead to an interruption in playback. Our seamless multi-period playback would no longer be seamless. As a slight improvement to this, we could keep track of all codec strings encountered in all manifest updates (even strings not currently present in the latest update of the manifest). Once we have seen all the codec strings a live stream is going to encounter (in your case, after one ad break and one content period), there would be no more need to tear down the media stack. This is probably the best option long-term, since it is the most robust.
6. We could offer a configurable codec string list that would supplement the manifest strings. A service provider could reasonably know these in advance, and could tell us about them before we parse the manifest. This would avoid the need to tear down the media stack, and would allow us to query all necessary codec strings early. This is probably the best option short-term, since it is easy.

---

Back to you, @vaage!

[SemihGk]

Thank you very much for your detailed reply, @joeyparrish . It is quite clear now. So, a stream should be able to switch codec profiles even though the stream does not have any clear segment.

I would like to speak out about some of those options:

1. We could assume that the CDM would support any profile we didn't ask about. This could be risky, as it could lead to playback failures if we are wrong.

This option seems the easiest solution. In addition to this, we could implement a 'soft' restriction as we have it on AbrManager. Let's say, we can still filter out the unsupported codec profiles, but if there is no available variant left, we can leave one type of codec profiled variant which we did not ask it to CDM yet.

4. We could, on a manifest update, renegotiate with the browser to find out if the CDM supports the new codec strings. This could lead to two problems, though:

This one sounds the most robust solution for me. Calling navigator. requestMediaKeySystemAccess once there are new codec profiles should solve this. However, I am not quite sure about your concerns.

a. If the platform can only support one CDM instance at a time, this negotiation may fail for reasons unrelated to the codec strings. This would lead to a playback failure.

I checked some versions of EME specs out, but I could not find any information about which drafts have EME is singleton. I checked 0.1b Feb draft, that's not a singleton. I assume previous versions does not support this navigator. requestMediaKeySystemAccess API. So, pollyfil would return a MediaSource.isTypeSupported result which shaka already calls it to filter the variants before. Thus, we should be safe.

b. If the platform supports multiple CDM implementations for the same key system (software-backed and hardware-backed), it may report success for the new codec strings, but for a different CDM implementation than the one currently in use. This would lead to a playback failure.

This sounds a platform CDM implementation mistake. If different CDMs return report different results, I personally believe we should report this bug to the platform owners. I am just curios do you know any platform which has multiple CDMs for the same keySystem?

5. We could tear down the media stack and renegotiate from scratch with the new details. This would lead to an interruption in playback. Our seamless multi-period playback would no longer be seamless. As a slight improvement to this, we could keep track of all codec strings encountered in all manifest updates (even strings not currently present in the latest update of the manifest). Once we have seen all the codec strings a live stream is going to encounter (in your case, after one ad break and one content period), there would be no more need to tear down the media stack. This is probably the best option long-term, since it is the most robust.

I can see two issues on this solution. First one as you pointed out, there would be an interruption and this is definitely a bad experience for users. Second one, the implementation would require a bit work. The last one is that the stream might have more than 2 codec profile changes. For our case, we do not know all the ads have the same codec profiles. We need to discuss this with backend team.

6. We could offer a configurable codec string list that would supplement the manifest strings. A service provider could reasonably know these in advance, and could tell us about them before we parse the manifest. This would avoid the need to tear down the media stack, and would allow us to query all necessary codec strings early. This is probably the best option short-term, since it is easy.

This might be tough. As for our case, none of our ad segment may have the same codec profile. It may vary as the ad urls are inserted by a 3rd party company. If you think other options are not solid, we will need to discuss this with out back-end team.

I really appreciate for your considerations. Please let me know if I can help you to figure out a possible solution. Thanks a lot.

[joeyparrish]

I would like to speak out about some of those options:

1. We could assume that the CDM would support any profile we didn't ask about. This could be risky, as it could lead to playback failures if we are wrong.

This option seems the easiest solution. In addition to this, we could implement a 'soft' restriction as we have it on AbrManager. Let's say, we can still filter out the unsupported codec profiles, but if there is no available variant left, we can leave one type of codec profiled variant which we did not ask it to CDM yet.

Easy, yes. But I don't think it's a good idea, because of the risk of playback failures.

The risk is this: if we assume the CDM supports anything we didn't ask about, we could fail playback in some cases. What if your content goes up to UHD resolutions, or uses an HDR color space, but those features aren't supported by the CDM? If we join during an ad break, we would perhaps only query SD & SDR profiles, then fail to decode later when the content is resumed. Worse, it might be non-deterministic, because only the users with the most bandwidth would attempt playback of those streams.

4. We could, on a manifest update, renegotiate with the browser to find out if the CDM supports the new codec strings. This could lead to two problems, though:

This one sounds the most robust solution for me. Calling navigator. requestMediaKeySystemAccess once there are new codec profiles should solve this. However, I am not quite sure about your concerns.

a. If the platform can only support one CDM instance at a time, this negotiation may fail for reasons unrelated to the codec strings. This would lead to a playback failure.

I checked some versions of EME specs out, but I could not find any information about which drafts have EME is singleton. I checked 0.1b Feb draft, that's not a singleton.

It's not about the EME spec. MediaKeys is never a singleton, but there is no requirement that a user agent support multiple concurrent instances of MediaKeys in the same page. In the past, I have worked on embedded HTML5 platforms which can only support a single video context at a time. So it's not a stretch to imagine an embedded system that might only be able to support one CDM instance at a time. After all, a CDM instance would consume limited hardware resources for hardware-based DRM.

If requestMediaKeySystemAccess is implemented universally in a way that is analogous to isTypeSupported for MediaSource, then you would expect to be able to simply query this for type support, without getting a rejection based on resources. The rejection, if any, would then occur on createMediaKeys. After re-reading the relevant parts of the spec, the spec text seems to support this. Perhaps it would be safe to re-query.

b. If the platform supports multiple CDM implementations for the same key system (software-backed and hardware-backed), it may report success for the new codec strings, but for a different CDM implementation than the one currently in use. This would lead to a playback failure.

This sounds a platform CDM implementation mistake. If different CDMs return report different results, I personally believe we should report this bug to the platform owners. I am just curios do you know any platform which has multiple CDMs for the same keySystem?

This is the case on Android, in which two CDM implementations can exist in parallel: hardware-backed, and pure-software implementations. Both use the same key system ID, but have different robustness strings. It's not a mistake, and it's not against the spec.

If we can assume that robustness strings or key system IDs will always differentiate between them, then we can make sure we are querying the same one as before. I don't know if this assumption will hold on all platforms and with all key systems, but it might work.

5. We could tear down the media stack and renegotiate from scratch with the new details. This would lead to an interruption in playback. Our seamless multi-period playback would no longer be seamless. As a slight improvement to this, we could keep track of all codec strings encountered in all manifest updates (even strings not currently present in the latest update of the manifest). Once we have seen all the codec strings a live stream is going to encounter (in your case, after one ad break and one content period), there would be no more need to tear down the media stack. This is probably the best option long-term, since it is the most robust.

I can see two issues on this solution. First one as you pointed out, there would be an interruption and this is definitely a bad experience for users. Second one, the implementation would require a bit work. The last one is that the stream might have more than 2 codec profile changes. For our case, we do not know all the ads have the same codec profiles. We need to discuss this with backend team.

I think we can abandon this idea, for all the reasons we discussed.

6. We could offer a configurable codec string list that would supplement the manifest strings. A service provider could reasonably know these in advance, and could tell us about them before we parse the manifest. This would avoid the need to tear down the media stack, and would allow us to query all necessary codec strings early. This is probably the best option short-term, since it is easy.

This might be tough. As for our case, none of our ad segment may have the same codec profile. It may vary as the ad urls are inserted by a 3rd party company. If you think other options are not solid, we will need to discuss this with out back-end team.

Understood.

It sounds like re-querying MediaKeys might be the best solution, then. We would have to test it thoroughly on all supported platforms, though, so we need to come up with a good automated test for this.

[joeyparrish]

Since the discussion was quite long, and many options were discussed, here's a summary for @vaage or whoever else winds up implementing what we discussed here:

When new periods are introduced, we should filter them by re-querying MediaKeys instead of using the list of supported codec strings from the first query. We expect this should work based on the spec text, but we also need to write a good automated test to ensure it works on all supported platforms.

The test would be something roughly like this:

1. Test provides codec strings (and other necessary metadata) to DrmEngine. The codec strings should be fairly universally-supported, such as baseline h264 and aac.
2. DrmEngine negotiates a MediaKeys configuration. (If this fails, skip the test. EME may be unavailable, or the key system may be unavailable.)
3. DrmEngine obtains a MediaKeys instance. (What we want to ensure is that we can still negotiate again after an instance is created, so this is critical.)
4. Test gets the list of supported codec strings from DrmEngine and stores them.
5. Test provides updated codec strings and metadata to DrmEngine. The updated strings are a superset of the first ones, such as higher level profiles of h264.
6. DrmEngine negotiates another MediaKeys configuration. (If this fails, fail the test. A failure here means that we can't renegotiate after an instance is created.)
7. Test checks that the supported codec strings reported by DrmEngine are a non-strict superset of the first codec strings. (If this fails, fail the test. A failure here means that we may have gotten a config for a different CDM implementation that supports different things. We are assuming that key system id + robustness maps to a specific implementation. Although this check can't always detect a different CDM implementation, it is a good start and increases our confidence in our assumptions.)

Repeat this test for:

1. Widevine, video & audio robustness set to SW_SECURE_CRYPTO for systems with software-based DRM.
2. Widevine, video & audio robustness set to HW_SECURE_ALL for systems with hardware-based DRM.
3. PlayReady, robustness strings empty.

[SemihGk]

I really appreciated your detailed feedback. I am glad that you are also comfortable with the re-querying solution. So, I just wanted to share a quick update. I tested this possible re-querying solution on my fork (it might be a bit messy though, but very simple) . This solution seems to be working on Chrome and Tizen. I will continue testing on other platforms to see if I can find any issue on other platforms as well. Thanks a lot.

[vaage]

@SemihGk @joeyparrish , Just so you both know, I have been following along. Will follow through on @joeyparrish recommended approach.

[SemihGk]

Thank you very much. I look forward to waiting for your fix. Meanwhile, I tested this approach and it is working on Tizen 17/18, LG 18 ( which those only support one video element at once), Chrome and Mozilla. Thus, this solution seems very safe to go.