feat(hls): Support AES-128 in HLS TS segments. #3880

wjywbs · 2022-01-19T22:29:03Z

Description

Part of #850. Use with mux.js:

<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/mux.min.js"></script>

These demo videos can play:
https://playertest.longtailvideo.com/adaptive/oceans_aes/oceans_aes.m3u8 (No IV)
https://bitmovin-a.akamaihd.net/content/art-of-motion_drm/m3u8s/11331.m3u8 (Const IV)

Key rotation (and maybe discontinuity) is not supported. (Existing TODO: May still need changes to support key rotation)

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to
not work as expected)
This change requires a documentation update

Checklist:

I have signed the Google CLA https://cla.developers.google.com
My code follows the style guidelines of this project
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
I have verified my change on multiple browsers on different platforms
I have run ./build/all.py and the build passes
I have run ./build/test.py and all tests pass

avelad · 2022-01-20T06:30:41Z

lib/hls/hls_parser.js

+          throw new shaka.util.Error(
+              shaka.util.Error.Severity.CRITICAL,
+              shaka.util.Error.Category.MANIFEST,
+              shaka.util.Error.Code.HLS_KEYFORMATS_NOT_SUPPORTED);


I think it is better to add a new error code for this case.

Thanks. Done.

avelad · 2022-01-20T06:31:24Z

lib/hls/hls_parser.js

+            throw new shaka.util.Error(
+                shaka.util.Error.Severity.CRITICAL,
+                shaka.util.Error.Category.MANIFEST,
+                shaka.util.Error.Code.HLS_KEYFORMATS_NOT_SUPPORTED);


I think it is better to add a new error code for this case. Because then there is a better explanation of what is happening.

avelad · 2022-01-20T06:31:33Z

lib/hls/hls_parser.js

+          throw new shaka.util.Error(
+              shaka.util.Error.Severity.CRITICAL,
+              shaka.util.Error.Category.MANIFEST,
+              shaka.util.Error.Code.HLS_KEYFORMATS_NOT_SUPPORTED);


I think it is better to add a new error code for this case.

avelad · 2022-01-20T06:33:32Z

lib/hls/hls_parser.js

+        throw new shaka.util.Error(
+            shaka.util.Error.Severity.CRITICAL,
+            shaka.util.Error.Category.MANIFEST,
+            shaka.util.Error.Code.HLS_AES_128_ENCRYPTION_NOT_SUPPORTED);


Perhaps it would be good to add an extra parameter that says the mimetype is not supported.

throw new shaka.util.Error( shaka.util.Error.Severity.CRITICAL, shaka.util.Error.Category.MANIFEST, shaka.util.Error.Code.HLS_AES_128_ENCRYPTION_NOT_SUPPORTED, mimeType);

I changed to the same error handling as above for raw and video/webm.

avelad · 2022-01-20T06:38:45Z

lib/hls/hls_parser.js

@@ -2145,6 +2197,8 @@ shaka.hls.HlsParser = class {
        },
    );

+    // If AES-128 is used, this option should be enabled to decrypt the full
+    // segment.
    if (this.config_.hls.useFullSegmentsForStartTime) {


I think that instead of doing this configuration manually, if it is detected that the stream is AES-128, the entire segment should always be requested automatically.

avelad · 2022-01-20T13:57:16Z

@wjywbs What would be missing for MP4 support?

wjywbs · 2022-01-20T14:02:27Z

Actually I don't have a MP4 AES test stream, and my use case doesn't need it yet.

wjywbs · 2022-01-20T14:12:57Z

From what I know, whether the mp4 init segment is encrypted depends on the order of EXT-X-MAP and EXT-X-KEY.

"If the #EXT-X-MAP tag follows #EXT-X-KEY tag, the initialization segment must also be encrypted."
https://developer.apple.com/forums/thread/131625
https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-10#section-4.4.4.4

avelad · 2022-01-20T14:20:32Z

https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-10#section-4.4.4.4

Media Segments MAY be encrypted. The EXT-X-KEY tag specifies how to
decrypt them. It applies to every Media Segment and to every Media
Initialization Section declared by an EXT-X-MAP tag that appears
between it and the next EXT-X-KEY tag in the Playlist file with the
same KEYFORMAT attribute or a METHOD of NONE (or the end of the
Playlist file). Any Media Segment or Media Initialization
Section that precedes the first EXT-X-KEY tag is unencrypted. Two or
more EXT-X-KEY tags with different KEYFORMAT attributes MAY apply to
the same Media Segment if they ultimately produce the same decryption
key

joeyparrish

Thanks so much for your contribution!

joeyparrish · 2022-01-28T19:40:58Z

externs/shaka/manifest.js

+ *
+ * @exportDoc
+ */
+shaka.extern.ClearKey;


I think the biggest issue I have with this PR is this name. ClearKey is an open, EME-compatible key system. AES-128 in HLS is not the same thing at all. ClearKey and other EME key systems are sample-encrypted, and AES-128 in HLS is whole-segment encryption.

If these were compatible, we could implement HLS's AES-128 encryption with a ClearKey CDM, but instead we have to use WebCrypto.

Similarly, to implement ClearKey DRM with WebCrypto, we would have to parse the segments to determine which parts to decrypt.

I know this is a confusing area of the media ecosystem, but I think it would be best to find another name for this type. How about something like shaka.extern.HlsAes128Key?

Sure. Changed.

joeyparrish · 2022-01-28T19:43:45Z

externs/shaka/manifest.js

@@ -332,6 +360,9 @@ shaka.extern.CreateSegmentIndexFunction;
 *   The stream's key IDs as lowercase hex strings. These key IDs identify the
 *   encryption keys that the browser (key system) can use to decrypt the
 *   stream.
+ * @property {(shaka.extern.ClearKey|undefined)} clearKey
+ *   <i>Defaults to undefined (i.e., no AES-128 encryption).</i> <br>


Since EME-compatible DRM also uses AES-128 (in CTR or CBC mode, encrypted at the sample level), this is ambiguous. We should reference HLS here for clarity, since this is specifically HLS's AES-128-CBC, full-segment encryption.

joeyparrish · 2022-01-28T19:45:25Z

lib/hls/hls_parser.js

    /** @type {!Array.<shaka.extern.DrmInfo>}*/
    const drmInfos = [];
    const keyIds = new Set();

-    // TODO: May still need changes to support key rotation.
+    /* eslint-disable no-await-in-loop */
+    // TODO: May still need changes to support key rotation and discontinuity.


If the keys can change across a discontinuity, then the key and IV need to be added to the SegmentReference. We can use the same object reference for key data in several SegmentReference objects. This is similar to how we handle init segments today.

If we will need to change the structure in a backward-incompatible way to handle discontinuities, I would prefer to make that change before we merge it.

Yes, I see other players store key info per segment, so key rotation can be handled. I'm checking how to add key info to SegmentReference, but likely only for AES-128. DRM is handled outside of segment for a long time, so I don't know how much change is needed to support DRM key rotation, or how unencrypted discontinuity is currently handled in a DRM stream.

Could you explain how unencrypted discontinuity is currently handled in a DRM stream? I could follow the same process.

Discontinuity results in two side-by-side references having certain parameters change. For example, different init segment references.

When StreamingEngine sees a change in init segment, timestamp offset, etc., it will handle that by adjusting offsets, fetching new init segments, etc.

So if a key can vary from segment to segment, you could put the key structure in SegmentReference instead of Stream. Then StreamingEngine becomes responsible for fetching new keys, just like it would fetch new init segments. (It is also smart enough not to fetch the same init segment over and over, so you should handle that for keys, too.)

joeyparrish · 2022-01-28T19:48:26Z

lib/hls/hls_parser.js

    /** @type {!Array.<shaka.extern.DrmInfo>}*/
    const drmInfos = [];
    const keyIds = new Set();

-    // TODO: May still need changes to support key rotation.
+    /* eslint-disable no-await-in-loop */


You can use eslint-disable-next-line and move this to just above the await statement.

joeyparrish · 2022-01-28T19:49:40Z

lib/hls/hls_parser.js

@@ -2080,14 +2133,15 @@ shaka.hls.HlsParser = class {
   * @param {string} type
   * @param {string} codecs
   * @param {number|undefined} bandwidth
+   * @param {(shaka.extern.ClearKey|undefined)} clearKey


JFYI, we're in process of removing the whole system of fetching and parsing segments for timestamps. We're switching to MSE sequence mode for HLS, which will both eliminate this code and add support for containerless formats (like raw AAC). If that change is merged first, you'll need to deal with merge conflicts here.

Sure, that may simplify the aes-128 process here.

That work has now been merged.

I rebased the change.

joeyparrish · 2022-01-28T20:04:47Z

lib/hls/hls_parser.js

@@ -2287,6 +2345,14 @@ shaka.hls.HlsParser = class {
    }

    if (mimeType == 'video/mp4' || mimeType == 'audio/mp4') {
+      if (clearKey) {
+        shaka.log.alwaysWarn(
+            'AES-128 in MP4 HLS is not yet supported.  Skipping ' + mimeType);


I would love to get full support without regard for type. If the concern is that order of tags might change whether a given segment (init segment, etc) is encrypted, then perhaps moving the encryption data to SegmentReference would help. See my comments above regarding discontinuities.

Could this be deferred to a future contribution? I don't have an HLS AES MP4 test stream, and it's less popular than AES TS.

Can you add some tests to the demo?

Added "Art of Motion".

avelad · 2022-02-08T16:57:03Z

lib/hls/hls_parser.js

+        const keyUri = shaka.hls.Utils.constructAbsoluteUri(
+            this.masterPlaylistUri_, drmTag.getRequiredAttrValue('URI'));
+        // eslint-disable-next-line no-await-in-loop
+        const keyResponse = await this.requestManifest_(keyUri);


I think this is not correct, it seems that you are trying to request a manifest, when a license request should be made. Ex: https://github.com/google/shaka-player/blob/master/lib/media/drm_engine.js#L1380

It's not a DRM license request. Otherwise some existing DRM license wrapping code may break.
https://shaka-player-demo.appspot.com/docs/api/tutorial-fairplay.html

So it would be best to create a new type, see: https://shaka-player-demo.appspot.com/docs/api/lib_net_networking_engine.js.html#line738
So in the application we can use specific filters for this.
@wjywbs what do you think?

I hope @joeyparrish could provide some input on this.

I think filters need to be aware of what they are playing. License-wrapping filters, for example, shouldn't be applied to content that doesn't use licenses.

That said, it's trivial to add another request type (such as KEY or something) that is distinct from LICENSE, to make the lives of developers somewhat easier.

I added KEY request type.

avelad · 2022-02-25T20:11:19Z

@joeyparrish can you review it? Thanks!

avelad · 2022-03-07T20:09:45Z

@joeyparrish is it possible include it in 4.0? Thanks!

avelad · 2022-03-14T16:28:36Z

@joeyparrish up! Thanks!

joeyparrish · 2022-03-22T16:00:26Z

@wjywbs, @avelad, sorry I lost track of this. I'll look at it this week, hopefully today.

avelad · 2022-03-29T15:23:22Z

@joeyparrish Please don't forget this PR

joeyparrish · 2022-03-30T23:29:59Z

Thanks, I won't forget. Though I will miss my own estimates/deadlines. :-(

test/hls/hls_parser_unit.js

externs/shaka/manifest.js

avelad · 2022-04-11T16:29:14Z

Ping @joeyparrish / @theodab

avelad · 2022-04-12T19:47:24Z

I hope to see it in 4.0 😄

joeyparrish · 2022-04-12T20:27:32Z

lib/hls/hls_parser.js

    /** @type {!Array.<shaka.extern.DrmInfo>}*/
    const drmInfos = [];
    const keyIds = new Set();

-    // TODO: May still need changes to support key rotation.
+    /* eslint-disable no-await-in-loop */
+    // TODO: May still need changes to support key rotation and discontinuity.


Discontinuity results in two side-by-side references having certain parameters change. For example, different init segment references.

When StreamingEngine sees a change in init segment, timestamp offset, etc., it will handle that by adjusting offsets, fetching new init segments, etc.

So if a key can vary from segment to segment, you could put the key structure in SegmentReference instead of Stream. Then StreamingEngine becomes responsible for fetching new keys, just like it would fetch new init segments. (It is also smart enough not to fetch the same init segment over and over, so you should handle that for keys, too.)

lib/media/streaming_engine.js

joeyparrish · 2022-04-21T20:16:13Z

The biggest thing holding me back from merging this is that it seems like this will not work with key rotation or discontinuities. It seems like those things will require the key metadata to be attached to SegmentReferences. I don't want to merge it as-is, only to have to make a breaking change to the structure later to support those things.

I would also like to have support for MP4 content, as well, but that could reasonably be done in a follow-up PR.

joeyparrish · 2022-04-21T20:17:48Z

If we can at least resolve the discontinuity issue in the next week or so, I could hold v4.0 for this. Otherwise, I think this may slip to v4.1+. Sorry for any inconvenience for those who are waiting for this.

avelad · 2022-05-03T10:25:14Z

@wjywbs , can you review the latest comments? (and resolve conflicts). Thanks!

Part of shaka-project#850.

wjywbs · 2022-05-04T23:21:51Z

I constructed a few discontinuity test cases and found that discontinuity in DRM HLS streams didn't work either. Both DRM and HLS AES key infos should be moved to SegmentReferences, but I'm not sure how large the change would be, or how feasible to do these in this change.

These 2 examples can play in video.js but can't play in shaka:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:8
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MAP:URI="h264_1080p.mp4",BYTERANGE="1397@0"
#EXT-X-KEY:METHOD=SAMPLE-AES,URI="data:text/plain;base64,AAAA....",KEYID=...,IV=...,KEYFORMATVERSIONS="1",KEYFORMAT="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed"
#EXTINF:4.004,
#EXT-X-BYTERANGE:750258@2233
h264_1080p.mp4
#EXTINF:4.004,
#EXT-X-BYTERANGE:863282
h264_1080p.mp4

#EXT-X-DISCONTINUITY
#EXT-X-KEY:METHOD=NONE
#EXT-X-MAP:URI="h264_1080p1.mp4",BYTERANGE="730@0"
#EXTINF:4.004,
#EXT-X-BYTERANGE:749437@1566
h264_1080p1.mp4

#EXT-X-DISCONTINUITY
#EXT-X-MAP:URI="h264_1080p.mp4",BYTERANGE="1397@0"
#EXT-X-KEY:METHOD=SAMPLE-AES,URI="data:text/plain;base64,AAAA....",KEYID=...,IV=...,KEYFORMATVERSIONS="1",KEYFORMAT="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed"

#EXTINF:4.004,
#EXT-X-BYTERANGE:943495@1615773
h264_1080p.mp4
#EXTINF:4.004,
#EXT-X-BYTERANGE:875954
h264_1080p.mp4
#EXT-X-ENDLIST

#EXTM3U

#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:4
#EXT-X-KEY:METHOD=AES-128,URI="encryption.key",IV=...

#EXTINF:4.0
segment_0_v.ts

#EXT-X-DISCONTINUITY
#EXT-X-KEY:METHOD=NONE
#EXTINF:6
test_90_v.ts

#EXT-X-DISCONTINUITY
#EXT-X-KEY:METHOD=AES-128,URI="encryption.key",IV=...

#EXTINF:4.0
segment_1_v.ts
#EXTINF:4.0
segment_2_v.ts
#EXTINF:4.0
segment_3_v.ts
#EXTINF:4.0
segment_4_v.ts
#EXTINF:4.0
segment_5_v.ts
#EXT-X-ENDLIST

Note: HLS AES-128 specific, if EXT-X-KEY does not have IV attribute, the Media Sequence Number (EXT-X-MEDIA-SEQUENCE) is used as IV. Since EXT-X-MEDIA-SEQUENCE can only appear once in the manifest, the packager needs to account for the Media Sequence Number increase of discontinuity segments.

magiskdev · 2022-06-02T11:31:58Z

Hello, will there be support for AES 128 for HLS MP4?

joeyparrish · 2022-06-07T19:49:11Z

Ultimately, yes. This PR doesn't provide MP4 support, though.

avelad · 2022-06-27T10:31:45Z

@wjywbs I couldn't find links to AES-128 with mp4, but I have seen someone on Exoplayer post examples with SAMPLE-AES.
I think this could be a help:
CBCS enc method in fMP4 : https://contents.pallycon.com/SHARE/cmaf_cbcs_clearkey/hls_ncg/master.m3u8
CENC enc method in fMP4 : https://contents.pallycon.com/SHARE/cmaf_cenc_clearkey_skipA/hls_ncg/master.m3u8
CBCS enc method in TS : https://contents.pallycon.com/SHARE/hls_cbcs_clearkey/hls_ncg/master.m3u8

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

tim-edgren · 2022-08-02T18:37:49Z

Awaiting this feature with breathless anticipation. Any hope of this making it into the next release?

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

theodab · 2022-08-03T22:08:14Z

@tim-edgren We now have an updated version of this PR (#4386) in review. So it's hopefully not going to be very long anymore.

joeyparrish · 2022-08-05T20:35:58Z

@wjywbs, I'm closing this now in favor of @theodab's version, which includes MP4 support and key rotation. You're being credited as a co-author, though, with a link back to this PR from the description. Thanks for getting this started!

Expands on the original PR (#3880) by adding support for MP4 and key rotation. Close #850 Co-authored-by: wjywbs <[email protected]>

Expands on the original PR (shaka-project#3880) by adding support for MP4 and key rotation. Close shaka-project#850 Co-authored-by: wjywbs <[email protected]>

wjywbs mentioned this pull request Jan 19, 2022

Support AES-128 on HLS #850

Closed

wjywbs marked this pull request as ready for review January 20, 2022 00:36

avelad reviewed Jan 20, 2022

View reviewed changes

joeyparrish requested changes Jan 28, 2022

View reviewed changes

wjywbs force-pushed the aes branch from 447f3f8 to 7018d07 Compare February 8, 2022 16:49

avelad reviewed Feb 8, 2022

View reviewed changes

wjywbs force-pushed the aes branch from 7018d07 to 541c5ea Compare February 19, 2022 03:55

wjywbs changed the title ~~Support AES-128 in HLS TS segments.~~ feat(hls): Support AES-128 in HLS TS segments. Feb 19, 2022

theodab requested changes Mar 31, 2022

View reviewed changes

test/hls/hls_parser_unit.js Outdated Show resolved Hide resolved

externs/shaka/manifest.js Show resolved Hide resolved

wjywbs force-pushed the aes branch from c397c69 to 4935145 Compare March 31, 2022 18:38

wjywbs requested a review from theodab March 31, 2022 18:54

joeyparrish requested changes Apr 12, 2022

View reviewed changes

avelad added type: enhancement New feature or request component: HLS The issue involves Apple's HLS manifest format labels May 2, 2022

Support AES-128 in HLS TS segments.

d5e4a61

Part of shaka-project#850.

wjywbs force-pushed the aes branch from 4935145 to d5e4a61 Compare May 4, 2022 22:32

joeyparrish assigned theodab Jul 19, 2022

theodab added a commit to theodab/shaka-player that referenced this pull request Jul 29, 2022

feat(hls): Support AES-128 in HLS

68a5413

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

theodab added a commit to theodab/shaka-player that referenced this pull request Jul 29, 2022

feat(hls): Support AES-128 in HLS

952620a

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

theodab added a commit to theodab/shaka-player that referenced this pull request Jul 29, 2022

feat(hls): Support AES-128 in HLS

213ca20

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

theodab added a commit to theodab/shaka-player that referenced this pull request Aug 3, 2022

feat(hls): Support AES-128 in HLS

b706539

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

theodab added a commit to theodab/shaka-player that referenced this pull request Aug 3, 2022

feat(hls): Support AES-128 in HLS

73b59b3

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

theodab added a commit to theodab/shaka-player that referenced this pull request Aug 3, 2022

feat(hls): Support AES-128 in HLS

235f587

Also adds support for key rotation in HLS, but only for AES-128. Based on shaka-project#3880 Issue shaka-project#850

theodab mentioned this pull request Aug 3, 2022

feat(hls): Support AES-128 in HLS #4386

Merged

joeyparrish closed this Aug 5, 2022

joeyparrish pushed a commit that referenced this pull request Aug 12, 2022

feat(hls): Support AES-128 in HLS (#4386)

6194021

Expands on the original PR (#3880) by adding support for MP4 and key rotation. Close #850 Co-authored-by: wjywbs <[email protected]>

github-actions bot added the status: archived Archived and locked; will not be updated label Jul 25, 2023

github-actions bot locked as resolved and limited conversation to collaborators Jul 25, 2023

feat(hls): Support AES-128 in HLS TS segments. #3880

feat(hls): Support AES-128 in HLS TS segments. #3880

Conversation

wjywbs commented Jan 19, 2022 • edited Loading

Description

Type of change

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

avelad commented Jan 20, 2022

wjywbs commented Jan 20, 2022

wjywbs commented Jan 20, 2022 • edited Loading

avelad commented Jan 20, 2022

joeyparrish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

avelad commented Feb 25, 2022

avelad commented Mar 7, 2022

avelad commented Mar 14, 2022 • edited Loading

joeyparrish commented Mar 22, 2022

avelad commented Mar 29, 2022

joeyparrish commented Mar 30, 2022

avelad commented Apr 11, 2022

avelad commented Apr 12, 2022

Choose a reason for hiding this comment

joeyparrish commented Apr 21, 2022

joeyparrish commented Apr 21, 2022

avelad commented May 3, 2022

wjywbs commented May 4, 2022

magiskdev commented Jun 2, 2022 • edited Loading

joeyparrish commented Jun 7, 2022

avelad commented Jun 27, 2022

tim-edgren commented Aug 2, 2022

theodab commented Aug 3, 2022

joeyparrish commented Aug 5, 2022

wjywbs commented Jan 19, 2022 •

edited

Loading

wjywbs commented Jan 20, 2022 •

edited

Loading

avelad commented Mar 14, 2022 •

edited

Loading

magiskdev commented Jun 2, 2022 •

edited

Loading