feat: Generate the entire AV1 codec string when the colr atom is present #1205

CaitlinOCallaghan · 2023-05-03T16:04:29Z

As per the AV1 spec, the codec string may contain optional color values. These color values are critical for detecting HDR video streams - see Issue 1007 and Issue 1202. This PR extracts the missing color information from the mp4 colr atom, if present, and generates the full AV1 codec string. Writing the colr information to the muxed mp4 will come in a later PR.

References:

Testing

Manual: AV1 video WITH colr atom

Note: Presence of the colr atom in hdr_av1_test.mp4 was verified with mp4dump

Generate Packager MPD with: ./out/Release/packager in=~/Downloads/bbbhdr_av1.mp4,stream=video,output=av1_with_colr.mp4 --mpd_output av1_with_colr.mpd
Check manifest generated by Shaka Packager for Representation codecs string:

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 31129ee-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT1.100000023841858S">
  <Period id="0">
    <AdaptationSet id="0" contentType="video" width="640" height="360" frameRate="15360/512" subsegmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="778146" codecs="av01.1.01M.10.0.000.09.16.09.0" mimeType="video/mp4" sar="1:1">
        <BaseURL>av1_with_colr.mp4</BaseURL>
        <SegmentBase indexRange="808-851" timescale="15360">
          <Initialization range="0-807"/>
        </SegmentBase>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

Generate FFMPEG MPD with: ffmpeg -i ~/Downloads/bbbhdr_av1.mp4 -c:v copy -an -seg_duration 2 -streaming 1 -utc_timing_url "https://time.akamai.com/?iso" -media_seg_name 'num_$Number%05d$.m4s' -init_seg_name 'num_init.m4s' -use_timeline 0 -f dash av1_with_colr.mpd
Check manifest generated by FFMPEG for Representation codecs string:

<?xml version="1.0" encoding="utf-8"?>
<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns="urn:mpeg:dash:schema:mpd:2011"
	xmlns:xlink="http://www.w3.org/1999/xlink"
	xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
	profiles="urn:mpeg:dash:profile:isoff-live:2011"
	type="static"
	mediaPresentationDuration="PT1.1S"
	minBufferTime="PT2.2S">
	<ProgramInformation>
		<Title>Big Buck Bunny, Sunflower version</Title>
	</ProgramInformation>
	<Period id="0" start="PT0.0S">
		<AdaptationSet id="0" contentType="video" segmentAlignment="true" bitstreamSwitching="true" lang="eng">
			<Representation id="0" mimeType="video/mp4" codecs="av01.1.01M.10.0.000.09.16.09.0" bandwidth="775498" width="640" height="360" frameRate="30/1">
				<SegmentTemplate timescale="1000000" duration="2000000" availabilityTimeOffset="1.967" initialization="num_init.m4s" media="num_$Number%05d$.m4s" startNumber="1">
				</SegmentTemplate>
			</Representation>
		</AdaptationSet>
	</Period>
	<UTCTiming schemeIdUri="urn:mpeg:dash:utc:http-xsdate:2014" value="https://time.akamai.com/?iso"/>
</MPD>

Compare - both manifests have codecs="av01.1.01M.10.0.000.09.16.09.0" :)

Manual: AV1 video WITHOUT colr atom

Note: The lack of the colr atom in av1-BBB.mp4 was verified with mp4dump

Generate Packager MPD with: ./out/Release/packager in=~/Downloads/av1-BBB.mp4,stream=video,output=av1_NO_colr.mp4 --mpd_output av1_NO_colr.mpd
Check manifest generated by Shaka Packager for Representation codecs string:

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 31129ee-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT10S">
  <Period id="0">
    <AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="15360/512" subsegmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="847200" codecs="av01.0.08M.08" mimeType="video/mp4" sar="1:1">
        <BaseURL>av1_NO_colr.mp4</BaseURL>
        <SegmentBase indexRange="805-848" timescale="15360">
          <Initialization range="0-804"/>
        </SegmentBase>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

Generate FFMPEG MPD with: ffmpeg -i ~/Downloads/av1-BBB.mp4 -c:v copy -an -seg_duration 2 -streaming 1 -utc_timing_url "https://time.akamai.com/?iso" -media_seg_name 'num_$Number%05d$.m4s' -init_seg_name 'num_init.m4s' -f dash av1_NO_colr.mp4
Check manifest generated by FFMPEG for Representation codecs string:

<?xml version="1.0" encoding="utf-8"?>
<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns="urn:mpeg:dash:schema:mpd:2011"
	xmlns:xlink="http://www.w3.org/1999/xlink"
	xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
	profiles="urn:mpeg:dash:profile:isoff-live:2011"
	type="static"
	mediaPresentationDuration="PT10.0S"
	minBufferTime="PT20.0S">
	<ProgramInformation>
		<Title>Big Buck Bunny, Sunflower version</Title>
	</ProgramInformation>
	<Period id="0" start="PT0.0S">
		<AdaptationSet id="0" contentType="video" segmentAlignment="true" bitstreamSwitching="true" lang="und">
			<Representation id="0" mimeType="video/mp4" codecs="av01.0.08M.08" bandwidth="845199" width="1920" height="1080" frameRate="30/1">
				<SegmentTemplate timescale="15360" initialization="num_init.m4s" media="num_$Number%05d$.m4s" startNumber="1">
					<SegmentTimeline>
						<S t="0" d="153600" />
					</SegmentTimeline>
				</SegmentTemplate>
			</Representation>
		</AdaptationSet>
	</Period>
	<UTCTiming schemeIdUri="urn:mpeg:dash:utc:http-xsdate:2014" value="https://time.akamai.com/?iso"/>
</MPD>

Compare - both manifests have codecs="av01.0.08M.08" :)

Unittest

./out/Release/codecs_unittest
./out/Release/mp4_unittest

Format

git clang-format --style Chromium --binary /usr/bin/clang-format 80e024013df87a4bfeb265c8ea83cfa2a0c5db0f

sr1990 · 2023-05-04T03:24:01Z

packager/media/formats/mp4/mp4_media_parser.cc

+            codec_string = av1_config.GetCodecString(
+                entry.colr.color_primaries, entry.colr.transfer_characteristics,
+                entry.colr.matrix_coefficients,
+                entry.colr.video_full_range_flag);


I think transfer_characteristics should be set to entry.colr.transfer_characteristics after this line.
It is used to create VideoStreamInfo on line 720 with which MediaInfo is created in HlsNotifyMuxerListener::OnMediaStart and MediaPlaylist::GetVideoRange() uses this MediaInfo to add the VIDEO-RANGE tag.

Good catch! :) I made the change and a few other improvements (like using FOURCC vs string).

Do you think it makes sense to add an E2E test (https://github.com/shaka-project/shaka-packager/blob/main/packager/app/test/packager_test.py)?

joeyparrish · 2023-07-05T21:43:29Z

packager/media/formats/mp4/box_definitions_unittest.cc

+    colr->video_full_range_flag = 0;
+  }
+
+  void Modify(ColorParameters* colr) {


Is this used anywhere?

Nope! I'll remove it.

CaitlinOCallaghan · 2023-08-02T23:49:30Z

Rebased after strange issue with unrelated commits being pulled in.

This PR is an extension of the full AV1 codec string feature: [PR 1205](#1205) and relates to [Issue 1007](#1007) and [Issue 1202](#1202). As per the AV1 spec, the codec string may contain optional color values. These color values are critical for detecting HDR video streams - see [Issue 1007](#1007). Color information is extracted from the input mp4's `colr` atom and used to generate the full AV1 codec string. This PR preserves the color information by writing the `colr` atom to the muxed mp4. **References**: - [AV1 Codec ISO Media File Format Binding](https://aomediacodec.github.io/av1-isobmff/#codecsparam) - [AV1 Bitstream & Decoding Process Specification - Section 6.4.2 Color config semantics (page 117)](https://aomediacodec.github.io/av1-spec/av1-spec.pdf) - [QuickTime File Format Specification](https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-125526)

CaitlinOCallaghan changed the title ~~feature: Generate the entire AV1 codec string when the colr atom is present~~ feat: Generate the entire AV1 codec string when the colr atom is present May 3, 2023

CaitlinOCallaghan marked this pull request as ready for review May 3, 2023 16:12

sr1990 reviewed May 4, 2023

View reviewed changes

joeyparrish previously approved these changes Jul 5, 2023

View reviewed changes

CaitlinOCallaghan dismissed joeyparrish’s stale review via bb2697a July 24, 2023 00:14

sr1990 mentioned this pull request Jul 27, 2023

feat(DASH): Add video transfer characteristics. #1210

Merged

CaitlinOCallaghan mentioned this pull request Jul 27, 2023

feat: Write colr atom to muxed MP4 when present in input AV1 file #1251

Closed

CaitlinOCallaghan added 12 commits August 2, 2023 16:31

Working Version

b425e76

Cleanup

f267a95

basic unittests

162100a

more cleanup

4c79cb3

Add/amend comments

666cc80

Delete an extra new line

ccc9ba5

Improvements

cb24084

Cleanup unused method

393ef56

Working Version

7d4c539

Cleanup

73f022b

basic unittests

64d5f45

more cleanup

31c590a

CaitlinOCallaghan force-pushed the cocallaghan/av1-codec-string branch from b5d5f74 to 31c590a Compare August 2, 2023 23:48

cosmin approved these changes Aug 4, 2023

View reviewed changes

cosmin merged commit cc9a691 into shaka-project:main Aug 4, 2023
11 of 20 checks passed

This was referenced Aug 29, 2023

feat: Write colr atom to muxed mp4 | Iteration 2 CaitlinOCallaghan/shaka-packager#11

Merged

feat: Write colr atom to muxed mp4 #1261

Merged

github-actions bot added the status: archived Archived and locked; will not be updated label Oct 3, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Generate the entire AV1 codec string when the colr atom is present #1205

feat: Generate the entire AV1 codec string when the colr atom is present #1205

CaitlinOCallaghan commented May 3, 2023 •

edited

Loading

sr1990 May 4, 2023

CaitlinOCallaghan May 11, 2023

sr1990 May 23, 2023

joeyparrish Jul 5, 2023

CaitlinOCallaghan Jul 17, 2023

CaitlinOCallaghan commented Aug 2, 2023

feat: Generate the entire AV1 codec string when the colr atom is present #1205

feat: Generate the entire AV1 codec string when the colr atom is present #1205

Conversation

CaitlinOCallaghan commented May 3, 2023 • edited Loading

Testing

Manual: AV1 video WITH colr atom

Manual: AV1 video WITHOUT colr atom

Unittest

Format

sr1990 May 4, 2023

Choose a reason for hiding this comment

CaitlinOCallaghan May 11, 2023

Choose a reason for hiding this comment

sr1990 May 23, 2023

Choose a reason for hiding this comment

joeyparrish Jul 5, 2023

Choose a reason for hiding this comment

CaitlinOCallaghan Jul 17, 2023

Choose a reason for hiding this comment

CaitlinOCallaghan commented Aug 2, 2023

CaitlinOCallaghan commented May 3, 2023 •

edited

Loading