Run quality measure via ffmpeg-qsv and/or Intel Media SDK Sample Multi-Transcode. There are 3 modes of operation to highlight.
-
Run quality measurement on the pre-defined set of workloads
measure quality <folder>
In this case
<folder>
sets the root folder to search for the pre-defined set of streams. Search is done in the root<folder>
and all subfolders. -
Run quality measurement for a specified stream
measure quality <stream.mp4> measure quality -w <width> -h <height> -f <framerate> <stream.yuv>
In this case the quality measure is run only for the specified stream. If the stream is in YUV format, then its characteristics must be specified via command line optins (
-w
,-h
,-f
, etc.). The script accepts only 8-bit I420 YUV input streams. If the input stream is uncompressed YUV or elementary compressed bitstream (e.g. standard compliant h264), quality will be measured using both ffmpeg-qsv and Media SDK Sample Multi-Transcode. On the other hand, a stream encapsulated in a container (e.g. MP4, TS, etc.) will be measured only using ffmpeg-qsv (since Sample Multi-Transcode does not support containers). -
Run quality measure for the pre-defined sequence
measure quality <pre-defined-sequence.yuv>
This is a simplified variant of the previous mode. If the stream name matches one of the following pre-defined streams you may skip specifying its characteristics:
Sequence Resolution FPS Frames MD5 Source BasketballDrive_1920x1080_50.yuv
1920x1080
50
500
e18034a26708a3c534a3b03d3bf82d61
MPEG Test Suite
BoatNF_1280x720_60.yuv
1280x720
60
300
45207fbd760394f011cff2af34d59ddc
bq_terrace_1920x1080p_600_60.yuv
1920x1080
60
600
cc17d5957b732ec5eab9234d6f5318e3
MPEG Test Suite
Bunny_1920x1080_24_600.yuv
1920x1080
24
600
987f1923ccf93d26271324b21c39ec45
Cactus_1920x1080_50.yuv
1920x1080
50
500
3fddb71486f209f1eb8020a0880ddf82
MPEG Test Suite
crowd_run_1280x720_50.yuv
1280x720
50
500
371e4d129556b27e17b1bc92c16a69d4
crowd_run_1920x1080p_500_50.yuv
1920x1080
50
500
da34812b5b2c316d40481c7b6c841e41
CSGO_1920x1080_60.yuv
1920x1080
60
600
5a7575d1c403a08347cffe88bcbc1805
DinnerScene_1920x1080_60.yuv
1920x1080
60
600
d1260db74160c61b72d7e1cee00e1ec2
DOTA2_1920x1080_60_600.yuv
1920x1080
60
600
a3a7d5e1c9964e5aa6f5e3e520320c32
FoodMarket2NF_1280x720_60.yuv
1280x720
60
300
f41cb6ddaaaae9fec392da4e2e47b07e
GTAV_1920x1080_60_600.yuv
1920x1080
60
600
22ad590c3f624ac0884062a68674ef4a
Hearthstone_1920x1080_60.yuv
1920x1080
60
600
d5eb7157f37386d5a2df0e789aed8909
Kimono1_1280x720_24.yuv
1280x720
24
240
e6bbaf876f00fe1709f5e8e1ec8da967
MPEG Test Suite
Kimono1_1920x1080_24.yuv
1920x1080
24
240
4a83005bc719012ac148dd3898e5e4ed
MPEG Test Suite
MINECRAFT_1920x1080_60_600.yuv
1920x1080
60
600
3bc4b5a002b5b4140e45bb0ded4a3620
MrFox_BlueBird_1920x1080_30.yuv
1920x1080
30
300
30801242685c4ed75c9eb748d5a4d0e7
VQEG Test Suite
park_joy_1280x720_50.yuv
1280x720
50
500
ef5868b66118c7fcbfdca069efdac684
park_joy_1920x1080_500_50.yuv
1920x1080
50
300
37dc2f9b6a2d1f4e50ac6cc432112733
ParkScene_1280x720_24.yuv
1280x720
24
240
d56b03ba9bf0afeac2800af9ab18c9eb
MPEG Test Suite
PierSeasideNF_1280x720_60.yuv
1280x720
60
600
ffd18a73e6d694097613cfd5228ec6c1
RedKayak_1920x1080_30.yuv
1920x1080
30
570
2901bec44d6f43af3e8316b94d8af02b
RushFieldCuts_1920x1080_30.yuv
1920x1080
30
570
055207f6a5819f3a1dc216a64f8634f9
Sintel_trailer_o537n480_1920x1080_24.yuv
1920x1080
24
480
1229ca7e98831ca85e6411e1bce12757
TangoNF_1280x720_60.yuv
1280x720
60
294
8ba856e08c3eefbe495a68f4df7ee0f5
touchdown_pass_1280x720_30.yuv
1280x720
30
570
db92db55a027922f7ea7276ae680f819
MPEG Test Suite
WITCHER3_1920x1080_60.yuv
1920x1080
60
600
cc082ec495a47085ba1c08b99e4de2e4
Script checks whether the input stream is YUV by the following file extensions: .yuv, .YUV. Any other file is considered to be an encoded bitstream (either raw or in a container).
Script default bitrates are given in the following tables:
-
Default bitrates for predefined sequences
Sequence
Resolution
High Quality Bitrates (Mb/s)
Low Delay Bitrates (Mb/s)
H.264/AVC
H.265/HEVC
AV1
H.264/AVC
H.265/HEVC
AV1
BasketBallDrive
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
BQTerrace
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
Cactus
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
CrowdRun
1920x1080
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
DinnerScene
1920x1080
1, 1.5, 2, 3, 4
3, 7, 11, 15, 20
3, 7, 11, 15, 20
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 7, 11, 15
Kimono
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3.5, 6, 9, 15
2, 3, 6, 9, 15
ParkJoy
1920x1080
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
15, 20, 25, 30, 35
RedKayak
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
9, 12, 15, 18, 22
2, 3, 6, 9, 15
RushFieldCuts
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
Boat
1280x720
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
CrowdRun
1280x720
6, 8, 10, 12, 15
6, 8, 10, 12, 15
6, 8, 10, 12, 15
3, 4.5, 7.5, 10, 12
3, 4.5, 7.5, 10, 12
3, 4.5, 7.5, 10, 12
FoodMarket
1280x720
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
Kimono
1280x720
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
ParkJoy
1280x720
6, 8, 10, 12, 15
6, 8, 10, 12, 15
6, 8, 10, 12, 15
3, 4.5, 7.5, 10, 12
3, 4.5, 7.5, 10, 12
3, 4.5, 7.5, 10, 12
ParkScene
1280x720
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
PierSeaSide
1280x720
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
Tango
1280x720
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
TouchDownPass
1280x720
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
Bunny
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
8, 9, 10, 11, 12
CSGO
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
1.5, 2, 3, 9, 15
DOTA2
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
GTAV
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
Hearthstone
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
Minecraft
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
MrFox_BlueBird
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
Sintel_o537n480
1920x1080
0.5, 1, 2, 6, 9
0.5, 1, 2, 6, 9
0.5, 1, 2, 6, 9
0.5, 1, 2, 6, 9
0.5, 1, 2, 6, 9
0.5, 1, 2, 6, 9
Witcher
1920x1080
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
-
Default bitrates for user-defined sequences (High Quality and Low Delay use cases):
Resolution
Bitrates (Mb/s)
H.264/AVC
H.265/HEVC
AV1
width > 1920 && height > 1088
6, 9, 15, 24, 40
6, 9, 15, 24, 40
6, 9, 15, 24, 40
width > 1280 && height > 720
2, 3, 6, 12, 24
2, 3, 6, 9, 15
2, 3, 6, 9, 15
other
1, 1.5, 3, 6, 12
1, 1.5, 3, 4.5, 7.5
1, 1.5, 3, 4.5, 7.5
For predefined sequences we have included precomputed golden references for BD-Rate calculation to remove the burden of having to compute them on the fly. The following table lists all currently available precomputed golden references along with the corresponding command line triggers:
-
Precomputed golden references:
Use case Codec Golden Reference Trigger High Quality
AVC
x264-medium (default)
--use-gold-ref
x264-veryslow
--use-gold-ref --ref-preset veryslow
HEVC
x265-medium (default)
--use-gold-ref
x265-veryslow
--use-gold-ref --ref-preset veryslow
AV1
x264-medium (default)
--use-gold-ref
x264-veryslow
--use-gold-ref --ref-preset veryslow
Low Delay
AVC
x264-medium-250ms (default)
--use-gold-ref
HEVC
x265-medium-250ms (default)
--use-gold-ref
AV1
x264-medium-500ms (default)
--use-gold-ref
x264-medium-250ms
--use-gold-ref --buffer-delay 250
The script outputs data in the --outdir
folder. Artifacts are encoded (transcoded)
streams and text files with the calculated metrics. Consider an example below
(--nframes=10
is given just for the simplicity).
# measure quality --nframes=10 /opt/data/embedded/WAR_TRAILER_HiQ_10_withAudio.mp4 <...> # ls -1 WAR_TRAILER_HiQ_10_withAudio.mp4.cbr.ffmpeg-qsv.metrics WAR_TRAILER_HiQ_10_withAudio.mp4.cbr.sample-multi-transcode.metrics WAR_TRAILER_HiQ_10_withAudio.mp4.vbr.ffmpeg-qsv.metrics WAR_TRAILER_HiQ_10_withAudio.mp4.vbr.sample-multi-transcode.metrics WAR_TRAILER_HiQ_10_withAudio.mp4_12Mbps_CBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_12Mbps_VBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_24Mbps_CBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_24Mbps_VBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_2Mbps_CBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_2Mbps_VBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_3Mbps_CBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_3Mbps_VBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_6Mbps_CBR_QSV.h264 WAR_TRAILER_HiQ_10_withAudio.mp4_6Mbps_VBR_QSV.h264 # cat WAR_TRAILER_HiQ_10_withAudio.mp4.cbr.ffmpeg-qsv.metrics WAR_TRAILER_HiQ_10_withAudio.mp4_12Mbps_CBR_QSV.h264:33712:99.73896508146471:56.57106818620179:0.9998045802116394:0.9996079494435991 WAR_TRAILER_HiQ_10_withAudio.mp4_24Mbps_CBR_QSV.h264:28605:99.73896508146471:55.73372554473565:0.9997134983539582:0.9994766423998278 WAR_TRAILER_HiQ_10_withAudio.mp4_2Mbps_CBR_QSV.h264:4681:98.18199953840941:49.40211511795687:0.9973068058490753:0.9969961779084621 WAR_TRAILER_HiQ_10_withAudio.mp4_3Mbps_CBR_QSV.h264:6927:99.14454456499617:50.71396113566765:0.9983673691749573:0.9979530665346188 WAR_TRAILER_HiQ_10_withAudio.mp4_6Mbps_CBR_QSV.h264:14923:99.68411421582299:53.147407625088:0.9992096841335296:0.9988773328701932
*.metrics
files contain calculated metrics. File names contain 4 parts:
<outprefix>.<codec>.<mode>.<application>.metrics
. <outprefix>
is
--outprefix
specified on the command line (running pre-defined set of streams
default one is always used). <codec>
is the video compression standard used
in quality measurement (AVC or HEVC or AV1), which can be specified on the command line with
--codec
option. <mode>
is a bitrate mode like cbr
(constant bitrate) or
vbr
(variable bitrate). application
is the application being measured for video
quality: ffmpeg or Intel Media SDK Sample Multi-Transcode. In addition, when BD-rate calculation
is enabled, application
is set to ref
indicating metrics files obtained from a
reference (e.g. x264 or x265) used for BD-rate calculation. Metrics data is in
CSV format with colon :
separator. The first field is a stream name, the second
is the bitrate, and the remaining fields are the metrics. Metrics are published in
the following order: VMAF, PSNR-Y, SSIM, MS-SSIM. If any of the --skip-*
metrics
options are specified, then the corresponding metrics are represented with an empty
string between the separators in the output report.
*bdrate
files contain calculated BD-rates. Bjøntegaard-Delta bitrate (BD-rate)
compares video quality between test and reference codecs by taking into account
bitrates and corresponding quality metrics (e.g. PSNR, VMAF, etc.). At least 4
(bitrate, metric) pairs (vector points) are needed for BD-rate calculation. The
BD-rate is a percentage indicating how much the bitrate is reduced (negative
percent values) or how much the bitrate is increased (positive percent values)
for the same value of a quality metric. The script supports 2 different BD-rate
modes:
-
Single BD-rate per sequence mode where a single BD-rate is computed from all available (bitrate, metric) vector points, for all enabled metrics. Minimum number of points required for BD-rate computation is 4.
-
Dual BD-rate per sequence mode where 2 BD-rates are computed per sequence: one from 4 points corresponding to the 4 highest bitrates (High Bitrates BD-rate), and one from 4 points corresponding to the 4 lowest bitrates (Low Bitrates BD-rate).
Dual mode is used by default. However, users can force Single BD-rate mode with
--single-bdrate
option. In addition, users are able to override default
bitrates with --bitrates
option. When overriding default bitrates, at least 4
distinct bitrates need to be specified. If N bitrates are specified (N >= 4) with
Dual BD-rate mode, then smallest 4 out of N bitrates are set as Low Bitrates, and
the highest 4 out of N bitrates are set as High Bitrates. Setting N < 8 is allowed
in which case there will be an overlap between Low and High bitrates. Also,setting
N > 8 is allowed, in which case mid-range bitrates will be ignored. On the other
hand, if Single BD-rate mode is used, then all N >= 4 bitrates are utilized.
*bdrate
file names contain 5 parts:
<outprefix>.<codec>.<mode>.<application>.<extension>
. The first 4 parts are
the same as in *.metrics
files. <extension>
can be one of the following:
-
bdrate
for Single BD-rate per sequence mode output -
hi-bdrate
for High Bitrates BD-rate mode output -
lo-bdrate
for Low Bitrates BD-rate mode output
After *bdrate
output files per sequence are generated, the script generates an
additional set of output files containing BD-rate averages. This set of output
files has the filename structure Average.<codec>.<mode>.<application>.<extension>
.
Additionally, the grand total average output file is generated for each enabled
application
and has the filename structure Average.<codec>.<application>.bdrate
.
In each *bdrate
file, the data is in CSV format with colon :
separator. The
first field is a stream name, followed by the BD-rate values for available metrics
published in the following order: VMAF BD-rate, PSNR-Y BD-rate, SSIM BD-rate,
MS-SSIM BD-rate. If any of the --skip-*
metrics options are specified, then the
BD-rates corresponding these metrics are represented with an empty string between
the separators in the output report.
- --codec AVC|HEVC|AV1
-
Sets an encoder to use (default:
AVC
). - --preset default|best
-
Sets a quality preset option (default:
best
). - --bitrates <float>[:<float>…]
-
Sets a list of bitrates in Mbps to use (default: use hardcoded bitrates which depend on stream resolution and codec). A minimum of 4 distinct bitrates are needed for calculating BD-rate.
- --tu veryslow|slower|slow|medium|fast|faster|veryfast
-
Sets a target usage preset (default:
medium
). - --buffer-delay <int>
-
Sets encoding buffer delay in milliseconds for low-delay mode (default:
250
for AVC/HEVC and500
for AV1). - --enctools-lad <int>
-
Sets the EncTools low power look ahead depth (default:
40
). To use EncTools with boosted performance, set depth to 8. Minimum value is 1. - --ref-preset veryslow|slower|slow|medium|fast|faster|veryfast
-
Sets a target usage preset for the reference encoder (default:
veryslow
). If low-delay mode is enabled, default is set tomedium
.
- --skip-metrics
-
Do not calculate any metrics.
- --skip-psnr
-
Do not calculate psnr.
- --skip-ssim
-
Do not calculate ssim.
- --skip-ms-ssim
-
Do not calculate msssim.
- --skip-vmaf
-
Do not calculate vmaf.
- --skip-encoding
-
Do not encode anything.
- --skip-ffmpeg
-
Do not run ffmpeg-qsv quality measurement.
- --skip-msdk
-
Do not run Intel Media SDK Sample Multi-Transcode quality measurement.
- --skip-cbr
-
Do not execute Constant Bitrate (CBR) encoding cases.
- --skip-vbr
-
Do not execute Variable Bitrate (VBR) encoding cases.
- --skip-reference
-
Do not run reference codec used for BD-rate calculation.
- --skip-bdrate
-
Do not calculate BD-rate.
- --single-bdrate
-
Force Single BD-rate per sequence mode (default is Dual, i.e. calculate BD-rates for lo- and hi-bitrate ranges).
- --use-vdenc
-
Force using low power VDEnc hardware mode (disabled by default).
- --use-gold-ref
-
Use precomputed golden references for BD-rate computation of predefined sequences (disabled by default).
- --use-enctools
-
Use EncTools BRC for encoding (disabled by default). If enabled, it forces low power VDEnc hardware mode.
- --use-lowdelay
-
Measure quality for low-delay/low-latency mode (disabled by default).
- --nframes|-n <uint>
-
Process (encode, calculate metrics) this number of frames and stop. If omitted or set to 0, all frames will be processed (default: 0).
- --dry-run
-
Do not execute any commands, but dump them to
stdout
. - --outdir|-o /path/to/artifacts
-
Generate output in the specified folder (default:
/opt/data/artifacts/measure/quality
if ran under docker,$HOME/measure/quality
otherwise) - --outprefix <string>
-
File prefix to append to output artifacts (default:
$(basename $inputfile)
). Not applicable in<folder>
mode.
Only valid for YUV input stream.
- --width|-w <uint>
-
Stream width
- --height|-h <uint>
-
Stream height
- --framerate|-f <uint>
-
Stream framerate
- ffmpeg
-
Used for quality measurement of ffmpeg-qsv (
--enable-libmfx
) and calculation of quality metrics (--enable-libvmaf
). - ffprobe
-
Used for getting information on the input/output stream(s).
- sample_multi_transcode
-
Used for direct quality measurement of Intel MediaSDK/OneVPL library.
- python3
-
Used for generic script purposes.