Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BBC /reel/video/ #21870

Closed
5 tasks done
toyotabedzrock opened this issue Jul 23, 2019 · 12 comments
Closed
5 tasks done

BBC /reel/video/ #21870

toyotabedzrock opened this issue Jul 23, 2019 · 12 comments

Comments

@toyotabedzrock
Copy link

Checklist

  • I'm reporting a broken site support
  • I've verified that I'm running youtube-dl version 2019.07.16
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.bbc.com/reel/video/p07f3mvf/stonewall-riots-i-wanted-to-kill-the-police-']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2019.07.16
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.10586
[debug] exe versions: ffmpeg N-93448-g5fceac1cdb, ffprobe N-93448-g5fceac1cdb
[debug] Proxy map: {}
[bbc] stonewall-riots-i-wanted-to-kill-the-police-: Downloading webpage
ERROR: no suitable InfoExtractor for URL https://www.bbc.co.uk/programmes/None
  File "__main__.py", line 19, in <module>
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\__init__.py", line 474, in main
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\__init__.py", line 464, in _real_main
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 2008, in download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 807, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 868, in process_ie_result
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 830, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 624, in report_error
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 586, in trouble

Description

ERROR: no suitable InfoExtractor for URL https://www.bbc.co.uk/programmes/None
Video url is https://www.bbc.com/reel/video/p07f3mvf/stonewall-riots-i-wanted-to-kill-the-police-
@hairycactus hairycactus mentioned this issue Dec 3, 2019
Closed
5 tasks
@Vangelis66 Vangelis66 mentioned this issue Feb 10, 2020
Closed
@ajj8
Copy link

ajj8 commented Mar 16, 2020

Fixed in #23415

@shahab-vahedi
Copy link

Fixed in #23415

I confirm. Just tried the bbc.py from the pull request and it works like a charm. Hopefully, it will be merged soon.

@DavidCWGA
Copy link

This seems broken again, in 2021.01.03. It fails at https://www.bbc.com/reel/video/p0931jtk/the-nuclear-lighthouses-built-by-the-soviets-in-the-arctic

$ youtube-dl --verbose "https://www.bbc.com/reel/video/p0931jtk/the-nuclear-lighthouses-built-by-the-soviets-in-the-arctic"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.bbc.com/reel/video/p0931jtk/the-nuclear-lighthouses-built-by-the-soviets-in-the-arctic']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.01.03
[debug] Git HEAD: dad7dc6a1
[debug] Python version 3.9.1 (CPython) - macOS-11.1-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[bbc] the-nuclear-lighthouses-built-by-the-soviets-in-the-arctic: Downloading webpage
ERROR: Unable to extract playlist data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/homebrew/Cellar/youtube-dl/2021.1.3/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 803, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/homebrew/Cellar/youtube-dl/2021.1.3/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 824, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/homebrew/Cellar/youtube-dl/2021.1.3/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/homebrew/Cellar/youtube-dl/2021.1.3/libexec/lib/python3.9/site-packages/youtube_dl/extractor/bbc.py", line 1174, in _real_extract
    self._search_regex(
  File "/usr/local/homebrew/Cellar/youtube-dl/2021.1.3/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 1010, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract playlist data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

@Vangelis66
Copy link

@DavidCWGA

There have been recent bbc.py related changes implemented since v2020.12.26,

c32a059 , and #23415 was never merged...

However, the workaround I posted here, 11 months ago, still works for your referenced reel clip:

https://www.bbc.com/reel/video/p0931jtk/the-nuclear-lighthouses-built-by-the-soviets-in-the-arctic => https://www.bbc.co.uk/programmes/p0931jtk

@DavidCWGA
Copy link

Thank you! That's very helpful.

@Vangelis66
Copy link

@ajj8 : Will you kindly consider reworking #23415 so as to apply on top of c32a059 ? 😺
Support for bbc.com/reel clips has been asked numerous times already but, sadly, hasn't been implemented yet by one of the main yt-dl devs 😭 ...

Best regards

@Vangelis66 Vangelis66 mentioned this issue Feb 10, 2021
Open
6 tasks
@dirkf
Copy link
Contributor

dirkf commented Feb 18, 2021 •
edited
Loading

This fragment of PR #23415 can be added to the current (well, 2021-02-04.1) master and the listed BBC Reel clip can then be downloaded:

--- youtube_dl/extractor/bbc.py.old
+++ youtube_dl/extractor/bbc.py
@@ -961,6 +961,16 @@
              r'videoId\s*:\s*["\'](%s)["\']' % self._ID_REGEX],
             webpage, 'vpid', default=None)
 
+       # bbc reel (e.g. https://www.bbc.com/reel/video/p07c6sb6/how-positive-thinking-is-harming-your-happiness)
+        initial_data = self._search_regex(
+            r'<script[^>]+id="initial-data"[^>]+data-json=\'(.+)\'>',
+            webpage, 'initial data', fatal=False, default=None)
+        if initial_data:
+            programme_id = self._search_regex(
+                r'"versionID":"(%s)"' % self._ID_REGEX,
+                unescapeHTML(initial_data),
+                'programme id', fatal=False, default=None)
+
         if programme_id:
             formats, subtitles = self._download_media_selector(programme_id)
             self._sort_formats(formats)

@dirkf dirkf mentioned this issue Feb 25, 2021
Closed
11 tasks
github-actions bot added a commit to hellopony/youtube-dl that referenced this issue Mar 2, 2021
@github-actions
[bot] AutoMerging: merge all upstream's changes:
953d334 
* https://github.com/ytdl-org/youtube-dl:
  [9c9media] fix extraction for videos with multiple ContentPackages(closes ytdl-org#28309)
  [bbc] correct catched exception type
  [bbc] add support for BBC Reel videos(closes ytdl-org#21870, closes ytdl-org#23660, closes ytdl-org#28268)
  release 2021.03.02
  [ChangeLog] Actualize [ci skip]
  [zdf] Rework extractors (closes ytdl-org#11606, closes ytdl-org#13473, closes ytdl-org#17354, closes ytdl-org#21185, closes ytdl-org#26711, closes ytdl-org#27068, closes ytdl-org#27930, closes ytdl-org#28198, closes ytdl-org#28199, closes ytdl-org#28274)
@Vangelis66
Copy link

@dirkf, @remitamine

While indeed e465b25 brought native support for BBC Reel videos to bbcIE, the tip/hack/workaround I posted previously in this thread is still valid in some cases if you're looking for a 720p50 HD encode, because current implementation maxes out at 540p25 SD encodes...

I just compiled locally from the master branch (snapshot 2021.03.02+3-git-20210302-g8f56907) and here are the results of
youtube-dl -F "https://www.bbc.com/reel/video/p0931jtk/the-nuclear-lighthouses-built-by-the-soviets-in-the-arctic" =>

[bbc] the-nuclear-lighthouses-built-by-the-soviets-in-the-arctic: Downloading we
bpage
[bbc] p0931jtm: Downloading media selection JSON
[bbc] p0931jtm: Downloading m3u8 information
[bbc] p0931jtm: Downloading m3u8 information
[bbc] p0931jtm: Downloading m3u8 information
[bbc] p0931jtm: Downloading m3u8 information
[bbc] p0931jtm: Downloading MPD manifest
[bbc] p0931jtm: Downloading MPD manifest
[bbc] p0931jtm: Downloading MPD manifest
[bbc] p0931jtm: Downloading MPD manifest
[info] Available formats for p0931jtm:
format code                     extension  resolution note
mf_akamai-audio_eng=96000-0     m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_akamai-audio_eng=96000-1     m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_limelight-audio_eng=96000-0  m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_limelight-audio_eng=96000-1  m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_akamai-video=86000-0         mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=86000-1         mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=86000-0      mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=86000-1      mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=156000-0        mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=156000-1        mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=156000-0     mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=156000-1     mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=281000-0        mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=281000-1        mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=281000-0     mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=281000-1     mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=437000-0        mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D4015, 25fps, video only
mf_akamai-video=437000-1        mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D4015, 25fps, video only
mf_limelight-video=437000-0     mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D4015, 25fps, video only
mf_limelight-video=437000-1     mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D4015, 25fps, video only
mf_akamai-video=827000-0        mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_akamai-video=827000-1        mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_limelight-video=827000-0     mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_limelight-video=827000-1     mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_akamai-video=1604000-0       mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.64001F, 25fps, video only
mf_akamai-video=1604000-1       mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.64001F, 25fps, video only
mf_limelight-video=1604000-0    mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.64001F, 25fps, video only
mf_limelight-video=1604000-1    mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.64001F, 25fps, video only
mf_akamai-0                     mp4        256x144     224k , h264
mf_akamai-1                     mp4        256x144     224k , h264
mf_limelight-0                  mp4        256x144     224k , h264
mf_limelight-1                  mp4        256x144     224k , h264
mf_akamai-349-0                 mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_akamai-349-1                 mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_limelight-349-0              mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_limelight-349-1              mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_akamai-2                     mp4        448x252     543k , h264
mf_akamai-3                     mp4        448x252     543k , h264
mf_limelight-2                  mp4        448x252     543k , h264
mf_limelight-3                  mp4        448x252     543k , h264
mf_akamai-565-0                 mp4        512x288     565k , avc1.4D4015@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-565-1                 mp4        512x288     565k , avc1.4D4015@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-565-0              mp4        512x288     565k , avc1.4D4015@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-565-1              mp4        512x288     565k , avc1.4D4015@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-4                     mp4        640x360     800k , h264
mf_akamai-5                     mp4        640x360     800k , h264
mf_limelight-4                  mp4        640x360     800k , h264
mf_limelight-5                  mp4        640x360     800k , h264
mf_akamai-979-0                 mp4        704x396     979k , avc1.4D401F@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-979-1                 mp4        704x396     979k , avc1.4D401F@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-979-0              mp4        704x396     979k , avc1.4D401F@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-979-1              mp4        704x396     979k , avc1.4D401F@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-1802-0                mp4        960x540    1802k , avc1.64001F@1604k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-1802-1                mp4        960x540    1802k , avc1.64001F@1604k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-1802-0             mp4        960x540    1802k , avc1.64001F@1604k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-1802-1             mp4        960x540    1802k , avc1.64001F@1604k,
 25.0fps, mp4a.40.5@ 96k (best)

whereas
youtube-dl -F "https://www.bbc.co.uk/programmes/p0931jtk" =>

[bbc.co.uk] p0931jtk: Downloading video page
[bbc.co.uk] p0931jtm: Downloading media selection JSON
[bbc.co.uk] p0931jtm: Downloading m3u8 information
[bbc.co.uk] p0931jtm: Downloading m3u8 information
[bbc.co.uk] p0931jtm: Downloading MPD manifest
[bbc.co.uk] p0931jtm: Downloading MPD manifest
[bbc.co.uk] p0931jtm: Downloading m3u8 information
[bbc.co.uk] p0931jtm: Downloading m3u8 information
[bbc.co.uk] p0931jtm: Downloading MPD manifest
[bbc.co.uk] p0931jtm: Downloading MPD manifest
[info] Available formats for p0931jtm:
format code                        extension  resolution note
mf_akamai-audio_eng_1=128000-0     m4a        audio only [en] DASH audio  128k
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai-audio_eng_1=128000-1     m4a        audio only [en] DASH audio  128k
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_limelight-audio_eng_1=128000-0  m4a        audio only [en] DASH audio  128k
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_limelight-audio_eng_1=128000-1  m4a        audio only [en] DASH audio  128k
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai-video=827000-0           mp4        704x396    DASH video  827k , mp4
dash container, avc3.4D401F, 25fps, video only
mf_akamai-video=827000-1           mp4        704x396    DASH video  827k , mp4
dash container, avc3.4D401F, 25fps, video only
mf_limelight-video=827000-0        mp4        704x396    DASH video  827k , mp4
dash container, avc3.4D401F, 25fps, video only
mf_limelight-video=827000-1        mp4        704x396    DASH video  827k , mp4
dash container, avc3.4D401F, 25fps, video only
mf_akamai-video=1570000-0          mp4        704x396    DASH video 1570k , mp4
dash container, avc3.64001F, 50fps, video only
mf_akamai-video=1570000-1          mp4        704x396    DASH video 1570k , mp4
dash container, avc3.64001F, 50fps, video only
mf_limelight-video=1570000-0       mp4        704x396    DASH video 1570k , mp4
dash container, avc3.64001F, 50fps, video only
mf_limelight-video=1570000-1       mp4        704x396    DASH video 1570k , mp4
dash container, avc3.64001F, 50fps, video only
mf_akamai-video=2812000-0          mp4        960x540    DASH video 2812k , mp4
dash container, avc3.64001F, 50fps, video only
mf_akamai-video=2812000-1          mp4        960x540    DASH video 2812k , mp4
dash container, avc3.64001F, 50fps, video only
mf_limelight-video=2812000-0       mp4        960x540    DASH video 2812k , mp4
dash container, avc3.64001F, 50fps, video only
mf_limelight-video=2812000-1       mp4        960x540    DASH video 2812k , mp4
dash container, avc3.64001F, 50fps, video only
mf_akamai-video=5070000-0          mp4        1280x720   DASH video 5070k , mp4
dash container, avc3.640020, 50fps, video only
mf_akamai-video=5070000-1          mp4        1280x720   DASH video 5070k , mp4
dash container, avc3.640020, 50fps, video only
mf_limelight-video=5070000-0       mp4        1280x720   DASH video 5070k , mp4
dash container, avc3.640020, 50fps, video only
mf_limelight-video=5070000-1       mp4        1280x720   DASH video 5070k , mp4
dash container, avc3.640020, 50fps, video only
mf_akamai-1013-0                   mp4        704x396    1013k , avc1.4D401F@ 8
7k, 25.0fps, mp4a.40.2@128k
mf_akamai-1013-1                   mp4        704x396    1013k , avc1.4D401F@ 8
7k, 25.0fps, mp4a.40.2@128k
mf_limelight-1013-0                mp4        704x396    1013k , avc1.4D401F@ 8
7k, 25.0fps, mp4a.40.2@128k
mf_limelight-1013-1                mp4        704x396    1013k , avc1.4D401F@ 8
7k, 25.0fps, mp4a.40.2@128k
mf_akamai-1800-0                   mp4        704x396    1800k , avc1.64001F@15
0k, 50.0fps, mp4a.40.2@128k
mf_akamai-1800-1                   mp4        704x396    1800k , avc1.64001F@15
0k, 50.0fps, mp4a.40.2@128k
mf_limelight-1800-0                mp4        704x396    1800k , avc1.64001F@15
0k, 50.0fps, mp4a.40.2@128k
mf_limelight-1800-1                mp4        704x396    1800k , avc1.64001F@15
0k, 50.0fps, mp4a.40.2@128k
mf_akamai-3117-0                   mp4        960x540    3117k , avc1.64001F@28
2k, 50.0fps, mp4a.40.2@128k
mf_akamai-3117-1                   mp4        960x540    3117k , avc1.64001F@28
2k, 50.0fps, mp4a.40.2@128k
mf_limelight-3117-0                mp4        960x540    3117k , avc1.64001F@28
2k, 50.0fps, mp4a.40.2@128k
mf_limelight-3117-1                mp4        960x540    3117k , avc1.64001F@28
2k, 50.0fps, mp4a.40.2@128k
mf_akamai-5510-0                   mp4        1280x720   5510k , avc1.640020@50
0k, 50.0fps, mp4a.40.2@128k
mf_akamai-5510-1                   mp4        1280x720   5510k , avc1.640020@50
0k, 50.0fps, mp4a.40.2@128k
mf_limelight-5510-0                mp4        1280x720   5510k , avc1.640020@50
0k, 50.0fps, mp4a.40.2@128k
mf_limelight-5510-1                mp4        1280x720   5510k , avc1.640020@50
0k, 50.0fps, mp4a.40.2@128k (best)

Both tests from an overseas location... Just as a FYI...
Thanks to both for your coding efforts, much appreciated! 👍

@dirkf
Copy link
Contributor

dirkf commented Mar 3, 2021

Not fetching the highest resolution formats has been reported for other bbc.com URIs. I assume it's because the BBCIE InfoExtractor used for these URIs has different settings from those in the BBCCoUkIE extractor (eg _MEDIA_SETS). I would have guessed that this is because formats available through BBCIE are available worldwide while BBCCoUkIE aren't, but your test disproves that, at least in this case.

The recent PR extracts everything from the Reel page, as only a subset of the available metadata is provided through the programmes URI.

There is a technique that gets the best of both worlds. The programme's metadata can be populated from the Reel page (only a subset of this metadata is available through the programmes URI) and then passed to the downloader with the programmes URI as a transparent_url.

@dirkf dirkf mentioned this issue Mar 3, 2021
Open
11 tasks
@Vangelis66
Copy link

Vangelis66 commented Mar 3, 2021 •
edited
Loading

@dirkf

As I'm not a Python coder myself, I won't pollute the conversation in #28332; but, here are some things to consider:

Not ALL reel clips return 720p50 encodes when the equivalent programmes/pid endpoint is used;
you are right that the different results between current (as of 2021.03.03) implementation and the ones from the programmes/pid endpoint is the use in the code of different mediasets in the mediaselector API; mediaset=iptv-all will indeed return 720p50 encodes when available, but when not available will fall back to just 396p50; that's because the BBC, in some perverse way, also equates 540p50 to HD 😠 ...
So, with due respect, I don't agree with @remitamine that just the mediaset=iptv-all (present in bbc.co.ukIE) should be used for stream retrieval, because that way we lose 540p25 that the current mediaset in bbcIE is able to fetch (and in my eyes, I believe also to others, 540p25 > 396p50) !
Case in point, with yt-dl 2021.03.03:

youtube-dl -F "https://www.bbc.com/reel/video/p07c6sb6/how-positive-thinking-is-harming-your-happiness" =>

[bbc] how-positive-thinking-is-harming-your-happiness: Downloading webpage
[bbc] p07c6sb9: Downloading media selection JSON
[bbc] p07c6sb9: Downloading m3u8 information
[bbc] p07c6sb9: Downloading m3u8 information
[bbc] p07c6sb9: Downloading m3u8 information
[bbc] p07c6sb9: Downloading m3u8 information
[bbc] p07c6sb9: Downloading MPD manifest
[bbc] p07c6sb9: Downloading MPD manifest
[bbc] p07c6sb9: Downloading MPD manifest
[bbc] p07c6sb9: Downloading MPD manifest
[info] Available formats for p07c6sb9:
format code                     extension  resolution note
mf_akamai-audio_eng=96000-0     m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_akamai-audio_eng=96000-1     m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_limelight-audio_eng=96000-0  m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_limelight-audio_eng=96000-1  m4a        audio only [en] DASH audio   96k , m4
a_dash container, mp4a.40.5 (48000Hz)
mf_akamai-video=86000-0         mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=86000-1         mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=86000-0      mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=86000-1      mp4        192x108    DASH video   86k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=156000-0        mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=156000-1        mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=156000-0     mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=156000-1     mp4        256x144    DASH video  156k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=281000-0        mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=281000-1        mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=281000-0     mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_limelight-video=281000-1     mp4        384x216    DASH video  281k , mp4_das
h container, avc3.42C015, 25fps, video only
mf_akamai-video=437000-0        mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_akamai-video=437000-1        mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_limelight-video=437000-0     mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_limelight-video=437000-1     mp4        512x288    DASH video  437k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_akamai-video=827000-0        mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_akamai-video=827000-1        mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_limelight-video=827000-0     mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_limelight-video=827000-1     mp4        704x396    DASH video  827k , mp4_das
h container, avc3.4D401E, 25fps, video only
mf_akamai-video=1604000-0       mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_akamai-video=1604000-1       mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_limelight-video=1604000-0    mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_limelight-video=1604000-1    mp4        960x540    DASH video 1604k , mp4_das
h container, avc3.4D401F, 25fps, video only
mf_akamai-0                     mp4        256x144     224k , h264
mf_akamai-1                     mp4        256x144     224k , h264
mf_limelight-0                  mp4        256x144     224k , h264
mf_limelight-1                  mp4        256x144     224k , h264
mf_akamai-349-0                 mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_akamai-349-1                 mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_limelight-349-0              mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_limelight-349-1              mp4        384x216     349k , avc1.42C015@ 281k,
 25.0fps, mp4a.40.5@ 48k
mf_akamai-2                     mp4        448x252     543k , h264
mf_akamai-3                     mp4        448x252     543k , h264
mf_limelight-2                  mp4        448x252     543k , h264
mf_limelight-3                  mp4        448x252     543k , h264
mf_akamai-565-0                 mp4        512x288     565k , avc1.4D401E@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-565-1                 mp4        512x288     565k , avc1.4D401E@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-565-0              mp4        512x288     565k , avc1.4D401E@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-565-1              mp4        512x288     565k , avc1.4D401E@ 437k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-4                     mp4        640x360     800k , h264
mf_akamai-5                     mp4        640x360     800k , h264
mf_limelight-4                  mp4        640x360     800k , h264
mf_limelight-5                  mp4        640x360     800k , h264
mf_akamai-979-0                 mp4        704x396     979k , avc1.4D401E@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-979-1                 mp4        704x396     979k , avc1.4D401E@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-979-0              mp4        704x396     979k , avc1.4D401E@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-979-1              mp4        704x396     979k , avc1.4D401E@ 827k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-1802-0                mp4        960x540    1802k , avc1.4D401F@1604k,
 25.0fps, mp4a.40.5@ 96k
mf_akamai-1802-1                mp4        960x540    1802k , avc1.4D401F@1604k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-1802-0             mp4        960x540    1802k , avc1.4D401F@1604k,
 25.0fps, mp4a.40.5@ 96k
mf_limelight-1802-1             mp4        960x540    1802k , avc1.4D401F@1604k,
 25.0fps, mp4a.40.5@ 96k (best)

i.e. maxes at 540p25, while

youtube-dl -F "https://www.bbc.co.uk/programmes/p07c6sb6" =>

[bbc.co.uk] p07c6sb6: Downloading video page
[bbc.co.uk] p07c6sb9: Downloading media selection JSON
[bbc.co.uk] p07c6sb9: Downloading m3u8 information
[bbc.co.uk] p07c6sb9: Downloading m3u8 information
[bbc.co.uk] p07c6sb9: Downloading MPD manifest
[bbc.co.uk] p07c6sb9: Downloading MPD manifest
[bbc.co.uk] p07c6sb9: Downloading m3u8 information
[bbc.co.uk] p07c6sb9: Downloading m3u8 information
[bbc.co.uk] p07c6sb9: Downloading MPD manifest
[bbc.co.uk] p07c6sb9: Downloading MPD manifest
[info] Available formats for p07c6sb9:
format code                        extension  resolution note
mf_akamai-audio_eng_1=128000-0     m4a        audio only [en] DASH audio  128k ,
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai-audio_eng_1=128000-1     m4a        audio only [en] DASH audio  128k ,
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_limelight-audio_eng_1=128000-0  m4a        audio only [en] DASH audio  128k ,
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_limelight-audio_eng_1=128000-1  m4a        audio only [en] DASH audio  128k ,
 m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai-video=827000-0           mp4        704x396    DASH video  827k , mp4_
dash container, avc3.4D401E, 25fps, video only
mf_akamai-video=827000-1           mp4        704x396    DASH video  827k , mp4_
dash container, avc3.4D401E, 25fps, video only
mf_limelight-video=827000-0        mp4        704x396    DASH video  827k , mp4_
dash container, avc3.4D401E, 25fps, video only
mf_limelight-video=827000-1        mp4        704x396    DASH video  827k , mp4_
dash container, avc3.4D401E, 25fps, video only
mf_akamai-video=1570000-0          mp4        704x396    DASH video 1570k , mp4_
dash container, avc3.64001F, 50fps, video only
mf_akamai-video=1570000-1          mp4        704x396    DASH video 1570k , mp4_
dash container, avc3.64001F, 50fps, video only
mf_limelight-video=1570000-0       mp4        704x396    DASH video 1570k , mp4_
dash container, avc3.64001F, 50fps, video only
mf_limelight-video=1570000-1       mp4        704x396    DASH video 1570k , mp4_
dash container, avc3.64001F, 50fps, video only
mf_akamai-1013-0                   mp4        704x396    1013k , avc1.4D401E@ 82
7k, 25.0fps, mp4a.40.2@128k
mf_akamai-1013-1                   mp4        704x396    1013k , avc1.4D401E@ 82
7k, 25.0fps, mp4a.40.2@128k
mf_limelight-1013-0                mp4        704x396    1013k , avc1.4D401E@ 82
7k, 25.0fps, mp4a.40.2@128k
mf_limelight-1013-1                mp4        704x396    1013k , avc1.4D401E@ 82
7k, 25.0fps, mp4a.40.2@128k
mf_akamai-1800-0                   mp4        704x396    1800k , avc1.64001F@157
0k, 50.0fps, mp4a.40.2@128k
mf_akamai-1800-1                   mp4        704x396    1800k , avc1.64001F@157
0k, 50.0fps, mp4a.40.2@128k
mf_limelight-1800-0                mp4        704x396    1800k , avc1.64001F@157
0k, 50.0fps, mp4a.40.2@128k
mf_limelight-1800-1                mp4        704x396    1800k , avc1.64001F@157
0k, 50.0fps, mp4a.40.2@128k (best)

... i.e., as said above, maxes at 396p50 👎

My proposal is to use both mediasets, so that we can at least guarantee the 540p25 encodes in the absence of BBC HD (540p50+720p50) ...

@remitamine
Copy link
Collaborator

well, it appears that this would require more testing, the pc mediaset provides the best quality for the last two examples, will need to test with more examples later, making more than one request will be considered only if there isn't a mediaset that always provide the best quality.

@Vangelis66
Copy link

I highly appreciate your reply @remitamine 😃

Here's the rundown on things:

Case A: BBC provide HD (540p50+720p50) encodes for a given vPID;

pid=p0931jtk => vpid=p0931jtm
The pc mediaset will retrieve 540p25, 396p25, 288p25 & 216p25 streams:

http://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/pc/vpid/p0931jtm/format/xml (MPEG-DASH+HLS streams)

while the iptv-all mediaset will retrieve 720p50, 540p50, 396p50 & 396p25 streams:

http://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/iptv-all/vpid/p0931jtm/format/xml (MPEG-DASH+HLS streams)

Querying both mediasets, removing duplication and combining results, one gets the full list of:

720p50,540p50,540p25,396p50,396p25,288p25,216p25

Case B: BBC do not provide HD (540p50+720p50) encodes for a given vPID;

pid=p07c6sb6 => vpid=p07c6sb9
The pc mediaset will retrieve (as before) 540p25, 396p25, 288p25 & 216p25 streams:

http://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/pc/vpid/p07c6sb9/format/xml (MPEG-DASH+HLS streams)

while the iptv-all mediaset will only retrieve 396p50 & 396p25 streams:

http://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/iptv-all/vpid/p07c6sb9/format/xml (MPEG-DASH+HLS streams)

Again, querying both mediasets, removing duplication and combining results, one gets the full list of:
540p25,396p50,396p25,288p25,216p25

making more than one request will be considered
only if there isn't a mediaset that always provide the best quality.

Is there a programmatic reason we don't want yt-dl to always query both mediasets? I'm asking because that other, perl based, script (we're not supposed to mention here...) always queries all three mediasets (it includes apple-ipad-hls), so no available combination of resolution+framerate evades detection...

leshasmlesha pushed a commit to leshasmlesha/youtube-dl that referenced this issue Mar 7, 2021
@dirkf @leshasmlesha
[bbc] add support for BBC Reel videos(closes ytdl-org#21870, closes y…
02c42b7 
…tdl-org#23660, closes ytdl-org#28268)
leshasmlesha pushed a commit to leshasmlesha/youtube-dl that referenced this issue Mar 7, 2021
@dirkf @leshasmlesha
[bbc] add support for BBC Reel videos(closes ytdl-org#21870, closes y…
aecd640 
…tdl-org#23660, closes ytdl-org#28268)
This was referenced Mar 15, 2021
Closed
Closed
Closed
Closed
Merged
Closed
Closed
Closed
Closed
@dirkf dirkf mentioned this issue Mar 24, 2021
Open
5 tasks
leshasmlesha pushed a commit to leshasmlesha/youtube-dl that referenced this issue Apr 3, 2021
@dirkf @leshasmlesha
[bbc] add support for BBC Reel videos(closes ytdl-org#21870, closes y…
62f96de 
…tdl-org#23660, closes ytdl-org#28268)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@toyotabedzrock @dirkf @DavidCWGA @Vangelis66 @remitamine @shahab-vahedi @ajj8