Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

youtube-dl downloading extremely slow (< 200KiB/s) #15271

Closed
4 tasks
zannix opened this issue Jan 16, 2018 · 22 comments
Closed
4 tasks

youtube-dl downloading extremely slow (< 200KiB/s) #15271

zannix opened this issue Jan 16, 2018 · 22 comments

Comments

@zannix
Copy link

zannix commented Jan 16, 2018

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.01.14. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • [x ] I've verified and I assure that I'm running youtube-dl 2018.01.14

Before submitting an issue make sure you have:

  • [x ] At least skimmed through the README, most notably the FAQ and BUGS sections
  • [x ] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • [x ] Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

root@srvr:/var/www/html# youtube-dl -v --format=140 -o songs/song11.m4a 'https://www.youtube.com/watch?v=4-BtZWGaPNY' 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--format=140', u'-o', u'songs/song11.m4a', u'https://www.youtube.com/watch?v=4-BtZWGaPNY']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2018.01.14
[debug] Python version 2.7.6 (CPython) - Linux-3.13.0-24-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.20-6, avprobe 9.20-6, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 4-BtZWGaPNY: Downloading webpage
[youtube] 4-BtZWGaPNY: Downloading video info webpage
[youtube] 4-BtZWGaPNY: Extracting video information
[youtube] {22} signature length 42.45, html5 player vflLCGcm0
[youtube] {43} signature length 42.45, html5 player vflLCGcm0
[youtube] {18} signature length 42.45, html5 player vflLCGcm0
[youtube] {36} signature length 42.45, html5 player vflLCGcm0
[youtube] {17} signature length 42.45, html5 player vflLCGcm0
[youtube] {136} signature length 42.45, html5 player vflLCGcm0
[youtube] {247} signature length 42.45, html5 player vflLCGcm0
[youtube] {298} signature length 42.45, html5 player vflLCGcm0
[youtube] {302} signature length 42.45, html5 player vflLCGcm0
[youtube] {135} signature length 42.45, html5 player vflLCGcm0
[youtube] {244} signature length 42.45, html5 player vflLCGcm0
[youtube] {134} signature length 42.45, html5 player vflLCGcm0
[youtube] {243} signature length 42.45, html5 player vflLCGcm0
[youtube] {133} signature length 42.45, html5 player vflLCGcm0
[youtube] {242} signature length 42.45, html5 player vflLCGcm0
[youtube] {160} signature length 42.45, html5 player vflLCGcm0
[youtube] {278} signature length 42.45, html5 player vflLCGcm0
[youtube] {140} signature length 42.45, html5 player vflLCGcm0
[youtube] {171} signature length 42.45, html5 player vflLCGcm0
[youtube] {249} signature length 42.45, html5 player vflLCGcm0
[youtube] {250} signature length 42.45, html5 player vflLCGcm0
[youtube] {251} signature length 42.45, html5 player vflLCGcm0
[debug] Invoking downloader on u'https://r5---sn-5hne6n7z.googlevideo.com/videoplayback?source=youtube&clen=2542652&ms=au&mt=1516109320&mv=m&id=o-AEMDEBbzlzj4SrO-hBEBcXGnvbTDkuN44VftfSf_imxk&dur=160.031&gir=yes&mime=audio%2Fmp4&mn=sn-5hne6n7z&ip=[MY_IP]&ei=iP5dWuSuDNvk1gKozqbIAg&requiressl=yes&itag=140&keepalive=yes&ipbits=0&initcwndbps=243750&pl=25&key=yt6&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cexpire&lmt=1507243774824422&expire=1516131048&mm=31&signature=80F2ECB5005CE0B0B493625E5E11F0B2DAC2C9F6.DF4FCAF5F8D7F3B5EE37A90EF6772D331317930F&ratebypass=yes'
[download] Destination: songs/song11.m4a
[download] 100% of 2.42MiB in 00:35
[ffmpeg] Correcting container in "songs/song11.m4a"
WARNING: Your copy of avconv is outdated, update avconv to version 10-0 or newer if you encounter any errors.
[debug] ffmpeg command line: avconv -y -i 'file:songs/song11.m4a' -c copy -f mp4 'file:songs/song11.temp.m4a
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

It was working fine until yesterday, but today the speed is ridiculously slow. I tried installing youtube-dl both locally and on my production server, but the speed remains painfully slow, hovering just above 100KiB/s.

I am running the newest version of youtube-dl, and I've pasted the log above. I don't know how to further debug this issue. Please help.

EDIT: running the following command:

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -

gives me results:

Testing download speed................................................................................
Download: 81.56 Mbit/s

and everything besides youtube-dl seems to be working within expected speed parameters.

EDIT2: sometimes running youtube-dl command simply produces output 'Segmentation fault'

EDIT3: adding --force-ipv4 did not change anything

@dstftw
Copy link
Collaborator

dstftw commented Jan 16, 2018

youtube-dl does not impose any implicit speed limitations. Low speed is the result of network issues between you and host you are being served by, shaping or any other network limitations.

@dstftw dstftw closed this as completed Jan 16, 2018
@dstftw dstftw added the invalid label Jan 16, 2018
@zannix
Copy link
Author

zannix commented Jan 16, 2018

@dsftw Can you please try running the following command:

youtube-dl -f 140 http://www.youtube.com/watch?v=4-BtZWGaPNY

I've tried running it on 3 different servers with same terrible speed.

I cannot accept that this is somehow a host network issue.

@Hrxn
Copy link

Hrxn commented Jan 16, 2018

PS E:\Test> youtube-dl -f 140 http://www.youtube.com/watch?v=4-BtZWGaPNY
[youtube] 4-BtZWGaPNY: Downloading webpage
[youtube] 4-BtZWGaPNY: Downloading video info webpage
[youtube] 4-BtZWGaPNY: Extracting video information
[youtube] 4-BtZWGaPNY: Downloading js player vflLCGcm0
[download] Destination: NAIMA - Let Me See You [NCS Release]-4-BtZWGaPNY.m4a
[download] 100% of 2.42MiB in 00:35
[ffmpeg] Correcting container in "NAIMA - Let Me See You [NCS Release]-4-BtZWGaPNY.m4a"
PS E:\Test>

That is definitely slow for YouTube. Strange, not sure what is going on.

@zannix
Copy link
Author

zannix commented Jan 16, 2018

I urge moderators to reconsider placing the Invalid tag for this issue

@yan12125
Copy link
Collaborator

I got the same result as @Hrxn (2.42MB in 35 seconds). Most likely it's an YouTube side issue and youtube-dl can do nothing for that.

@Hrxn
Copy link

Hrxn commented Jan 16, 2018

Yes, I assume that this is not a persistent problem. Probably just some internal "restructuring" on their own network or something.

@antdude
Copy link

antdude commented Jan 16, 2018

Yes, I am also having the same problem since yesterday morning PST. I wonder if Google is throttling. I notice my downloads start very fast and the slow. For an example with a Verizon FIOS pipe (also seen on Time Warner/Spectrum Cable's pipe):

$ youtube-dl https://www.youtube.com/watch?v=sC_qXF5UOtg
[youtube] sC_qXF5UOtg: Downloading webpage
[youtube] sC_qXF5UOtg: Downloading video info webpage
[youtube] sC_qXF5UOtg: Extracting video information
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Destination: DO TEENS KNOW 2000s TEEN MOVIES (REACT - Do They Know It)-sC_qXF5UOtg.f137.mp4
[download] 83.8% of 135.64MiB at 611.51KiB/s ETA 00:36

@Hrxn
Copy link

Hrxn commented Jan 16, 2018

Good lord.. let's hope that not..

@matttc111
Copy link

I am willing to bet that the initial speed is buffered data.

@jayktaylor
Copy link

Hi. I'm experiencing these same issues when using --format bestaudio. I know you believe it is on YouTube's side, but even manually downloading the webm file (through a browser or other tools) that youtube-dl tries to download works completely fine for me at a normal speed. Could it be that something has changed in YT's backend that yt-dl does not account for?

@zannix
Copy link
Author

zannix commented Jan 16, 2018

I second that. After some testing I've come to conclusion that the speed of download starts off high but significantly drops after a short time. However, if you send consecutive requests, youtube-dl will always download from where it left off (it caches data), using high speed for each consecutive request (provided you keep them short enough). Of course this is terribly dirty but perhaps may give a further insight into the problem.

@antdude
Copy link

antdude commented Jan 17, 2018

FYI. No problems with vimeo.com's videos. Fast on there.

@rafaelwastaken
Copy link

rafaelwastaken commented Jan 17, 2018

Also having the same problem. Downloads start off fast and then abruptly slow down to really slow speeds. Only seems to happen on YouTube.

From what I've witnessed, webm files seem to have the worst speeds.

@ChicBrother
Copy link

Have you try to remove ffmpeg and run youtubedl again ? My problem is same with you , but it solved after removing ffmpeg , so weird

@yan12125
Copy link
Collaborator

@zannix's proposal is the most likely solution. Please move on to #6923 for further discussions and #11476 for a possible solution.

@ytdl-org ytdl-org locked and limited conversation to collaborators Jan 17, 2018
@dstftw
Copy link
Collaborator

dstftw commented Feb 3, 2018

2018.02.03 introduces a new experimental option --http-chunk-size for chunk-based downloading. In particular it allows to speed up downloading of DASH formats on Youtube. According to my observations optimal chunk size for Youtube is <=10MiB (apparently that's where throttling starts): --http-chunk-size 10M.
Enforced in 2018.02.04.

@yan12125 yan12125 added bug and removed invalid labels Feb 3, 2018
@yan12125
Copy link
Collaborator

yan12125 commented Feb 3, 2018

That sounds great! I haven't had time to look into such a big patch, while I guess it's time to unlock this issue for user feedback?

@dstftw dstftw removed the bug label Feb 3, 2018
@dstftw dstftw reopened this Feb 3, 2018
@ytdl-org ytdl-org unlocked this conversation Feb 3, 2018
@dstftw dstftw closed this as completed Feb 3, 2018
@Hrxn
Copy link

Hrxn commented Feb 4, 2018

Seems to work, I'd say..

But I just have to ask: Where is the catch?

There should not be one, I'm aware, maybe this is only paranoia..

Did someone compare the checksums of the downloaded data so far?

@antdude
Copy link

antdude commented Feb 6, 2018

Hmm. It is fast again today. Not sure if youtube-dl has a fix (2018.02.04) or something else was fixed?

@dstftw
Copy link
Collaborator

dstftw commented Feb 6, 2018

#15271 (comment)

@zannix
Copy link
Author

zannix commented Feb 6, 2018

I just want to give props to whoever implemented the workaround. Good job and thank you.

@Nicks182
Copy link

Nicks182 commented Mar 3, 2018

I was only having issues with super slow downloads for the last 2 weeks or so. At the same time we were having some issues just watching youtube normally. My ISP insisted that they were not the cause. Then watching youtube in the browser fixed it self, but youtube-dl would still download at like 50kb/s on a 20Mbps line. So I came here and saw others had a similar issue. Downloaded a new exe and it works. However, it would go at full speed, start to slow down, then jump back up to full speed. I assume this is expected behavior?

Seems the youtube throttling is being implemented bit by bit across the globe. From South Africa here.

Thanks to the people maintaining this. Being able to queue a bunch of vids for download at night in high quality means I can watch these HQ vids any time of the day, after downloading, without interfering with my brother's online gaming. Before my 20Mbps fiber, we had 4mbps ADSL... third world problems. youtube-dl was essential if we wanted to watch anything at higher than 360p without weird buffering issues. Google don't give a sh!t about us peasants. So the work you guys do does make a difference. Keep it up.

@ytdl-org ytdl-org deleted a comment from Techwolf Mar 18, 2018
@ytdl-org ytdl-org locked and limited conversation to collaborators Mar 18, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants