Download of songs very slow

[samuscherer]

I recently installed the bot on a server with ~600Mbit/s up and downstream but for some reason downloading a song after sending the !play command takes ages. There's no stuttering or similar issues while playing. I can't explain this behaviour, I already checked the cpu load, memory usage etc. and nothing indicates that the server is overloaded. It might totally be a problem on my end but I can't think of another reason why this keeps happening - any suggestions?
Thanks!

[jayktaylor]

Hey - yeah this is an issue I've verified myself. It looks like an issue with youtube-dl (see ytdl-org/youtube-dl#15271), which is a library that MusicBot depends on. Unfortunately, they don't really believe it's their problem. I'll keep an eye on it for if they updates on it in the future, but it may be required for us to find a workaround to the issue.

As such, it isn't something we can do something about, but I'll keep this thread open as I expect it to become a common issue.

[samuscherer]

Thanks for your reply! I was able to reproduce the exact behaviour described in the issue you referenced. So definitely not your fault and something we just have to keep our eyes on.

At this point I also want to say thank you for you guys' work!

[lilchancep]

I'm going to go ahead and confirm this is happening to me as well. Updated bot to the latest version and now it takes ages for it to actually download anything from youtube. Any other source say SoundCloud works quickly as intended. It used to not be this way.

Specs:
Internet: 150/150 Fiber Connection
RAM: 16GB RAM
CPU; Ryzen 1700

[jayktaylor]

The bot's version has nothing to do with this issue - it's something to do with youtube-dl as previously mentioned, you can confirm it by running youtube-dl -v --format bestaudio <url> on the command line (that command is equivalent to what the bot calls yt-dl with). Until the maintainers of yt-dl acknowledge the issue or there is some indication as to why it's happening, there's not much we can do. So far, they have said it's a server-side problem on YouTube's side, but I'm sceptical because alternative tools work fine.

An alternative, if yt-dl doesn't resolve the issue themselves, is for us to change how the bot currently works for YouTube and download the full video instead, then convert it to an audio-only format ourselves. This is something that we really shouldn't have to do though, and will use more bandwidth as the full video files are a lot bigger. It would be slightly slower than the speed the bot downloaded files before too, but not as slow as it currently is.

I'm keeping an eye on it, but I want to make sure that it's clear that this issue is not our issue, and isn't confined to MusicBot. It's likely that other music bots and tools that rely on yt-dl in the same way we do will be experiencing this problem too. We're just gonna have to be patient until there's some update as to what is going on.

[Mindtroll]

got the issue here to

[mLgz0rn]

Same issue here.

I tried downloading a video from youtube manually, and that also took forever.
So seems like it's actually on youtube's end, dunno if they have put some kind of limit on?

[SoapTastesBad]

same issue
appeared yesterday after i updated dependencies

[jayktaylor]

Update
youtube-dl still refuse to do anything about this, so I've implemented the dirtiest patch in the history of dirty patches (37db56e), which detects when your download speed for a file is <200KiB/s and restarts the download (yt-dl caches downloads, and restarting seems to put it back at its normal speed).

To use this patch, you will need to use the developmental version of the bot (which may be buggy). You can switch to it by running git checkout review and then git pull in your MusicBot folder. Please indicate if you have issues with downloading after doing this, and if it helped combat the issue.

You will get an indication of if the download was restarted because of the download's speed if you see Detected low download speed (< 200KiB/s). Trying to restart... in your console. After that, it should complete within your normal speed.

[samuscherer]

Your patch works great - I agree that it's extremely dirty but it shouldn't be something you have to fix anyway. Thanks though for the incredibly fast response!

[mLgz0rn]

After updating to the dev version, the bot will only play 1 track, then it dies.

Traceback (most recent call last): File "C:\Users\jacob\AppData\Local\Programs\Python\Python35\lib\site-packages\youtube_dl\downloader\common.py", line 198, in try_rename os.rename(encodeFilename(old_filename), encodeFilename(new_filename)) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'audio_cache\\youtube-s_xJYHQGsLE-UGANDAN_KNUCKLES_-_KNOW_DE_WAY_REMIX_PROD._BY_ATTIC_STEIN.webm.part' -> 'audio_cache\\youtube-s_xJYHQGsLE-UGANDAN_KNUCKLES_-_KNOW_DE_WAY_REMIX_PROD._BY_ATTIC_STEIN.webm'

musicbot.exceptions.ExtractionError: ERROR: unable to rename file: [WinError 32] The process cannot access the file because it is being used by another process: 'audio_cache\\youtube-s_xJYHQGsLE-UGANDAN_KNUCKLES_-_KNOW_DE_WAY_REMIX_PROD._BY_ATTIC_STEIN.webm.part' -> 'audio_cache\\youtube-s_xJYHQGsLE-UGANDAN_KNUCKLES_-_KNOW_DE_WAY_REMIX_PROD._BY_ATTIC_STEIN.webm'

You can see the whole log here (https://github.com/Just-Some-Bots/MusicBot/files/1643300/musicbot.log)

[jayktaylor]

I've added a new patch, which forces yt-dl to check for socket timeouts that last over a second and retry them (up to 100 times). This seems to work better and is less dirty than the previous patch, but I want to make sure it has no issues before I push it to the master branch.

@mLgz0rn can you try a git pull and seeing how it behaves now?

[mLgz0rn]

@jaydenkieran It seems to be working now 👍

I do however got another issue with the dev build.
I've set myself as MusicMaster in the permissions with both my role id and my own id, but im not able to instaskip.
I still want's other people to vote.

!restart does not work either.
Want me to create seperate issues for those 2?

[jayktaylor]

The skip behaviour has changed in the dev build - everyone, including the owner and those with perms, will vote to skip when using the command normally due to previous complaints. However, if you are an owner or have perms to InstaSkip, you can still do so by using !skip f instead of !skip. A full changelog and other info will be provided when the build becomes a public release.

Also I believe we're aware of the restart issue, but just in case it's something different, feel free to drop a log/error here.

[mLgz0rn]

Ah okay! thanks alot for that info!

[jayktaylor]

Socket timeout patch has been rolled out to the main branch as release 1.9.6_2.

[wiiaboo]

Alternatively, the workaround is changing ytdl_format to just 'best'. That's still not traffic shaped by youtube.

[jayktaylor]

That isn't really the best workaround, as you're then downloading the video too, increasing the download size overall. Why download a 30MB video file if you can download a 3MB audio file?

[wiiaboo]

Why download a 3MB audio file at 130KB/s when you can download 30MB video at 20MB/s and then extract the audio?

[MattBSG]

More bandwidth (for those trying to keep that down), more cpu time and delay with queuing music, as well as not being efficient at all

[wiiaboo]

More bandwidth, sure.
More cpu time, not that much if you just extract the audio instead of reencoding.
Delay with queuing music, much less, actually, since you don't take 5 minutes to download a 8 minute audio track.

But sure, ideally there would be some workable workaround to youtube's shaping. Even changing the downloader to aria2c and splitting the download into chunks doesn't seem to do anything.

[jayktaylor]

Nah I know what you're saying, I'd just rather keep bandwidth low for those that use hosts that only provide a limited amount of it. I know it doesn't really affect the CPU much more than it would with an audio-only track. Changing it to best is a good option for those that aren't bothered about the increase in bandwidth.

As it stands, the socket timeout patch seems to work relatively well, but really it needs to be the maintainers of yt-dl that actually do something to combat the throttling, such as looking into merging this PR that has been dormant for a while: ytdl-org/youtube-dl#11476. If the current 'patch' starts playing up or not working as well as I'd hope then I'll likely commit changes where the bot downloads the full video instead (until YT starts throttling that too).

[mLgz0rn]

@jaydenkieran
So the musicbot is working some what alright.
But i'm having an issue where it after a while just suddenly stops playing and in the end completely dies.

It just did it right now, it threw out a traceback, and then it started trying to download and completed the download for new tracks over and over again, but never started playing anything, it just keept downloading.

I've attached the logs
Where it has happend
musicbot.log
musicbot_last.log

[jayktaylor]

The first log seems to be an unrelated issue to do with your bot disconnecting from Discord's websocket (perhaps the voice region was having issues?), but the second one indicates that you had another issue where yt-dl couldn't complete a download after 100 retries. I'd assume that was because the video you'd queued was over an hour long, and because of low download speeds it kept reconnecting until it eventually hit 100 attempts.

An alternative solution to the socket timeout/retries thing, as mentioned above, might be for us to change the bot to download the best quality video and then process it as we would with audio. The downside is that it is obviously a larger file size and bandwidth.

[mLgz0rn]

Well the discord had no issues, since we were talking on it when it happend, could however have been the connection from the bot to the discord I guess?

Ah yeah that makes sense.
But why would it not just skip that song then and continue the playlist and playing after that?
Instead it just completely stopped playing anything :D
And i'm the only one having access to the server, and with the !restart issue my mates are not able to just restart the bot :p

[Mindtroll]

the loading still long here after the update and can actualy stop also for no reason ... so you can blame youtube for that but can you explain why i can download a video in a flash of light but not on the bot .

[Mindtroll]

get a 10 minute video in one sec

[Mindtroll]

so the probleme probably come from the bot or discrod

[jayktaylor]

YouTube throttles downloads. youtube-dl, a library that the bot uses, doesn't have a good solution for bypassing it. Nothing to do with the bot/Discord.

[Mindtroll]

i understand that but why anywhere i DOWNLAUD (dl) a video or mp3 from youtube it work perfectly fine then .

[jayktaylor]

Because we download the audio directly. Sites like 'youtube to mp3' download the entire video (which isn't throttled), and then convert it to an audio format.

[jayktaylor]

This should be fixed as of a recent ytdl version. Update your dependencies, then update the bot.