Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ytdl] Directory passed to yt-dlp makes aria2c downloader error on attempted directory creation #3021

Closed
Bubsy-26d opened this issue Oct 6, 2022 · 3 comments
Closed

[ytdl] Directory passed to yt-dlp makes aria2c downloader error on attempted directory creation #3021

Bubsy-26d opened this issue Oct 6, 2022 · 3 comments
Labels
bug enhancement

Comments

@Bubsy-26d
Copy link

I'm attempting to use the aria2c downloader for yt-dlp to increase download speeds, but am running into download failures as it seems gallery-dl is sending in a weirdly-formatted directory structure that aria2c doesn't know what to do with.

When calling yt-dlp through gallery-dl, it attempts to download to the directory "\\?\Z:\YT Channels\HD Mixtapes", while using yt-dlp directly uses "Z:\YT Channels\HD Mixtapes" as its directory.

From the verbose gallery-dl output:

gallery-dl -v --no-skip -o extractor.ytdl.cmdline-args="--downloader-args aria2c:'-l log2.txt'" ytdl:https://www.youtube.com/watch?v=cOYmxRclKwQ
[gallery-dl][debug] Version 1.23.2
[gallery-dl][debug] Python 3.10.5 - Windows-10-10.0.19044-SP0
[gallery-dl][debug] requests 2.28.0 - urllib3 1.26.9
[gallery-dl][debug] Starting DownloadJob for 'ytdl:https://www.youtube.com/watch?v=cOYmxRclKwQ'
[ytdl][debug] Using YoutubeDLExtractor for 'ytdl:https://www.youtube.com/watch?v=cOYmxRclKwQ'
[ytdl][debug] Using <module 'yt_dlp' from 'C:\\Users\\{username}\\AppData\\Roaming\\Python\\Python310\\site-packages\\yt_dlp\\__init__.py'>
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading webpage
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading android player API JSON
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading MPD manifest
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading MPD manifest
[ytdl][debug] Using download archive './gallery-dl.db'
Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].mkv[ytdl][debug] [info] Downloading video thumbnail 41 ...
[ytdl][debug] [info] Writing video thumbnail 41 to: \\?\Z:\YT Channels\HD Mixtapes\Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].webp
[ytdl][debug] [download] Destination: \\?\Z:\YT Channels\HD Mixtapes\Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4

10/06 02:08:56 [ERROR] CUID#7 - Download aborted. URI=https://rr5---sn-vgqsrnsd.googlevideo.com/videoplayback?expire=1665061734&ei=Bn8-Y5zaELSb2_gPtriY-Ao&ip=147.219.172.43&id=o-AM1bYJV8pHKFMc3snbRKCi-JOujdQMGllCDrLpWGowy9&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C26&mn=sn-vgqsrnsd%2Csn-q4fl6nsy&ms=au%2Conr&mv=m&mvi=5&pl=16&initcwndbps=2021250&spc=yR2vp2_OWW7sZkg0rXQAyB8a_QuSqsg&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665039676&fvip=3&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRAIgT66SqUekF7O5HJ65B0FpPQHiHlRBHRDxQadTDM-kbxACIF8PSKxqTUVkeed59Ju8l5WKdGdORXVdGZSG1caWc-rd&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALBRFSSL2tSf9jdRiY-JokZSdn1aHvYYgmLqz7LEOc37AiEA4JvPabVc2xbl5Wm_XZPnYKtvKF63_CFVtmtnJ1EwVgU%3D
Exception: [AbstractCommand.cc:403] errorCode=18 URI=https://rr5---sn-vgqsrnsd.googlevideo.com/videoplayback?expire=1665061734&ei=Bn8-Y5zaELSb2_gPtriY-Ao&ip=147.219.172.43&id=o-AM1bYJV8pHKFMc3snbRKCi-JOujdQMGllCDrLpWGowy9&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C26&mn=sn-vgqsrnsd%2Csn-q4fl6nsy&ms=au%2Conr&mv=m&mvi=5&pl=16&initcwndbps=2021250&spc=yR2vp2_OWW7sZkg0rXQAyB8a_QuSqsg&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665039676&fvip=3&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRAIgT66SqUekF7O5HJ65B0FpPQHiHlRBHRDxQadTDM-kbxACIF8PSKxqTUVkeed59Ju8l5WKdGdORXVdGZSG1caWc-rd&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALBRFSSL2tSf9jdRiY-JokZSdn1aHvYYgmLqz7LEOc37AiEA4JvPabVc2xbl5Wm_XZPnYKtvKF63_CFVtmtnJ1EwVgU%3D
  -> [RequestGroup.cc:761] errorCode=18 Download aborted.
  -> [util.cc:1948] errNum=17 errorCode=18 Failed to make the directory //?/Z:/YT Channels/HD Mixtapes//., cause: File exists

From the aria2c logs:

2022-10-06 01:55:29.475633 [INFO] [DefaultBtProgressInfoFile.cc:415] The segment file //?/Z:/YT Channels/HD Mixtapes//./Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4.part.aria2 does not exist.
2022-10-06 01:55:29.475633 [DEBUG] [File.cc:203] UNC Prefix //?/Z:
2022-10-06 01:55:29.475633 [DEBUG] [File.cc:234] Making directory //?/Z:/YT Channels
2022-10-06 01:55:29.475633 [DEBUG] [File.cc:240] Failed to create //?/Z:/YT Channels
2022-10-06 01:55:29.475633 [ERROR] [AbstractCommand.cc:401] CUID#7 - Download aborted. URI=https://rr5---sn-vgqsknek.googlevideo.com/videoplayback?expire=1665060929&ei=4Xs-Y7_0A8uLlu8P-bGkSA&ip=147.219.172.43&id=o-AAWy9eNE9BNjErMuFYcFbSgsgyW3oeh_HzNO9RXtqPSy&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C29&mn=sn-vgqsknek%2Csn-vgqsrnsd&ms=au%2Crdu&mv=m&mvi=5&pl=16&initcwndbps=2083750&spc=yR2vp72ztODvimxPGgQXh8fUN-VJOSo&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665038945&fvip=5&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAI9pf7uAYkz78zJqHcYVm_YkrvUKOzK8zaKV4meKaXwyAiEA3G2dsKKtca1bgJjAfggYgt8q6j5kzjRwvLLv71co3nY%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAP-nDb5KI-FlgWEQy9yQ0XxiJeGzZ3FMA0na5FY1L5BuAiEA_njlC3ATY3J1Tti4v6GpiZetMQlrCGwrvI3r9V8BnxI%3D
Exception: [AbstractCommand.cc:403] errorCode=18 URI=https://rr5---sn-vgqsknek.googlevideo.com/videoplayback?expire=1665060929&ei=4Xs-Y7_0A8uLlu8P-bGkSA&ip=147.219.172.43&id=o-AAWy9eNE9BNjErMuFYcFbSgsgyW3oeh_HzNO9RXtqPSy&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C29&mn=sn-vgqsknek%2Csn-vgqsrnsd&ms=au%2Crdu&mv=m&mvi=5&pl=16&initcwndbps=2083750&spc=yR2vp72ztODvimxPGgQXh8fUN-VJOSo&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665038945&fvip=5&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAI9pf7uAYkz78zJqHcYVm_YkrvUKOzK8zaKV4meKaXwyAiEA3G2dsKKtca1bgJjAfggYgt8q6j5kzjRwvLLv71co3nY%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAP-nDb5KI-FlgWEQy9yQ0XxiJeGzZ3FMA0na5FY1L5BuAiEA_njlC3ATY3J1Tti4v6GpiZetMQlrCGwrvI3r9V8BnxI%3D
  -> [RequestGroup.cc:761] errorCode=18 Download aborted.
  -> [util.cc:1948] errNum=17 errorCode=18 Failed to make the directory //?/Z:/YT Channels/HD Mixtapes//., cause: File exists

Compared to the aria2c logs of using yt-dlp directly:

2022-10-06 01:58:06.155875 [INFO] [DefaultBtProgressInfoFile.cc:415] The segment file Z:/YT Channels/HD Mixtapes/./Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4.part.aria2 does not exist.
2022-10-06 01:58:06.155875 [DEBUG] [DefaultPieceStorage.cc:168] usedPieces_.size()=1
2022-10-06 01:58:06.155875 [DEBUG] [SegmentMan.cc:140] Attach segment#0 to CUID#7.
2022-10-06 01:58:06.155875 [DEBUG] [SegmentMan.cc:146] Flushing cached data, size=0
2022-10-06 01:58:06.155875 [DEBUG] [WrDiskCache.cc:95] Update cache entry size=0, delta=0, clock=1
2022-10-06 01:58:06.155875 [DEBUG] [SegmentMan.cc:163] index=0, length=1048576, segmentLength=1048576, writtenLength=0
2022-10-06 01:58:06.156375 [DEBUG] [WrDiskCache.cc:95] Update cache entry size=0, delta=0, clock=2
2022-10-06 01:58:06.156877 [INFO] [DefaultBtProgressInfoFile.cc:205] Saving the segment file Z:/YT Channels/HD Mixtapes/./Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4.part.aria2

I'm seeing this behavior no matter what drive or directory I download to. The yt-dlp native downloader seems to handle the passed-in directory... kind of. I've been having issues where the first time I attempt downloading a file it's 50-50 whether yt-dlp will error out on attempting to download the first fragments. Deleting the files it created and re-downloading usually works the 2nd time. I don't have any settings for directory, base-directory, or any extractor-specific output directory settings in my gallery-dl config, nor in my yt-dlp config.
@mikf
Copy link
Owner

mikf commented Oct 7, 2022

it seems gallery-dl is sending in a weirdly-formatted directory structure
\?\Z:\YT Channels\HD Mixtapes

Prepending a path with \\?\ is perfectly valid, but a lot of tools throw an error when they get a path like this.

Initially this path modification was done to get around the 260 character limit for filesystem paths, but it shouldn't be all that necessary anymore with Win10 now having native support for long paths.

I'll just add an option to disable this behavior.

@Bubsy-26d
Copy link
Author

Interesting, I've never seen that before. But an option to disable sounds like it would handle this, thanks!

mikf added a commit that referenced this issue Oct 8, 2022
@mikf
add 'path-extended' option (#3021)
7f30a0d
@mikf
Copy link
Owner

mikf commented Oct 8, 2022 •
edited
Loading

These are called "extended-length path[s]"

… the maximum length for a path is MAX_PATH, which is defined as 260 characters
To specify an extended-length path, use the "\?" prefix. For example, "\?\D:\very long path".

(https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)
(https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#win32-file-namespaces)

Anyway, with commit 7f30a0d you can disable them by setting path-extended to false

@mikf mikf closed this as completed Oct 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikf @Bubsy-26d