RTMP downloader does not support unicode filenames #23765
Comments
|
works fine with both rtmpdump and FFmpeg: |
|
@remitamine Can you try this on Windows? This doesn't seem to be an issue on linux This is why it works on linux. Linux supports UTF8 as an encoding while windows does not, only codepages |
|
the output I have posted shows that the initial claim that rtmpdump does not work with UTF names is incorrect, and even if it doesn't, it's an external bug that needs to be addressed in the external project(rtmpdump), not in youtube-dl, and as you can see, you have also the option to use FFmpeg an external downloader. |
|
The point of this ticket is to make a workaround for this issue. rtmpdump has not been maintained for 10 years and likely will never receive a fix for this, especially considering that rtmpdump is a linux-orientated tool in the first place. Nor does using ffmpeg as an extractor, as windows builds of ffmpeg do not come with librtmp integrated, meaning that it's impossible to use it for downloading RTMP streams. There is no solution for this problem that doesn't involve changes within youtube-dl, unless you want to enforce a custom fork of rtmpdump or ffmpeg on windows users, which is laughable. Just because this works out of the box on Linux doesn't mean that my claim is wrong. I have already made a fix in my fork which requires a change to a single line of code that checks for windows, and then strips unicode characters. If it takes a PR for this to be fixed then so be it. |
|
There is already a generic purpose workaround |
|
It's not a clean workaround. Having to run a I'm not sure why this is considered duct tape but allowing streams to cut out at 99.8% due to some sites not reporting stream length correctly isn't, when technically that should be an external bug too. |
last commit from 2019.
website owners wouldn't fix the streams they serve just to make them downloadable with youtube-dl. and BTW flash player end of life will be at the end of this year and think 99% of the websites serving videos offer at least alternative streams that do not rely on browser plugins(Flash Player and Silver light). |
There has not been a release since 2011, and the maintainers don't participate in the mailing list anymore. I highly doubt that they would implement a fix for it, let alone me figuring out how to contact them in the first place.
This happens on all sites that use RTMP. I can find you another site if you don't believe me. And keep in mind that this is for the biggest video sharing/streaming site in Japan. Ignoring the fact that the extractor for regular niconico videos is broken in the first place, the reason why this is important is that niconico is a website that is technically inept (they haven't added HTTPS until very recently) and it is likely that they will be removing videos once flash player is deprecated, so it's important that old livestreams that they still host are archived. All streams created prior to March 2019 use RTMP, while all ones after use HLS (with their own complex websocket heartbeat and manifest retrieval method). All of the videos from the site will use UTF for titles as it's orientated for a Japanese audience. It's something that would be encountered 99% of the time.
None of them are user friendly. I either have to tell people who want to download from this site to either use a specific fork (which will not be obvious as I doubt this will be put in youtube-dl documentation, and would raise a lot of issues from people using the much more popular forks), or they have to rename every old stream that they download, which is very inconvenient when this should be automated by youtube-dl itself. And if this is to be automated, the title has to be extracted from youtube-dl run a second time without the flag to be able to find out what the original filename would be. And if RTMP is not being used anymore by most sites, it only means that there will not be many sites at risk of breaking. This is a very difficult workaround for something that can be easily fixed with two lines of code, that will have minimal impact (if any) to the downloader that saves a bunch of people the headache of having to create a 50 line batch script just to get around this. I really don't understand why there is so much friction for this. I might as well just tell people to use my fork where this works out of the box without needing to jump through any hoops. |
Checklist
Verbose log
Description
I've been working on implementing #17415 which uses RTMP, but I've discovered that it doesn't support unicode filenames. This can be tested with any extractor that uses the RTMP downloader, and specifying the
-fformat flag with a filename that uses unicode. Inversely, my particular issue with the above log is fixed if I specify an ASCII filename.After some investigation, this seems to be an issue with rtmpdump itself as its code uses
charinstead ofwcharfor output filenames, meaning that it's incapable of actually using unicode.There are multiple fixes that could be used for this:
The text was updated successfully, but these errors were encountered: