SubRenderIntf: Use libass to render ASS subtitles

[jesec]

It supports both internal subtitles and external subtitles. It also supports "Open a subtitle file".

It is quite stable at the moment I believe. I haven't notice any issue yet.

I also added a toggle in "More" page. Users can use toggle to switch between libass and VSFilter rendering method in real time. It is not a permanent property. The property will get reset to default (libass) each time subtitle renderer restarts.

concurrency function is used in SubFrame::Flatten function currently. I am not sure about its performance on lower end platform. It does work fine for me, though. I will probably need to compare its performance with OpenMP's.
@Cyberbeing Could you please test how it works on your environment?

Binary: x64: jesec@8c23283
XySubFilter_git-8c23283_x64.zip

Note that you probably need to handle some includes (notably libass) by yourself. Those are not included in this change due to environment differences. libass can be grabbed via vcpkg or you can include it into src\thirdparty.

And IMO we probably should jump the major version (3 to 4) for this.

[Cyberbeing]

One note just by your screenshot, at least with XySubFilter setting Customize will break the expected behavior of scripts, so you shouldn't do that. That setting is not supposed to be set to your desktop resolution but rather the subtitle script authoring resolution if for some reason it doesn't match the video resolution. In terms of XySubFilter, this ensures scaling is done correctly, since the ASS spec is quirky and very resolution dependent (there is a script/playres resolution, a hidden authoring resolution, and output resolution) which can effect the appearance of rendering of fonts and some tags. The goal of the Layout Resolution implementation is to ensure subtitle appearance remains perceptually and mathematically identical no matter what resolution we scale to and from.

The main use-case of the override setting is for fixing scripts when you have strange stuff like a 640x480 playres, authored to a 1280x720 video, and then embedded into both 1280x720 & 1920x1080 videos without re-authoring the script for 1080p. In that case, if you set the XySubFilter to Customize 1280x720 which fixes the scaling to 1920x1080 without the user needing to re-author the script. For standard usage, it should be set to either Original Video Resolution (classic VSFilter.dll behavior), or AR Adjusted Video Size (which fixes anamorphic scripts not authored with VSFilter.dll behavior in mind). Please never set Customize to some value and use it on everything, things will break. This is an advanced user override which generally the end-user should never touch.

Common confusion is that setting controls rendering or output resolution, but that's not what it does at all. Rendering Resolution = Output Resolution no matter what Layout Resolution is set to. The only exception being that "Render to Original Video Size" checkbox (disabled by default) in which Rendering Resolution = Original Video Resolution, which mimics the behavior of the VSFilter.dll transform filter, and functions as the ultimate typesetting appearance compatibility option.

I wouldn't suggest you change the meaning of that setting for Libass, but rather just gray out and disable it in settings if the Libass backend finds it meaningless.

@jesec Do you have a reliable build script for libass & dependencies for msvc? Or do I really need to build it with gcc & MSYS? When I tried the other day with vcpkg the build failed hard with VC2019. Short of that, it may be helpful if you just attached the required libass.lib for the moment so I don't need to hassle with it. I also notice your zip has fribidi, while I believe focus at Libass has been more on harfbuzz nowadays? Any particular reason for that choice?

[jesec]

One note just by your screenshot, at least with XySubFilter setting Customize will break the expected behavior of scripts, so you shouldn't do that. That setting is not supposed to be set to your desktop resolution but rather the subtitle script authoring resolution if for some reason it doesn't match the video resolution. In terms of XySubFilter, this ensures scaling is done correctly, since the ASS spec is quirky and very resolution dependent (there is a script/playres resolution, a hidden authoring resolution, and output resolution) which can effect the appearance of rendering of fonts and some tags. The goal of the Layout Resolution implementation is to ensure subtitle appearance remains perceptually and mathematically identical no matter what resolution we scale to and from.

The main use-case of the override setting is for fixing scripts when you have strange stuff like a 640x480 playres, authored to a 1280x720 video, and then embedded into both 1280x720 & 1920x1080 videos without re-authoring the script for 1080p. In that case, if you set the XySubFilter to Customize 1280x720 which fixes the scaling to 1920x1080 without the user needing to re-author the script. For standard usage, it should be set to either Original Video Resolution (classic VSFilter.dll behavior), or AR Adjusted Video Size (which fixes anamorphic scripts not authored with VSFilter.dll behavior in mind). Please never set Customize to some value and use it on everything, things will break. This is an advanced user override which generally the end-user should never touch. Common confusion is that setting controls rendering or output resolution, but that's not what it does at all.

Thanks for your hints.

@jesec Do you have a reliable build script for libass & dependencies for msvc? Or do I really need to build it with gcc & MSYS? When I tried the other day with vcpkg the build failed hard with VC2019. Short of that, it may be helpful if you just attached the required libass.lib for the moment so I don't need to hassle with it.

Here is the libass.lib. libass.lib.zip

Yeah. It is quite painful that it doesn't support MSVC but I think I am going to propose some changes to that soon. For now, I think we can just import libass to this project as a thirdparty library. jesec@5983c11

[jesec]

I changed the CSRIAPI to let unit_test use libass rendering. jesec@17a8b6f We can now do some objective automated testing here.
My results are quite impressive:
script: hi1.ass
fps = 25
width=3840
height=2160,
start=0
end=60

[       OK ] OverallTest.xxx (37305 ms)
[----------] 1 test from OverallTest (37305 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (37306 ms total)
[  PASSED  ] 1 test.

That's 18% (37305ms vs 43987ms) improvement comparing to parallelized default renderer and 59% (37305ms vs 59168ms) improvement comparing to non-parallelized.

[madshi]

Awesome, many thanks Jesse! :)

(Making the VSFilter vs libass switch "permanent" would be even better, but this is probably nitpicking.)

[Cyberbeing]

That's 18% (37305ms vs 43987ms) improvement comparing to parallelized default renderer and 59% (37305ms vs 59168ms) improvement comparing to non-parallelized.

That's about what I'd expect when it comes to 2160p rendering when rasterization throughput is the bottleneck, before any caching kicks in. As mentioned in the other thread, 4K perf is already rather sketchy, so by the time 8K video becomes common XySubFilter's VSFilter-based core will likely become completely unusable unless a way is found to get away of 8x8 supersampling everything, while still retaining anti-aliasing and sub-pixel positioning.

[jesec]

That's 18% (37305ms vs 43987ms) improvement comparing to parallelized default renderer and 59% (37305ms vs 59168ms) improvement comparing to non-parallelized.

That's about what I'd expect when it comes to 2160p rendering when rasterization throughput is the bottleneck, before any caching kicks in. As mentioned in the other thread, 4K perf is already rather sketchy, so by the time 8K video becomes common XySubFilter's VSFilter-based core will likely become completely unusable unless a way is found to get away of 8x8 supersampling everything, while still retaining anti-aliasing and sub-pixel positioning.

Given the current development state, it is pretty much unlikely that someone will fix XySubFilter. I guess switch to libass is the only way out. Other subtitles formats (SRT or PGS) can be very easily handled in any resolution as they don't have any effect features or don't require any processing so it should be fine.

[madshi]

Would it make sense, maybe, to auto prefer VSFilter for low res ASS subtitles (e.g. max 1280x720, or even 1920x1080) and to auto prefer libass for 4K subtitles? Just a wild idea, though...

[jesec]

Would it make sense, maybe, to auto prefer VSFilter for low res ASS subtitles (e.g. max 1280x720, or even 1920x1080) and to auto prefer libass for 4K subtitles? Just a wild idea, though...

IIRC for madVR and XySubFilter, currently the frame will always be rendered in subtitleTargetRect. madVR will give current window resolution to video renderer so even if the subtitle is 720p, 1080p layout, the rendering resolution will be the resolution madVR upscale to.

[madshi]

You're right, of course.

[kkren]

libass seems to not support sup/pgs subtitles?

[JustaPeriod]

@jesec Been using your build here for many months now, up against some heavier typesetting too, without issue.

Would be very interested in another build of it with a more recent commit of libass if you've still got your environment set up.

[Tempest32]

@jesec does this require an AVX2 CPU like your XySubFilter_git-c786186 release?

[othersbet]

hey, how do i add xysubfilter to mpc player?