Add support for native PGS subtitle rendering without transcoding

[Arcus92]

Changes
This PR allows the client to request and render native PGS subtitle streams without transcoding. PGS is a graphical subtitle format. The subtitles are rendered locally by the client on-top of the video element.

This adds a new dependency libpgs-js. I just created this library for the Jellyfin project. If desired, you can fork it into the @jellyfin domain.

Tasks

• Disable native PGS support if 'Subtitle burn-in' is not set to 'auto'.
• Improve performance by using WebWorkers to render subtitles in a separate thread.

Related Pull Request (backend): jellyfin/jellyfin#12056
Related Pull Request (vue): jellyfin/jellyfin-vue#2404

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[Arcus92]

I might need a little help over at libpgs-js on how to get the web-worker output ES compatible.

[ferferga]

@Arcus92 May I ask you why webpack over rollup or Vite?

[Arcus92]

@ferferga To be honest ... I haven't worked on web projects in a long time and it is the only bundler I kinda know. I'll look into other tools over the weekend.

[ferferga]

I know the basics of rollup, but I know that Vite has this option, which I use in Vue. I'm not sure how good Vite is for libraries though, and that's why I suggested rollup (since Vite is based on it). Your library is rather simple in terms of building, so I believe it shouldn't be too problematic with either bundler, but you'll likely have much more luck with both instead of webpack.

Don't hesitate in pinging me in Matrix if I can be of some help

[Arcus92]

As suggested in the backend PR, there is now an opt-in setting to enable experimental Pgs rendering. I'm working on a few performance and load improvements.

[Sab44]

I just found this PR, thanks for putting in the effort for this!
One question, once this is merged will we also be able to adjust vertical subtitle position for PGS subtitles (similar to SRT subtitles)?
If not, what would need to be done to make that possible?
Thanks!

[dmitrylyzo]

@Sab44 PGS subtitles are image-based. Changing their position will result in markers/labels being displayed in the wrong place. The same goes for SSA/ASS.

[dmitrylyzo]

Regarding Tizen and webOS:
transferControlToOffscreen is supported since Chrome 69. So libpgs-js will only work on Tizen 5.5+ and webOS 6+.

webOS 5 emulator:

Uncaught (in promise) TypeError: this.canvas.transferControlToOffscreen is not a function

[Arcus92]

Regarding Tizen and webOS: transferControlToOffscreen is supported since Chrome 69. So libpgs-js will only work on Tizen 5.5+ and webOS 6+.

@dmitrylyzo Thanks for testing this on webOS! I could add a fallback in libpgs-js and use single thread rendering if web worker transfer is not supported. But I don't think rendering on the main thread on these devices is a good idea. It could lead to playback stuttering.
For now I'll exclude pgs format support if transferControlToOffscreen isn't available.

[dmitrylyzo]

@dmitrylyzo Thanks for testing this on webOS! I could add a fallback in libpgs-js and use single thread rendering if web worker transfer is not supported. But I don't think rendering on the main thread on these devices is a good idea. It could lead to playback stuttering. For now I'll exclude pgs format support if transferControlToOffscreen isn't available.

JSO renders in the worker, preparing images to be put on the canvas in the main thread. So the main thread only copies the image buffer from the worker and overlays it onto the canvas.

You probably can do the same: parse PGS in the worker and produce a set of images.

[Arcus92]

I should be able to refactor the lib to support both render paths: Rendering in worker by default and rendering in main thread as a fallback. I also try to setup a webOS emulator to verify the changes. Sounds like a fun weekend project 😄

[Arcus92]

I haven't tested it in Tizen yet but it's now working in webOS 5.

[Copper-Bot]

First, thanks for this PR, it will be really great to have native PGS sub support on Tizen without transcoding 🎉
I was wondering if this work can be transposed for DVD sub (or VobSub) as well, as they are image based subtitles like PGS ?
This will probably require opening another PR, but I was just curious if it could easily be done or not.
I don't fully understand the difference between VobSub and PGS, so maybe it's a dumb question (and off-topic).

[dmitrylyzo]

I know console isn't available in that Chromium version

Have you tried using Chrome 79 (or older)? AFAIK, Chrome 80+ doesn't work.

[Arcus92]

Have you tried using Chrome 79 (or older)? AFAIK, Chrome 80+ doesn't work.

@dmitrylyzo WebOS 5 is using Chromium 68. Workers should work since version 4.

[dmitrylyzo]

@dmitrylyzo WebOS 5 is using Chromium 68. Workers should work since version 4.

I mean, you need to use Chrome 79 (and older) for debugging webOS 5 because Google removed some API.

[Arcus92]

I mean, you need to use Chrome 79 (and older) for debugging webOS 5 because Google removed some API.

Oh, now I got you. Thanks for the hint. The oldest working build I could find is 75 but debugging doesn't change. However, the web worker is working on Desktop. 68 won't boot up on my desktop.

[dmitrylyzo]

@Arcus92 I got it working on webOS 5. We need to polyfill Array.flatMap:

this.displaySets[a].windowDefinitions.flatMap is not a function

I'm debugging using Chrome 79.

[Arcus92]

@dmitrylyzo Already fixed this in the webos branch. Haven't merged it yet because the worker is not ... working.

[dmitrylyzo]

@dmitrylyzo Already fixed this in the webos branch. Haven't merged it yet because the worker is not ... working.

I'm testing with libpgs-0.5.0, and I managed to load the worker even on webOS 1.2.
At a minimum, it requires polyfilling of Promise and fetch. I did't get it to parse PGS (it's too late currently 🛌).

FIY, you can get console in the worker on webOS 1.2:

• add to worker:
  https://github.com/jellyfin/JavascriptSubtitlesOctopus/blob/ec9f7e91a9b082dae18380898b2cd10413cc7e20/src/pre-worker.js#L1-L33
  https://github.com/jellyfin/JavascriptSubtitlesOctopus/blob/ec9f7e91a9b082dae18380898b2cd10413cc7e20/src/post-worker.js#L622-L625
• add to main:
  https://github.com/jellyfin/JavascriptSubtitlesOctopus/blob/ec9f7e91a9b082dae18380898b2cd10413cc7e20/src/subtitles-octopus.js#L623-L642

[dmitrylyzo]

@Arcus92 My test branch with your webos branch:

• No-offscreen worker starts on webOS 1.2, but fails on Can't find variable: ImageData (no ImageData in the worker scope?).
• Render in the main thread works on webOS 1.2.

[jellyfin-bot]

This pull request has merge conflicts. Please resolve the conflicts so the PR can be successfully reviewed and merged.

[Arcus92]

@dmitrylyzo Sorry I was busy for a few days. Thanks for your contribution. I'll have a look at your commits right now.

[Arcus92]

I finally fixed the issues by adding a browser detection in libpgs. WebOS with a Chrome version below 68 will now use the main-thread renderer. Firefox and Chrome version without ImageData-in-worker support will also fallback to the main-thread.

Normally I wouldn't check feature support via the userAgent, but in this case some of these legacy browser (like WebOS 5.0) have a broken worker implementation. I may need to add exceptions for other platforms if future issues are reported.

[dmitrylyzo]

PR works on:

• webOS 1.2 (main thread)
• webOS 5 (main thread)
• Chrome 128 (worker)
• Firefox 130 (worker)

I think we can use worker even on webOS 1.2, but it can be implemented separately.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[jellyfin-bot]

Cloudflare Pages deployment

Latest commit	e560d37f99951c8be9d828d065fb2c146a3107d6
Status	✅ Deployed!
Preview URL	https://97f6818e.jellyfin-web.pages.dev
Type	🔀 Preview