ffmpeg hwaccel + container image?

[CastawayEGR]

First and foremost thank you for the great work here!

Reviewing the docs I can't seem to find anything related to using hwaccel for ffmpeg like the windows ui has available for the container image. Is this something that can be done with the default container image or should I go about building my own to solve this?

I fully understand from reading that hardware based decoding can cause quality issues but I want to test it out since software seems to be taking ages to upscale footage.

Thanks again for any insight in advance!

[h9419]

I am just sharing my attempt in applying waifu2x on videos. Yes, you can speed up the processing time by 2-3x easily if you strictly put everything in GPU memory and never let the CPU touch it.

I am NOT using video2x, just similar codes I wrote over the years. Following is what have:

Hardware:

• laptop RTX3060 (only 70w and 2.6GB out of 6GB VRAM used)
• 5800h (not fully utilized, only here for IO)

Tools used:

• Nvidia VPF
• yu45020's Waifu2x pyTorch port
• WSL with GPU-enabled Docker and pytorch docker image with GPU

Threaded Pipeline (3 threads in total):

• Compressed video file (CPU ->GPU) Nvidia VPF (nvdec) to pytorch GPU tensor
• pytorch GPU tensor -> model inference -> upscaled GPU tensor
• GPU tensor -> VPF (nvenc) -> (CPU) compressed video file

I am able to get around 6-7fps upscaling 480x848 to 960x1696, and around 1.3fps from 1080p to 4k on my laptop all power limited to 25w CPU and 70w GPU. For comparison, my setup to 2x upscale a 240p clip at 26 it/s with power consumption of 35w CPU + 70w GPU. Which means my method on a laptop is 2-3x faster than previous video2x benchmarks on desktop hardware.

The pipeline was originally written for running a small Resnet on video, BackgroundMattingV2, where the CPU data loader and encoder accounts for over 95% of the execution time. Removing that bottleneck resulted in 16-20x speedup when all raw data stays on the GPU's higher bandwidth memory. My pipeline needs a custom build of VPF that takes patches from master branch to the last release to add pytorch support, and I cannot distribute binaries because H.264 license still cost something.

I can send you the code if you want to have a look, but I wrote it a while ago and structured it in favor of performance over maintainability. I did manage to adapt it to run inference of waifu2x with the help of this blog post. Note that I am limited to the 2x models unless I figure out how to port the models myself