History
Video2X came a long way from its original concepts to what it has become today. It started as a simple concept of "waifu2x can upscale images, and a video is just a sequence of images". Then, a PoC was made which can barely upscale a single video with waifu2x-caffe and with fixed settings. Now, Video2X has become a comprehensive and customizable video upscaling tool with a nice GUI and a community around it. This article documents in detail how Video2X's concept was born, and what happened during its development.
The story started with me watching Bad Apple!!'s PV in early 2017. The original PV has a size of 512x384, which is quite small and thus, quite blurry.
A screenshot of the original Bad Apple!! PV
Around the same time, I was introduced to this amazing project named waifu2x, which upscales (mostly anime) images using machine learning. This created a spark in my head: if images can be upscaled, aren't videos just a sequence of images? Then, I started making a proof-of-concept by manually extracting all frames from the original PV using FFmpeg, putting them through waifu2x-caffe, and assembling the frames back into a video again using FFmpeg. This was how the "4K BadApple!! waifu2x Lossless Upscaled" video was created.
Thumbnail of the "4K BadApple!! waifu2x Lossless Upscaled" video
After this experiment completed successfully, I started thinking about making an automation pipeline, where this manual process will be streamlined, and each of the steps will be handled automatically.
When I signed up for Hack the Valley II in late 2017, I didn't know what I was going to make during that hackathon. Our team sat down and thought about what to make for around an hour, but no one came up with anything interesting. All of a sudden, I remembered, "Hey, isn't there a PoC I wanted to make? How about making that our hackathon project?" I then temporarily name the project Video2X, following waifu2x's scheme. Video2X was then born.
I originally wanted to write Video2X for Linux, but it's too complicated to get the original nagadomi/waifu2x's version of waifu2x running, so waifu2x-caffe written for Windows was used to save time. This is why the first version of Video2X only supports Windows, and can only use waifu2x-caffe as its upscaling driver.
video2x.py file in the first version of Video2X
At the end of the hackathon, we managed to make a sample comparison video based on Spirited Away's official trailer. This video was then published on YouTube and is the same demo video showcased in Video2X's repository. The original link was at https://www.youtube.com/watch?v=PG94iPoeoZk, but it has been moved lately to another account under K4YT3X's name.
Upscale Comparison Demonstration
When we demoed this project, there wasn't so much interest expressed by the judges. We were, however, suggested to pitch our project to Adobe. That didn't end up going anywhere, either. Like most of the other projects in a hackathon, this project didn't win any awards, and just almost vanished after the hackathon was over.
Our team in Hack the Valley II. You can see Video2X's demo video on the computer screens. Image blurred for privacy.
Roughly three months after the hackathon, I came back to this project and decided it was worth continuing. Although not many people in the hackathon found this project interesting or useful, I saw value in this project. This was further reinforced by the stars I've received in the project's repository.
I continued working on enhancing Video2X and fixing bugs, and Video2X 2.0 was released. The original version of Video2X was only made as a proof-of-concept for the hackathon. A lot of the usability and convenience aspects are ignored in exchange for development speed. The 2.0 version addressed a lot of these issues and made Video2X usable for regular users. Video2X has then also been converted from a hackathon project to a personal open-source project.
Screenshot of Video2X 2.0