Add image-to-image task w/ Swin2SR (for super-resolution)

[xenova]

This PR adds support for image-to-image translation, starting with the Swin2SR family of models for super-resolution. See here for the list of already-converted models, including 2x and 4x upscalers.

Closes #138

Example usage

Pipeline API

Example code adapted from here.

import { pipeline } from '@xenova/transformers';

let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/butterfly.jpg';
let upscaler = await pipeline('image-to-image', 'Xenova/swin2SR-classical-sr-x2-64');
let output = await upscaler(url);
// RawImage {
//   data: Uint8Array(786432) [ 41, 31, 24,  43, ... ],
//   width: 512,
//   height: 512,
//   channels: 3
// }

AutoClasses

Example code adapted from here.

import { AutoProcessor, Swin2SRForImageSuperResolution, RawImage } from '@xenova/transformers';

// Load processor and model
const model_id = 'Xenova/swin2SR-classical-sr-x2-64';
const processor = await AutoProcessor.from_pretrained(model_id);
const model = await Swin2SRForImageSuperResolution.from_pretrained(model_id);

// Prepare model inputs
const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/butterfly.jpg';
const image = await RawImage.fromURL(url);
const inputs = await processor(image);

// Run model
const outputs = await model(inputs);

// Convert Tensor to RawImage
const output = outputs.reconstruction.squeeze().clamp_(0, 1).mul_(255).round_().to('uint8');
const outputImage = RawImage.fromTensor(output);
// RawImage {
//   data: Uint8Array(786432) [ 41, 31, 24, ...],
//   width: 512,
//   height: 512,
//   channels: 3
// }

Example output

• input (256x256):
  

• output w/ unquantized model (512x512):
  
  note: produces the exact same output as the python implementation (within floating-point precision errors of course).

• output w/ quantized model (512x512):
  

• side-by-side (input vs. unquantized output):
  

[xenova]

cc @josephrocca :)

I also intend to replicate/showcase the results from their README.

[xenova]

Example using https://huggingface.co/Xenova/swin2SR-compressed-sr-x4-48:

import { pipeline } from '@xenova/transformers';

let url = 'https://huggingface.co/spaces/jjourney1125/swin2sr/resolve/main/testsets/real-inputs/shanghai.jpg';
let upscaler = await pipeline('image-to-image', 'Xenova/swin2SR-compressed-sr-x4-48');
let output = await upscaler(url);

Input:

Output:

[josephrocca]

Awesome!! Seems to take quite a while to load the model - about 40 seconds, not including the download. I'm guessing it's a similar problem to this: microsoft/onnxruntime#11217 since Netron also complains that there are lots of nodes, and takes a very long time to load.

The actual inference is about 40 seconds on 8 threads - not bad! WebGPU will get this to a very usable inference time. Exciting!