mpr_net.md

MPR_Net

1 Introduction

MPR_Net is an image restoration method published in CVPR2021. Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. MPR_Net propose a novel synergistic design that can optimally balance these competing goals. The main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, the model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch that retains local information. At each stage, MPR_Net introduce a novel per-pixel adaptive design that leverages in-situ supervised attention to reweight the local features. A key ingredient in such a multi-stage architecture is the information exchange between different stages. To this end, MPR_Net propose a two-faceted approach where the information is not only exchanged sequentially from early to late stages, but lateral connections between feature processing blocks also exist to avoid any loss of information. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets across a range of tasks including image deraining, deblurring, and denoising.

2 How to use

2.1 Quick start

After installing PaddleGAN, you can run python code as follows to generate the restorated image. Where the task is the type of restoration method, you can chose in Deblurring、Denoising and Deraining, and PATH_OF_IMAGEis your image path.

from ppgan.apps import MPRPredictor
predictor = MPRPredictor(task='Deblurring')
predictor.run(PATH_OF_IMAGE)

Or run such a command to get the same result:

python applications/tools/mprnet.py --input_image ${PATH_OF_IMAGE} --task Deblurring

Where the task is the type of restoration method, you can chose in Deblurring、Denoising and Deraining, and PATH_OF_IMAGEis your image path.

2.1 Prepare dataset

The Deblurring training datasets is GoPro. The GoPro datasets used for deblurring consists of 3214 blurred images with a size of 1,280×720. These images are divided into 2103 training images and 1111 test images. It can be downloaded from here. After downloading, decompress it to the data directory. After decompression, the structure of GoProdataset is as following:

GoPro
├── train
│   ├── input
│   └── target
└── test
    ├── input
    └── target

The Denoising training datasets is SIDD, an image denoising datasets, containing 30,000 noisy images from 10 different lighting conditions, which can be downloaded from training datasets and Test datasets. After downloading, decompress it to the data directory. After decompression, the structure of SIDDdataset is as following:

SIDD
├── train
│   ├── input
│   └── target
└── val
    ├── input
    └── target

Deraining training datasets is Synthetic Rain Datasets, which consists of 13,712 clean rain image pairs collected from multiple datasets (Rain14000, Rain1800, Rain800, Rain12), which can be downloaded from training datasets and Test datasets. After downloading, decompress it to the data directory. After decompression, the structure of Synthetic_Rain_Datasets is as following:

Synthetic_Rain_Datasets
├── train
│   ├── input
│   └── target
└── test
    ├── Test100
    ├── Rain100H
    ├── Rain100L
    ├── Test1200
    └── Test2800

2.2 Training

An example is training to deblur. If you want to train for other tasks, you can replace the config file.

python -u tools/main.py --config-file configs/mprnet_deblurring.yaml

2.3 Test

test model:

python tools/main.py --config-file configs/mprnet_deblurring.yaml --evaluate-only --load ${PATH_OF_WEIGHT}

3 Results

Deblurring

model	dataset	PSNR/SSIM
MPRNet	GoPro	33.4360/0.9410

Denoising

model	dataset	PSNR/SSIM
MPRNet	SIDD	43.6100 / 0.9586

Deraining

model	dataset	PSNR/SSIM
MPRNet	Rain100L	36.2848 / 0.9651

4 Download

model	link
MPR_Deblurring	MPR_Deblurring
MPR_Denoising	MPR_Denoising
MPR_Deraining	MPR_Deraining

References

• Multi-Stage Progressive Image Restoration

  @inproceedings{Kim2020U-GAT-IT:,
    title={Multi-Stage Progressive Image Restoration},
    author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat and Fahad Shahbaz Khan and Ming-Hsuan Yang and Ling Shao},
    booktitle={CVPR},
    year={2021}
  }