SpecGAN: Large-factor Super-resolution of Remote Sensing Images

Yapeng Meng, Wenyuan Li, Sen Lei, Zhengxia Zou, and Zhenwei Shi*

The official PyTorch implementation of SpecGAN, Large-factor Super-resolution of Remote Sensing Images with Spectra-guided Generative Adversarial Networks, accepted by IEEE Transactions on Geoscience and Remote Sensing（TGRS).

Abstract

Large-factor image super-resolution is a challenging task due to the high uncertainty and incompleteness of the missing details to be recovered. In remote sensing images, the sub-pixel spectral mixing and semantic ambiguity of ground objects make this task even more challenging. In this paper, we propose a novel method for large-factor super-resolution of remote sensing images named ``Spectra-guided Generative Adversarial Networks (SpecGAN)''. In response to the above problems, we explore whether introducing additional hyperspectral images to GAN as conditional input can be the key to solving the problems. Different from previous approaches that mainly focus on improving the feature representation of a single source input, we propose a dual branch network architecture to effectively fuse low-resolution RGB images and corresponding hyperspectral images, which fully exploit the rich hyperspectral information as conditional semantic guidance. Due to the spectral specificity of ground objects, the semantic accuracy of the generated images is guaranteed. To further improve the visual fidelity of the generated output, we also introduce the Latent Code Bank with rich visual priors under a generative adversarial training framework so that high-resolution, detailed, and realistic images can be progressively generated. Extensive experiments show the superiority of our method over the state-of-art image super-resolution methods in terms of both quantitative evaluation metrics and visual quality. Ablation experiments also suggest the necessity of adding spectral information and the effectiveness of our designed fusion module. To our best knowledge, we are the first to achieve up to 32x super-resolution of remote sensing images with high visual fidelity under the premise of accurate ground object semantics.

Visualization results

Installation

SpecGAN builds on MMEditing and MMCV. We make some necessary modifications in order to load hyperspectral images (HSI).

Step 1. Clone our repository and create conda environment

conda env create -f environment.yml

Step 2. Install gdal

If you use Windows and Python 3.7, just download gdal wheel.

pip install GDAL-3.4.2-cp37-cp37m-win_amd64.whl

Step 3. Install MMCV with MIM.

pip install openmim
mim install mmcv-full==1.3.11

Please use our provideSpecGAN/photometric.py to replace yourcondaroot/Anaconda3/envs/SpecGAN/Lib/site-packages/mmcv/image/photometric.py to support muti-channel image processing.

Step 4. Install our modified mmediting.

pip install -e .

Train

Step 1.

Edit config files at configs/SpecGAN_32x_sr.py or configs/SpecGAN_16x_sr.py to modify line 87-112 to switch to your dataset path.

The data set should be place as here.

Step 2.

x32 super-resolution:

python tools/train.py configs/SpecGAN_32x_sr.py

x16 super-resolution:

python tools/train.py configs/SpecGAN_16x_sr.py

It needs a really long training time, about 14 days on a GTX 1080Ti graphic card.

Evaluate

x32 super-resolution:

python tools/test.py configs/SpecGAN_32x_sr.py work_dirs/xxx.pth --save-path xxx

x16 super-resolution:

python tools/test.py configs/SpecGAN_16x_sr.py work_dirs/xxx.pth --save-path xxx

Pretrained model and dataset

Our pre-trained models can be download from Google Drive or 百度网盘.

The dataset should be arranged as follows:

--data_root_path
    --train
        --RGBHR
            --1.tif
            --2.tif
            --...
        --HSI
            --1.tif
            --2.tif
            --...
    --valid
        ...
    --test
        ...

Our paper uses dataset from the IEEE data fusion contest 2019.

The HSI input image should be resized to the same size as low resolution RGB image. We concat the two input as [HSI;RGBLR], which has 48 channels and 3 channels, respectively.

Citation

If our research is helpful to your research, please cite it as below.

@ARTICLE{9950553,
  author={Meng, Yapeng and Li, Wenyuan and Lei, Sen and Zou, Zhengxia and Shi, Zhenwei},
  journal={IEEE Transactions on Geoscience and Remote Sensing}, 
  title={Large-Factor Super-Resolution of Remote Sensing Images With Spectra-Guided Generative Adversarial Networks}, 
  year={2022},
  volume={60},
  number={},
  pages={1-11},
  doi={10.1109/TGRS.2022.3222360}}