Comparison of the multispectral (MS) and hyperspectral (HS) image fusion techniques used for the spatial resolution enhancement of HS images.
Existing hyperspectral imaging systems produce images that lack spatial resolution due to hardware limitations. Even with the proven efficacy of this technology in several computer vision tasks, the aforementioned limitation obstructs its applicability. Contrarily, conventional RGB images have a much larger resolution with just three spectra. Since the issue of low resolution images arises from hardware limitations, there have been several developments in software-based approaches to improve the spatial resolution of hyperspectral images.
This work allows for an easy-to-use framework for testing and comparing existing hyperspectral image fusion (HIF) methods for spatial resolution enhancement.
If you use any part of this work, please use the following citation:
Magalhรฃes, Miguel. โHyperspectral Image Fusion: A Comprehensive Reviewโ. Masterโs Programme in Imaging and Light in Extended Reality (IMLEX). MSc. thesis. KU Leuven, 2022.
@mastersthesis{hif_review_2022,
title={Hyperspectral Image Fusion: A Comprehensive Review},
author={Miguel Magalhรฃes},
year={2022},
school={KU Leuven},
note={Masterโs Programme in Imaging and Light in Extended Reality (IMLEX)}
}
Download and process dataset(s) (e.g.: CAVE, Harvard). This will also create MS image and downsampled HS image by a factor of 4, 8 and 16 (or any other power of 2 that you add as input to the script):
python main/dataset_CAVE.py
Run all algorithms over the datasets (you can edit run.py
to customize the combinatory that you wish to process in terms of datasets, methods and scaling factors):
python main/run.py
Finally, compute the metrics that compare the output of the image fusion methods with the ground truth data:
python main/metrics.py
Compilation of publically available hyperspectral datasets. The datasets in bold can be automatically downloaded and processed using the respective script main/dataset_{name}.py
as per the instructions above.
Dataset | Year | Qty | Resolution* | Download | Paper |
---|---|---|---|---|---|
CAVE | 2008 | 32 | 512x512x31 [400,700]nm | All | Yasuma, F., Mitsunaga, T., Iso, D., & Nayar, S. K. (2010). Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE transactions on image processing, 19(9), 2241-2253. |
Harvard | 2011 | 77 | 1040x1392x31 [420,720]nm | All | Chakrabarti, A., & Zickler, T. (2011, June). Statistics of real-world hyperspectral images. In CVPR 2011 (pp. 193-200). IEEE. |
NUS** | 2014 | 88 | ?ร?x31 [400,700]nm | - | Nguyen, R. M., Prasad, D. K., & Brown, M. S. (2014, September). Training-based spectral reconstruction from a single RGB image. In European Conference on Computer Vision (pp. 186-201). Springer, Cham. |
iCVL** | 2016 | 201 | 1392ร1300x519 [400,1000]nm | All | Arad, B., & Ben-Shahar, O. (2016, October). Sparse recovery of hyperspectral signal from natural RGB images. In European Conference on Computer Vision (pp. 19-34). Springer, Cham. |
As a demo image, we include the hyperspectral measurement of a resolution chart (ISO 12233:2017 Edge eSFR Inkjet chart) with a resolution of 512x512x108 and a wavelength interval from 403.09nm to 717.54nm, measured with a Specim IQ camera.
Additionally, remote sensing hyperspectral scenes are also available and widely used accross the field.
Click to show list of Hyperspectral Remote Sensing Scenes
Below, we list the publically available hyperspectral remote sensing scenes. The ones in italic were collected by the GIC from EHU, and can be downloaded using main/_dataset_EHU.py
, the processing part to generate the MS and downsampled HS images is still missing.
Further remote sensing scenes can be found at rslab.ut.ac.ir/data.
* the first line represents the size of the spectral cubes (width x height x spectral bands), and the second line the wavelength interval of the dataset.
** script for automatic download and processing not implemented yet.
*** some bands in between were removed.
Hyperspectral image fusion (HIF) methods with code publicly available.
Methods with code available together with an implemented wrapper in this repository (some of the wrappers are adapted from "Hyperspectral and Multispectral Data Fusion: A Comparative Review" 1).
* pan-sharpening methods adapted to HSโMS fusion 1 via hypersharpening 2.
Code is available but wrapper is not implemented yet.
* code available in another repo (from a different paper)
Extensions of HSI methods with publicly available code. These should be regarded as extensions to the base pipelines and not as a separate methods. These take as input a super-resolution image (output of the HSI method) together with the MS and HS images (original HSI method input); and provide as input an improved super-resolution image. The wrappers for these extensions are not implemented in this repository yet.
To evaluate the quality of the methods, the output of the superresolution methods is compared with the ground truth of the dataset. We compute several metrics (listed below) using sewar.
* to be implemented in the future.
pip install -r auxiliary/requirements.txt
- SPAMS-2.6
- MatConvNet
Footnotes
-
Yokoya, N., Grohnfeldt, C., & Chanussot, J. (2017). Hyperspectral and multispectral data fusion: A comparative review of the recent literature. IEEE Geoscience and Remote Sensing Magazine, 5(2), 29-56. [paper] [code] โฉ โฉ2
-
Selva, M., Aiazzi, B., Butera, F., Chiarantini, L., & Baronti, S. (2015). Hyper-sharpening: A first approach on SIM-GA data. IEEE Journal of selected topics in applied earth observations and remote sensing, 8(6), 3008-3024. [paper] โฉ