# Hyperspectral imagery super-resolution by sparse representation and spectral regularization

- Yongqiang Zhao
^{1}Email author, - Jinxiang Yang
^{1}, - Qingyong Zhang
^{1}, - Lin Song
^{1}, - Yongmei Cheng
^{1}and - Quan Pan
^{1}

**2011**:87

https://doi.org/10.1186/1687-6180-2011-87

© Zhao et al; licensee Springer. 2011

**Received: **31 March 2011

**Accepted: **12 October 2011

**Published: **12 October 2011

## Abstract

For the instrument limitation and imperfect imaging optics, it is difficult to acquire high spatial resolution hyperspectral imagery. Low spatial resolution will result in a lot of mixed pixels and greatly degrade the detection and recognition performance, affect the related application in civil and military fields. As a powerful statistical image modeling technique, sparse representation can be utilized to analyze the hyperspectral image efficiently. Hyperspectral imagery is intrinsically sparse in spatial and spectral domains, and image super-resolution quality largely depends on whether the prior knowledge is utilized properly. In this article, we propose a novel hyperspectral imagery super-resolution method by utilizing the sparse representation and spectral mixing model. Based on the sparse representation model and hyperspectral image acquisition process model, small patches of hyperspectral observations from different wavelengths can be represented as weighted linear combinations of a small number of atoms in pre-trained dictionary. Then super-resolution is treated as a least squares problem with sparse constraints. To maintain the spectral consistency, we further introduce an adaptive regularization terms into the sparse representation framework by combining the linear spectrum mixing model. Extensive experiments validate that the proposed method achieves much better results.

## Keywords

## 1. Introduction

Hyperspectral sensor can acquire imagery in many contiguous and very narrow (such as 10 nm) spectral bands that typically span the visible, near-infrared, and mid-infrared portions of the spectrum (0.4-2.5 μm) [1]. A continuous radiance spectrum for every pixel can be constructed from hyperspectral imagery, and it makes the identification of land-covers of interest possible based on their spectral signatures [1, 2]. Hyperspectral remote sensing has widely been used for aerial and space imaging applications, including land use analysis, pollution monitoring, wide-area reconnaissance, and battle-field surveillance [3].

Although hyperspectral sensor can acquire higher spectral resolution information, for the instrument limitation and imperfect imaging optics, it is difficult to acquire high spatial resolution imagery. The spatial resolution is a key parameter in many applications related to space images (object detection and precise location to name a few); it is obvious that any improvement here is important [3–5]. Low spatial resolution will result in a lot of mixed pixels and greatly degrade the detection and recognition performance, affect the related application in civil and military fields. There is significant sense to enhance the hyperspectral imagery's spatial resolution. In practice, modifying the imaging optics or the sensor array is not a good option, resolution enhancement using post-processing is a better way. Super-resolution image reconstruction offers the promise of overcoming the inherent resolution limitation of imaging sensors [4, 5]. Conventional approaches to generating a super-resolution image normally require as input multiple low-resolution image of the same scene, which are registered with sub-pixel accuracy [6]. But, it is difficult for hyperspectral aerial and space remote sensing. Based on the hyperspectral imaging model [5], the super-resolution task is cast as the inverse problem of recovering the original high-resolution images, based on reasonable assumptions or prior knowledge about the observation model that maps the high-resolution image to the low-resolution ones [7]. For hyperspectral images, the assumptions and prior knowledge are not only limited to spatial domain, but also spectral domain. How to represent and utilize these assumptions and prior will affect the performance of super-resolution. For example, to ensure the spatial satisfaction of constraints, total variation (TV) model is used as prior to regularize the super-resolution problem [8]. Guo et al. [9] proposed a hyperspectral super-resolution algorithm using the spectral unmixing information and TV model. Based on the pixel neighborhood, the piecewise autoregressive (AR) models can be used to enhance the performance of restoration [10]. To get the better performance, the prior and assumption can be represented in Fourier-Wavelet domain [11].

As an emerging image modeling technique, sparse representation has successfully been used in various image super-resolution applications. The success of sparse representation owes to the development of *l*_{1}-norm optimization techniques, and the fact that natural images are intrinsically sparse in some domain [7]. Natural images can be sparsely represented using a dictionary of atoms, the dictionary can be gotten through DCT/wavelet transforming or learning. It has been proven that the sparsity of the coefficients can be served as good prior during the panchromatic image super-resolution [7, 12–15]. But, for hyperspectral image, the contents across different bands are related tightly. For hyperspectral image super-resolution, correlation among bands can be used as a prior, and more important thing is, after spatial super-resolution, the endmember of scene should not be changed. Based on these ideas, a novel hyperspectral image super-resolution method is proposed in this article. By analyzing the hyperspectral and panchromatic imaging process, we prove that dictionary learned from panchromatic images can be used to represent the hyperspectral images. Then the spectral mixing model is used to test the spectral consistency, and take it as a regularization term in super-resolution process. Finally, we present results obtained from experiments carried out on two datasets, namely a 118-band hyperspectral images captured under a controlled illumination laboratory environment and a 224-band airborne visible/infrared imaging spectrometer (AVIRIS) image.

The rest of the article is organized as follows. Section 2 introduces the related works. Section 3 presents the super-resolution algorithm based on spatial sparsity and endmember regularization. Section 4 presents experimental results and Section 5 concludes the article.

## 2. Related studies

### 2.1. Sparse representation

It has been found that natural images can generally be coded by structured primitives, e.g., edges and line segments [7], and these primitives are qualitatively similar in form to simple cell receptive fields. Olshausen and Field [16] proposed to represent a natural image using a small number of basis functions chosen out of an over-complete code set. The sparse representation of a signal over an over-complete dictionary is achieved by optimizing an objective function that includes two terms: one measures the signal reconstruction error and the other measures the signal sparsity.

**x**∈

*R*

^{ m }admits a sparse approximation over an over-complete dictionary

**Φ**∈

*R*

^{ n×m }(

*K > m*) with

*K*atoms. Then

**x**can approximately be represented as a linear combination of a few atoms from

**Φ**. An over-complete dictionary

**Φ**and the sparse coefficients

**α**are obtained by solving the following optimization problem:

where ||·||_{2} is the 2 norm. For **Φ**, each of its atoms (columns) is a unit vector in the *l*_{2} norm. They are learned by solving the above minimization problem. *l*_{1} norm regularization constraint is used to guarantee the sparseness of **α**, where $\parallel \mathbf{\alpha}{\parallel}_{1}=\sum _{i}\mid {\alpha}_{i}\mid $ and **α** = [*α*_{1};...; *α*_{
m
}]. Positive constant *λ* controls the trade-off between accuracy of reconstruction and sparseness of **α**. The cost function given above is non-convex with respect to both **Φ** and **α**. However, it is convex when one is fixed. Thus, this problem can be alternating between learning **Φ** using K-SVD [17], MOD [18], or gradient descent [19] while fixing **α** and inferring **α** using orthogonal matching pursuit (OMP) [20] while fixing **Φ**.

### 2.2. Hyperspectral and panchromatic images representation

*λ*

_{1}and

*λ*

_{2}can be written as

*L*(

*λ*) is the spectral radiance at the sensor's entrance pupil,

*g*

_{ i }(

*λ*) is the spectral response function of the sensor between wavelength

*λ*

_{1}and

*λ*

_{2}.

*K*

_{ i }is the constant related to sensor such as electronic gain, the detector saturation electrons, the quantization levels, the area of the entrance aperture, and so on. ${B}_{\left({\lambda}_{1}~{\lambda}_{2}\right)}$ is the noise caused by dark signal. For same scene imaged by different sensors,

*L*(

*λ*) is same. Figure 1 shows the principle of hyperspectral and panchromatic imaging processes.

Although image contents can vary a lot from image-to-image, it has been found that the micro-structures of images can be represented by a small number of structural primitives (e.g., edges, line segments, and other elementary features) [7]. Here, we can assume that spatial micro-structures information in hyperspectral bands can be derived from a series of panchromatic images. Based on this assumption, we can use the panchromatic images to enhance the spatial resolution of hyperspectral images.

## 3. Hyperspectral imagery super-resolution (HISR) with sparsity based regularization

*X*from its degraded measurement

*Y*. The hyperspectral imagery acquisition process can be modeled as:

*W*represents the down-sampling operator and

*H*represents a blurring filter, and

*υ*is the additive noise [2]. In this article, we just consider the spatial down-sampling and blurring operator. We can get the estimation of high-quality imagery $\widehat{X}$ through resolving the inverse of (3):

where ||·||_{
F
} is the F norm. The estimation of $\widehat{X}$ by formula (4) is a ill-posed inverse problem, since for a given low-resolution input *Y*, infinitely many high-resolution images *X* satisfy the reconstruction constraint. To find a better solution, prior knowledge of hyperspectral imagery can be used to regularize the HISR problem. We regularize the problem via the following prior on small patch *x* of *X*.

### 3.1. Spatial sparsity regularization

**Φ**= [

*ϕ*

_{1,}...,

*ϕ*

_{ m }]∈

*R*

^{ n×m }trained from high-resolution images to model the structure from panchromatic images. Based on this dictionary

**Φ**, the hyperspectral imagery acquisition process can be modeled as

**Λ**= [

**α**

_{1};...;

**α**

_{ m }] is the

*m*×

*N*matrix where most of the elements in

**α**

_{i}(

*i*= 1, 2,...,

*m*) are close to zero.

**Λ**= [

**α**

_{1},

**α**

_{2},...,

**α**

_{ N }] where

*N*is the number of band. That is for

*n*th band, we have

### 3.2. Spectral regularization

*N*endmember material signals. We will arrange the signals

*m*

_{ i }as the columns of the endmember matrix

*M*.

*α*

_{ i }is the abundance of endmember

*m*

_{ i }, and it satisfies the two constraints: non-negative and normality. We will write the abundance values

*α*

_{1},...,

*α*

_{ N }as a column vector

*a*. For mathematical simplicity, it is common to assume a linear mixing model:

*f*is the given spectral signal,

*n*is the noise. The most straightforward approach for solving the linear problem (8) is by constrained least squares minimization

*γ*is a constant balancing the contribution of the spectral regularization term. For a small patch, we write the third term $\sum _{{x}_{i}\in x}\parallel {x}_{i,n}-Ma{\parallel}_{2}^{2}$ as $\parallel \left(I-B\right)\varphi {\mathbf{\alpha}}_{n}{\parallel}_{2}^{2}$, where

*I*is the identity matrix and

This is a reweighted *l*_{1}-minimization problem, which can effectively be solved by the iterative shrinkage algorithm [21].

## 4. Experiment results and analysis

To test the performance of the super-resolution reconstruction algorithm, two kinds of experiments are designed. In the first experiment, the proposed super-resolution algorithm is tested on data which is collected under controlled illumination. The spectral span of images used for training and testing is same in this scene. In the second experiment, the algorithm is tested on the AVIRIS hyperspectral datasets. The super-resolution results by proposed algorithm are compared with interpolation technique and hyperspectral super-resolution method using endmember-based TV model [9].

### 4.1. Evaluation measures

where *S*_{peak,b}is the peak signal value at *b*'th band, MSE is the mean square error between the ground truth and the estimated high-resolution signal.

**x**and

**y**can be defined as [22]

**x**and

**y**can be defined as [9]

where Ω means the whole image spatial domain. *PC*_{
m
}(**z**) = max{*PC*_{
x
}(**z**), *PC*_{
y
}(**z**)}, where *PC*_{
x
}(**z**) is phase congruency for a given position **z** of image **x**. *S*_{
L
} (**z**) is the gradient magnitude for a given position **z**.

### 4.2. **Indoor experiments**

## 5. Outdoor experiments

## 5. Conclusion

We propose a novel hyperspectral super-resolution algorithm by utilizing the sparse representation and spectral mixing model. Single hyperspectral image super-resolution is typical ill-posed inverse problem, prior knowledge of data can be used to regularize the super-resolution problem. Considering the fact that the micro-structures of images can be represented as linear combination of atoms in the pre-trained dictionaries, we utilize the sparsity of combination coefficient to solve the inverse problem. To further improve the spectral quality of reconstructed images, we introduced a spectral mixing model-based image restoration framework. Spectral mixing models were learned from the training dataset and were used to regularize the image local smoothness. The experimental results on two hyperspectral images showed that the proposed approach outperforms state-of-the-art methods in both PSNR, visual, and spectral quality.

## Declarations

### Acknowledgements

This work was supported by the Natural Science Foundation of China under grants Nos. 61071172, 60602056, and 60634030, and the Aviation Science Funds 20105153022, Sciences Foundation of Northwestern Polytechnical University No. JC200941.

## Authors’ Affiliations

## References

- Gu Y, Zheng Y, Zhang J:
**Integration of spatial-spectral information for resolution enhancement in hyperspectral images.***IEEE Trans Geosci Remote Sens*2008,**46**(5):1347-1357.View ArticleGoogle Scholar - Akgun T, Altunbasak Y, Mersereau RM:
**Super-resolution reconstruction of hyperspectral images.***IEEE Trans Image Process*2005,**14**(11):1860-1875.View ArticleGoogle Scholar - Zhao Y, Zhang L, Kong SG:
**Band-subset-based clustering and fusion for hyperspectral imagery classification.***IEEE Trans Geosci Remote Sens*2011,**49**(2):747-756.View ArticleGoogle Scholar - Mianji FA, Zhang Y, Sulehria HK:
**Super-resolution challenges in hyperspectral imagery.***Inf Technol J*2008,**7**(7):1030-1036. 10.3923/itj.2008.1030.1036View ArticleGoogle Scholar - Akgun T, Altunbasak Y, Mersereau RM:
**Super-resolution reconstruction of hyperspectral images.***IEEE Trans Image Process*2005,**14**(11):1860-1873.View ArticleGoogle Scholar - Choi M, Kim R, Nam M, Kim HO:
**Fusion of multispectral and panchromatic satellite images using the curvelet transform.***IEEE Geosci Remote Sens Lett*2005,**2**(2):136-139. 10.1109/LGRS.2005.845313View ArticleGoogle Scholar - Dong W, Zhang L, Shi G, Wu X:
**Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization.***IEEE Trans Image Process*2011,**20**(7):1838-1857.MathSciNetView ArticleGoogle Scholar - Chan T, Esedoglu S, Park F, Yip A:
**Recent development in total variation image restoration.**In*Mathematical Models of Computer Vision*. Edited by: Paragios N, Chen Y, Faugeras O. Springer, New York; 2005.Google Scholar - Guo Z, Wittman T, Osher S:
**L1 unmixing and its application to hyperspectral image enhancement.**In*Proc SPIE Conference on Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XV*. Orlando, Florida; 2009.Google Scholar - Minami K, Kawata S, Minami S:
**Superresolution of Fourier transform spectra by autoregressive model fitting with singular value decomposition.***Appl Optics*1985,**24**(2):162-167. 10.1364/AO.24.000162View ArticleGoogle Scholar - Robinson MD, Toth CA, Lo JY, Farsiu S:
**Efficient Fourier-wavelet super-resolution.***IEEE Trans Image Process*2010,**19**(10):2669-2681.MathSciNetView ArticleGoogle Scholar - Yang J, Wright J, Huang TS, Ma Y:
**Image super-resolution via sparse representation.***IEEE Trans Image Process*2010,**19**(11):2861-2873.MathSciNetView ArticleGoogle Scholar - Dong W, Shi G, Zhang L, Wu X:
**Super-resolution with nonlocal-regularized sparse representation.***SPIE VCIP*2010, 7744.Google Scholar - Dong W, Zhang L, Shi G, Wu X:
**Nonlocal-back-projection for adaptive image enlargement.***ICIP*2009.Google Scholar - Dong W, Zhang L, Shi G:
**Centralized sparse representation for image restoration.***ICCV*2011.Google Scholar - Olshausen BA, Field DJ:
**Emergence of simple-cell receptive field properties by learning a sparse code for natural images.***Nature*1996,**381:**607-609. 10.1038/381607a0View ArticleGoogle Scholar - Aharon M, Elad M, Katz A, Bruckstein Y:
**The K-SVD: an algorithm for designing of overcomplete dictionaries for sparse representations.***IEEE Trans Signal Process*2006,**54**(11):4311-4322.View ArticleGoogle Scholar - Lee H, Battle A, Raina R, Andrew YN:
**Efficient sparse coding algorithms.***NIPS*2007,**19:**801-808.Google Scholar - Mairal J, Bach F, Ponce J, Sapiro G:
**Online learning for matrix factorization and sparse coding.***J Mach Learn Res*2010,**11**(1):19-60.MathSciNetGoogle Scholar - Rubinstein R, Zibulevsky M, Elad M:
**Efficient implementation of the k-svd algorithm using batch orthogonal matching pursuit.***Technical Report - CS Technion*2008.Google Scholar - Daubechies I, Defriese M, DeMol C:
**An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.***Commun Pure Appl Math*2004,**57:**1413-1457. 10.1002/cpa.20042View ArticleGoogle Scholar - Wang Z, Bovik AC, Rahim Sheikh H, Simoncelli EP:
**Image quality assessment: from error visibility to structural similarity.***IEEE Trans Image Process*2004,**13**(4):600-612. 10.1109/TIP.2003.819861View ArticleGoogle Scholar - Zhang L, Zhang L, Mou X, Zhang D:
**FSIM: a feature similarity index for image quality assessment.***IEEE Trans Image Process*2011.Google Scholar - Foster DH, Nascimento SMC, Amano K:
**Information limits on neural identification of coloured surfaces in natural scenes.***Visual Neurosci*2004,**21:**331-336. 10.1017/S0952523804213335View ArticleGoogle Scholar - Zhao Y, Gong P, Pan Q:
**Object detection by spectropolarimeteric imagery fusion.***IEEE Trans Geosci Remote Sens*2008,**46**(10):3337-3345.View ArticleGoogle Scholar - Plaza A, Martinez P, Perez R, Plaza J:
**A quantitative and comparative analysis of endmember extraction algorithm from hyperspectral data.***IEEE Trans Geosci Remote Sens*2004,**42**(3):650-663. 10.1109/TGRS.2003.820314View ArticleGoogle Scholar

## Copyright

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.