Multilayer graph spectral analysis for hyperspectral images

Abstract

Hyperspectral imaging has broad applications and impacts in areas including environmental science, weather, and geo/space exploration. The intrinsic spectral–spatial structures and potential multi-level features in different frequency bands make multilayer graph an intuitive model for hyperspectral images (HSI). To study the underlying characteristics of HSI and to take the advantage of graph signal processing (GSP) tools, this work proposes a multilayer graph spectral analysis for hyperspectral images based on multilayer graph signal processing (M-GSP). More specifically, we present multilayer graph (MLG) models and tensor representations for HSI. By exploring multilayer graph spectral space, we develop MLG-based methods for HSI applications, including unsupervised segmentation and supervised classification. Our experimental results demonstrate the strength of M-GSP in HSI processing and spectral–spatial information extraction.

1 Introduction

1.1 Background and significance

Hyperspectral imaging is an analytical technique operating on images at different wavelengths for given geographical areas [1,2,3]. Exploiting a wealth of spectral–spatial information, hyperspectral images (HSIs) have seen broad applications in areas such as urban mapping, environment management, crop analysis and food safety inspection [4,5,6]. Among a variety of hyperspectral imaging analysis for these applications, graph-based approaches have recently attracted substantial attentions due to its power in uncovering the underlying spatial structures of HSI. Modeling pixels as nodes and their internal interactions as edges, graph models can be constructed to capture the geometric information of hyperspectral images. Advanced tools, such as graph signal processing (GSP) [7] and graph convolutional networks (GCN) [8], can next be applied in HSI processing tasks, including coding [9], classification [10] and reconstruction [11].

Despite the successes in hyperspectral imaging analysis, most of the graph-based methods focus primarily on spatial geometry and consider the same graph connections for the multiple spectral bands, shown as Fig. 1b. However, such a single-layer graph model fails to explore different spectrum features of each individual band. For example, Fig. 1a shows that different spectral bands may display different distributions of the pixel volumes, which could mean different spatial structures. The high dimensionality of HSIs and their heterogeneous underlying structures for different bands do not lend themselves to traditional single-layer graphs. A more generalized geometric model needs to be explored to represent HSI efficiently.

Fig. 1

Examples of graph models for HSI: a Stack of 3-band images in Indian Pines where six pixels/superpixels are selected as examples of nodes; b A single-layer graph model for the 3-band HSI in (a) based on feature similarity; c Example of a multilayer graph model with 3 layers

To capture the complex spectral–spatial interactions among pixels, recent attention has been attracted in a high-dimensional graph named as multilayer networks (also known as multilayer graphs) [12]. In a multilayer network, nodes are grouped into meaningful clusters called “layers” depending on their features. Different layers could have heterogeneous graph structures and interact with each other. An example of multilayer network (graph) is shown in Fig. 1c. Modeling each spectral frame as layers, the multilayer graph (MLG) can be an intuitive model to capture the spectral–spatial interactions. Now, one major problem is how to generalize traditional graph-based analysis to multilayer graphs and process the high-dimensional multilayer structures efficiently. To compensate, some works would represent each layer with an individual graph while neglecting the interlayer correlations [13]. Although one can also construct interlayer connections for spectral frames [9], the spectral (interlayer) and spatial (intralayer) connections are still processed separately [14], where certain joint spectral–spatial information in HSI can be neglected. Some works also consider a two-step analysis for MLG. For example, in [15], a two-step transform is proposed, where graph Fourier transform (GFT) is applied first to interlayer connections and then to interlayer interactions. However, the interlayer and intralayer connections are still processed separately in different graph Fourier spaces. For these reasons, one untackled challenge is how to capture such heterogeneous spectral–spatial structures for the MLG model of HSI in an integrative manner jointly. Recently, a tensor-based framework of multilayer graph signal processing (M-GSP) is proposed in [16] to analyze the inter- and intra-layer connections. Defining MLG spectral space based on tensor decomposition, MLG spectral analysis can be implemented for HSI, which allows both joint and individual analysis for spectral–spatial interactions in HSI by incorporating the information of inter- and intra-layer connections.

Our goal is to explore MLG in HSI processing and provide a guideline for analyzing the high-dimensional spectral–spatial correlations of HSI based on M-GSP. In this work, we revisit the M-GSP framework and introduce MLG-based spectral analysis for hyperspectral imaging. More specifically, we first introduce the models and algebraic representation of HSI in M-GSP. Next, we develop M-GSP spectral analysis to extract features for HSI analysis. To illustrate the power of MLG in HSI, we investigate the applications of M-GSP in the feature extraction for both unsupervised HSI segmentation and supervised HSI classification as guidelines of MLG-based HSI analysis. We summarize our contributions in this work as follows:

• We introduce an MLG model, together with an alternative singular space for HSI analysis.

• We provide guidelines of applying M-GSP in HSI and suggest several application examples with novel M-GSP algorithms:

  • We develop an unsupervised HSI segmentation method based on the M-GSP spectral clustering as an application example.

  • We derive an MLG-based method for supervised HSI classification by jointly utilizing multi-resolution information. We further propose several novel schemes for decision fusion of the results from different resolutions.

We test our algorithms using the widely used Indian Pines dataset, Pavia University dataset and Salinas dataset. Our experimental results demonstrate the strength of M-GSP in modeling some spectral–spatial structures in HSI, and the efficiency of the proposed HSI segmentation algorithms.

We organize the rest of this manuscript as follows. Starting with the introduction of MLG model and M-GSP representations of HSI in Sect. 2.1, we investigate unsupervised HSI segmentation based on M-GSP spectral clustering in Sect. 2.2. Next, we introduce the M-GSP framework of supervised HSI classification in Sect. 2.3, and further propose several methods for the decision fusion of different resolutions. Section 3 presents details on the experiments and results of the proposed methods. Finally, we summarize our works in Sect. 4.

1.2 Related work

In this section, we provide an overview on graph signal processing and graph learning in hyperspectral images.

1.2.1 Graph signal processing and graph convolutional networks

Graph signal processing (GSP) has emerged as an exciting and promising new paradigm for processing large datasets with complex structures, owing to its power to extract underlying relationships among signals [7, 17]. Modeling data points and their interactions as a graph, a graph Fourier space can be defined according to the eigenspace of a graph representation for data analysis [18] including image processing [19], point cloud resampling [20] and video compression [21]. The framework of GSP is further generalized with fundamentals within the graph Fourier space, such as graph Fourier transform [22], sampling theory [23], graph wavelet [24] and stationary processing [25]. Beyond normal graphs, GSP has also been considered for high-dimensional graphs, such as multilayer graphs [16], hypergraphs [26] and simplicial complexes [27]. In addition, Graph neural networks (GNN) and graph convolutional networks (GCN) have become important tools in data analysis [8].

1.2.2 Graph-based hyperspectral image analysis

With the development of signal processing and learning over graphs, graph-based tools have attracted significant attentions in HSI analysis. In [9], a coding method is developed for HSI based on graph wavelets. Inspired by graph spectral transform, GSP-based methods have also been developed for the compression of hyperspectral scenes [28]. Exploring spectral–spatial features of HSI in graph spectral space, a feature extraction algorithm is proposed using collaboration-competition graphs [29]. In addition to basic GSP analysis, graph learning also has shown promise in hyperspectral imaging. For example, leveraging graph spectral convolution, GCN becomes an important tool in HSI applications, including HSI classification and reconstruction [10, 11, 30,31,32]. In addition to GCNs, a semi-supervised hyperspectral dimensionality reduction approach is introduced in [33] by propagating labels on a learnable graph. In [34], a novel semi-supervised cross-modality learning framework is introduced for HSI analysis with a learnable manifold. Other graph-based learning frameworks for HSI include X-ModalNet [35] and Deep-hybrid [36]. Interested readers are referred to a recent survey [37] to explore further.

2 Methods

In this section, we introduce the models and representations for hyperspectral images within the M-GSP framework. Then, we will introduce the M-GSP-based approaches for HSI unsupervised segmentation and supervised classification.

To avoid confusion of the term “network” because of its different meanings in communication and deep learning, we will use the less ambiguous term of “multilayer graph” instead of “multilayer network” in the remaining of this work.

2.1 Models and representations for hyperspectral images

We first introduce the MLG models and representations for HSI.

2.1.1 Superpixel segmentation for HSI

Before venturing into the M-GSP analysis, we first introduce the superpixel segmentation for HSI. In traditional graph-based HSI analysis, image pixels act as nodes and their pair-wise distances are calculated to form a graph [38]. However, given a large number of pixels, it becomes inefficient and sometimes impossible to implement full graph-based analysis for pixel-based HSIs. Practically, since pixels within a small region may share similar features, grouping neighboring pixels into superpixels could be a more practical way for graph construction.

In general, a suitable superpixel segmentation algorithm for HSI classification should exhibit low computation complexity and accurate detection of the object boundaries [39]. One category of superpixel segmentation in HSI uses image features such as brightness, color and texture cues, to estimate the location of segment boundaries. In [40], a superpixel estimation is developed by adopting the ultrametric contour map (UCM) approaches to the hyperspectral volumes. In [41], the band smoothness is jointly considered with general features to group pixels. Graph-based segmentation approaches are also common in superpixel segmentation [42]. In [43], an eigen-based solution to normalized cuts (NCuts) is used for superpixel group. However, such eigen-based methods tend to suffer from time-consuming graph construction and matrix decomposition.

For more efficient superpixel segmentations, our work in this manuscript considers the entropy rate superpixel segmentation (ERS). Compared to other superpixel construction algorithms, ERS is very efficient as it only takes about 2.5 seconds to segment an image of size $481\times 321$, while achieving superior performances in terms of standard metrics such as undersegmentation errors [44]. In ERS [44], a dataset is modeled as a graph ${\mathcal {G}}=\{{\mathcal {V}}, {\mathcal {E}}\}$, in which the pixels serve as the nodes ${\mathcal {V}}$ and their pairwise similarities are represented by edges ${\mathcal {E}}$. Next, a subgraph ${\mathcal {A}}=\{{\mathcal {V}},{\mathcal {L}}\}$ is formed by choosing a subset of edges ${\mathcal {L}}\subseteq {\mathcal {E}}$, such that ${\mathcal {A}}$ consists of fewer connected components. To obtain such a subgraph, the problem can be formulated as

$$\mathcal{L}^{*} = \arg _{\mathcal{L}} \max \;{\text{Tr}}\left\{ {H\left( \mathcal{L} \right) + \alpha T\left( \mathcal{L} \right)} \right\}$$

(1)

$$\begin{aligned}&s.t. \quad {\mathcal {L}}\subseteq {\mathcal {E}}, \end{aligned}$$

(2)

in which entropy rate term $H({\mathcal {L}})$ favors more compact clusters, and regularizing term $T({\mathcal {L}})$ punishes large cluster size. The balancing parameter is given by $\alpha =\beta K\epsilon$ where $\epsilon =0.5$, K is the number of superpixels, and $\beta$ is the calculated based on the maximal entropy rate increase. Interested readers may refer to [39,44] for more details about the design of $H({\mathcal {L}})$ and $T({\mathcal {L}})$, as well as other details on ERS algorithm. Based on the objective function of Eq. (1), a greedy algorithm can be implemented to solve the problem.

Fig. 2

Scheme of MLG-based unsupervised HSI segmentation

ERS can segment pixels into superpixels for M-GSP with low complexity and good efficiency. From the constructed superpixels, we average the positions and features of all the pixels within each superpixel as its new position and feature for MLG construction. Note that, our goal is to provide a guideline of applying M-GSP in HSI and show M-GSP can achieve robust performance even with simple methods. We adopt ERS superpixel construction and averaging-based feature aggregation in this work. We plan to consider other advanced approaches for superpixel construction and feature generation in future works.

2.1.2 Multilayer graph construction for HSI datasets

With superpixel-represented HSI, we now begin multilayer graph construction outlined as Fig. 2. An HSI ${\mathbf {X}}\in {\mathbb {R}}^{K\times N}$, containing K spectral frames and N superpixels, can be modeled by a multilayer graph with M layers and N nodes in each layer. Specifically, the MLG consists of the following attributes

• Layers: To construct a MLG, we define layers based on the spectral bands. Since different spectral frames may share similar features, we first divide the bands into M clusters, i.e., ${\mathbf {X}}_i\in {\mathbb {R}}^{{K}_i\times N}$, $i=1,\cdots ,M$ and $\sum _{i=1}^{M}K_i=K$. Next, each cluster serves as one layer in the multilayer graph. Various clustering methods can generate features ${\mathbf {X}}_i$ for layer i. For example, one can divide spectral band based on a range of wavelengths. To capture correlation across different bands more efficiently, the k-means clustering is applied for band division.

• Nodes: Shown as Fig. 2, the multilayer graph can be viewed as embedding N superpixels $\{x_1,x_2,\cdots ,x_N\}$ into M layers $\{l_1,\cdots ,l_M\}$. Then, we can form a multilayer graph with M layers and N nodes in each layer. We characterize each superpixel using the divided attributes, i.e., ${\mathbf {X}}_{\alpha ,j}\in {\mathbb {R}}^{K_i}$ for the superpixel j’s embedded node in layer $\alpha$.

• Interlayer connections: For interlayer connections, each embedded node is connected to its counterparts in other layers, i.e., fully connected for all the embedded nodes of the same superpixel (multiplex structure [12]). Denote the weight of edge between superpixel j’s embedded node in layer $\beta$ and superpixel i’s embedded node in layer $\alpha$ as $A_{\alpha i \beta j}$. The weights of interlayer connections can be calculated as

  $$\begin{aligned} A_{\alpha i \beta j}=\left\{ \begin{aligned} 1,&\quad \alpha \ne \beta , i=j;\\ 0,&\quad {\text {otherwise}}, \end{aligned} \right. \end{aligned}$$

  (3)

  where each term indicates link presence. One can also assign weights to $A_{\alpha i \beta i}$ based on the feature similarities. Note that, here we introduce the weighted multiplex structure as an example for interlayer connections. Our M-GSP framework is applicable to various interlayer structures. Readers could construct graphs depending on the specific dataset and learning task.

• Intralayer connections: For the intralayer connections, we calculate the weights between the embedded nodes of superpixels i and j in layer $\alpha$ based on the localized Gaussian distance as follows:

  $$\begin{aligned} A_{\alpha i \alpha j}=\left\{ \begin{aligned} e^{-\frac{||{\mathbf {X}}_{\alpha ,i}-{\mathbf {X}}_{\alpha ,j}||^2_2}{\sigma ^2}},&\quad \text{ dis}_1({\mathbf {X}}_{\alpha ,i},{\mathbf {X}}_{\alpha ,j})<p,\\&\quad \text{ dis}_2(p(\alpha ,i),p(\alpha , j))<q;\\ 0,&\quad \text{ otherwise. } \end{aligned} \right. \end{aligned}$$

  (4)

  where $\sigma$, p and q are design parameters and $p(\alpha ,i)$ is the position of the superpixel i in layer $\alpha$.

Beyond the traditional Gaussian distance, our intralayer connections consider two conditions for determining the presence of links: (1) features between two nodes should be similar; and (2) two connected superpixels should be in a localized region in the HSI. The first condition ensures the similarity of connected nodes while the second condition emphasizes geometric closeness in the HSI. For an initial setup of the parameters, we define $\text{ dis}_1=||{\mathbf {X}}_{\alpha ,i}-{\mathbf {X}}_{\alpha ,j}||_2$, and define $\text{ dis}_2$ as the Euclidean distance between the respective centroids of two superpixels. In terms of designing parameters, we set p as the mean of all pairwise similarities and tune the parameters q, $\sigma$ based on the specific dataset. Details of parameter selection in practical applications can be found in Sect. 3.2.

2.1.3 Algebraic representations for HSI in M-GSP

Following the aforementioned process, we can easily construct HSI as a MLG with M layers and N nodes in each layer. In M-GSP [16], such multilayer graph structure can be intuitively represented by a fourth-order tensor ${\mathbf {A}}\in {\mathbb {R}}^{M\times N\times M \times N}$ defined as

$$\begin{aligned} {\mathbf {A}}=(A_{\alpha i \beta j})\quad 1\le \alpha ,\beta \le M, 1\le i,j\le N, \end{aligned}$$

(5)

where each entry is calculated as Eqs. (3) and (4). Similar to normal graphs, Laplacian tensor $\mathbf {L=D-A}\in {\mathbb {R}}^{M\times N\times M \times N}$ can be defined as the alternative representation of MLG, where ${\mathbf {D}}$ is the degree tensor with the node degrees as its diagonal entries [16].

Discussion Before we venture into the M-GSP spectral representation of HSI, we provide a discussion on the interpretation of tensor representation and its relationship to the aforementioned “embedding” process in Sect. 2.1.2. Given a set of superpixels ${\mathcal {X}}=\{x_1,x_2,\cdots ,x_N\}$, one can construct a vector ${\mathbf {z}}_i\in {\mathbb {R}}^{N}$ to characterize the underlying structural features of each superpixel i. Next, the interaction between two superpixels can be represented by a second-order tensor ${\mathbf {A}}_X=\sum _{i,j=1}^N a_{ij} {\mathbf {z}}_i \circ {\mathbf {z}}_j\in {\mathbb {R}}^{N\times N}$, where $a_{ij}$ quantifies the relationship between superpixel i and j. Similarly, given a set of clustered bands (layers) ${\mathcal {L}}=\{l_1,l_2,\cdots ,l_M\}$, a vector ${\mathbf {z}}_{\alpha }\in {\mathbb {R}}^{M}$ can capture the structural properties of layer $\alpha$, and the connectivity between two layers could be represented by ${\mathbf {A}}_L=\sum _{\alpha ,\beta =1}^M b_{\alpha \beta }{\mathbf {z}}_\alpha \circ {\mathbf {z}}_\beta \in {\mathbb {R}}^{M\times M}$. Following this approach, connectivity between the embedded nodes of superpixels in the layers can be represented by a fourth-order tensor to describe the features of superpixels and layers, i.e.,

$$\begin{aligned} {\mathbf {A}}=\sum _{\alpha ,\beta =1}^{M}\sum _{i,j=1}^N w_{\alpha i \beta j}{\mathbf {z}}_\alpha \circ {\mathbf {z}}_i\circ {\mathbf {z}}_\beta \circ {\mathbf {z}}_j \in {\mathbb {R}}^{M\times N\times M \times N}, \end{aligned}$$

(6)

where $\circ$ is the tensor outer product [45], $w_{\alpha i \beta j}$ is the weight of connection between the superpixel i’s embedded node in layer $\alpha$ and the superpixel j’s embedded node in layer $\beta$. More specially, if we select the vector ${\mathbf {z}}_i=[0,\cdots ,0,1,0,\cdots ,0]^\mathrm {T}$ in which the only nonzero element is the ith element (equal to 1) for both layers and superpixels, the fourth-order tensor becomes the adjacency tensor of the multilayer network. Interested readers may refer to [12, 16] for more details of the adjacency and Laplacian tensor.

2.1.4 Spectral representations for HSI in M-GSP

We now introduce the spectral representation of HSI in M-GSP. In M-GSP, multiple MLG spectra are introduced for different purposes. Since the MLG singular space is more robust when analyzing the order-wise features, we focus on the singular analysis in this work. More details of other MLG spectral analysis, e.g., M-GSP eigen-tensor analysis, can be found in [16]. With the adjacency tensor ${\mathbf {A}}\in {\mathbb {R}}^{M\times N\times M \times N}$, it can be decomposed via higher-order singular value decomposition (HOSVD) [46] as

$$\begin{aligned} {\mathbf {A}}= {\mathbf {S}}\times _1 {\mathbf {U}}^{(1)}\times _2 {\mathbf {U}}^{(2)}\times _3 {\mathbf {U}}^{(3)}\times _4 {\mathbf {U}}^{(4)}, \end{aligned}$$

(7)

where $\times _n$ is the n-mode product [45] and ${\mathbf {U}}^{(n)}=[{\mathbf {u}}^{(n)}_1\quad {\mathbf {u}}^{(n)}_2\quad \cdots \quad {\mathbf {u}}^{(n)}_{I_n}]$ is a unitary $(I_n\times I_n)$ matrix, with $I_1=I_3=M$ and $I_2=I_4=N$. ${\mathbf {S}}$ is a $(I_1\times I_2\times I_3 \times I_4)$-tensor of which the subtensor ${\mathbf {S}}_{i_n}$ obtained by freezing the nth index to $\alpha$:

• $<{\mathbf {S}}_{i_n=\alpha },{\mathbf {S}}_{i_n=\beta }>=0$ where $\alpha \ne \beta$.

• $||{\mathbf {S}}_{i_n=1}||\ge ||{\mathbf {S}}_{i_n=2}||\ge \cdots \ge ||{\mathbf {S}}_{i_n=I_n}||\ge 0$.

The Frobenius-norms $\sigma _i^{(n)}=||{\mathbf {S}}_{i_n=i}||$ is the n-mode singular value, with corresponding singular vectors in ${\mathbf {U}}^{(i)}$. Since the representing tensor shows partial symmetry in the undirected MLG, there are two modes of singular spectrum, i.e., ${\mathbf {U}}^{(1)}={\mathbf {U}}^{(3)}=({\mathbf {f}}_\alpha )$ characterizes the features of layers and ${\mathbf {U}}^{(2)}={\mathbf {U}}^{(4)}=({\mathbf {e}}_i)$ characterizes the nodes. Renotating ${\mathbf {U}}^{(1)}={\mathbf {U}}^{(3)}$ as ${\mathbf {F}}_s=[{\mathbf {f}}_1\cdots {\mathbf {f}}_M]\in {\mathbb {R}}^{M\times M}$ and ${\mathbf {U}}^{(2)}={\mathbf {U}}^{(4)}$ as ${\mathbf {E}}_s=[{\mathbf {e}}_1\cdots {\mathbf {e}}_N]\in {\mathbb {R}}^{N\times N}$, Eq. (7) can be written as

$$\begin{aligned} { {\mathbf {A}}= {\mathbf {S}}\times _1 {\mathbf {F}}_s\times _2 {\mathbf {E}}_s\times _3{\mathbf {F}}_s\times _4 {\mathbf {E}}_s.} \end{aligned}$$

(8)

With the singular tensors ${\mathbf {F}}_s$ and ${\mathbf {E}}_s$, the MLG singular transform (M-GST) for a MLG signal ${\mathbf {s}}\in {\mathbb {R}}^{M\times N}$ can be defined as

$$\begin{aligned} \check{{\mathbf {s}}}={\mathbf {F}}_s^{\mathrm {T}}{\mathbf {s}}{\mathbf {E}}_s\in {\mathbb {R}}^{M\times N}. \end{aligned}$$

(9)

Suppose that $\gamma _i$’s are the layer-wise singular values and $\sigma _i$’s are the node-wise singular values. The M-GSP spectral filter can be designed as

$$\begin{aligned} {\mathbf {s}}'={\mathbf {F}}_s \begin{bmatrix} g(\gamma _1) &{} \cdots &{} 0\\ \vdots &{} \ddots &{} \vdots \\ 0&{}\cdots &{}g(\gamma _N) \end{bmatrix} {\mathbf {F}}_s^{\mathrm {T}}{\mathbf {s}} {\mathbf {E}}_s \begin{bmatrix} f(\sigma _1) &{} \cdots &{} 0\\ \vdots &{} \ddots &{} \vdots \\ 0&{}\cdots &{}f(\sigma _N) \end{bmatrix} {\mathbf {E}}_s^{\mathrm {T}}, \end{aligned}$$

(10)

where functions $g(\cdot )$ and $f(\cdot )$ are designed by the specific tasks.

Here, we mainly focus on fundamentals of singular analysis of the undirected multilayer graphs. For more details on other concepts, such as MLG Fourier transform and M-GSP filter design, interested readers are referred to [16].

2.2 MLG-based unsupervised HSI segmentation

In this part, we propose an unsupervised segmentation approaches based on M-GSP spectral clustering.

Spectral clustering is an efficient method for unsupervised HSI segmentation [38]. Modeling HSI by a normal graph before spectral clustering, significant improvement is possible owing to its power in capturing the underlying structures [47]. However, by representing HSI by a single-layer graph, distinction of individual spectral bands might be overlooked. To capture the heterogeneous spectral–spatial structure in HSI, we propose to segment the HSI based on the M-GSP spectral clustering.

Given a HSI, we construct the multilayer graph as Sect. 2.1.2. We then calculate the spectra ${\mathbf {F}}_s=[{\mathbf {f}}_1\cdots {\mathbf {f}}_M]\in {\mathbb {R}}^{M\times M}$ and ${\mathbf {E}}_s=[{\mathbf {e}}_1\cdots {\mathbf {e}}_N]\in {\mathbb {R}}^{N\times N}$ to characterize the bands and superpixels, respectively, according to Eq. (8). Since we aim to segment superpixels into meaningful clusters, we focus on the superpixel-wise spectrum ${\mathbf {E}}_s$. Arranging ${\mathbf {e}}_i$ in the descending order of its corresponding singular value $\sigma _i$, i.e.,

$$\begin{aligned} \sigma _i=||{\mathbf {S}}_{i_2=i}||, \end{aligned}$$

(11)

where ${\mathbf {S}}_{i_2=i}\in {\mathbb {R}}^{M\times {1}\times {M} \times N}$ is the subtensor of the core tensor ${\mathbf {S}}$ in Eq. (7) by freezing the second-order $i_2=i$, we pick the first P singular vectors to preserve the most critical information for HSI based on the largest gap among the singular values. Clustering based on the P selected singular vectors and labeling each pixel within the superpixel, we can obtain a segmentation of the given HSI. The major process of MLG-based unsupervised segmentation is provided in Algorithm 1.

Fig. 3

Distribution of superpixel-wise singular values in HSI dataset

2.2.1 Discussion

Before we dive further to develop MLG-based supervised HSI classification, we provide a short conceptual discussion on the M-GSP singular tensors. In literature, SVD is an efficient method to obtain the spectrum for signal analysis, such as spectral clustering and PCA analysis. In MLG-GSP, the order-wise singular vectors can be interpreted as subspaces characterizing features of frames and superpixels, respectively. Since HOSVD is robust and efficient, transforming signals to the MLG singular space (M-GST) for the analysis of underlying structures can be a useful alternative for M-GFT. More discussions on physical meaning of MLG spectrum can be found in [16]. To better understand the property of MLG-based singular tensors, we graphically illustrate the distribution of singular values compared to a graph-based model in Fig. 3. As shown, the energy of MLG-based singular values is more concentrated in the first few dominant singular vectors in low frequency when compared against graph-based singular values. This energy concentration indicates a more convenient and low degradation implementation of spectral clustering within our proposed M-GSP framework.

2.3 MLG-based supervised HSI classification

In this part, we introduce the HSI classification based on M-GSP feature extraction.

2.3.1 Single-resolution of MLG-based HSI segmentation

We start with the single-resolution of the superpixels. In the superpixel-based classification, superpixel resolutions affect the final performance: finer resolution could capture more details, whereas coarse resolution captures the global information more efficiently. To benefit from both fine and coarse resolutions, we introduce the MLG-based spectral clustering on the fine resolution to regroup superpixels into a coarse resolution (the number of regrouped superpixels should still be larger than the number of classes) and use the regrouped features as classifier inputs. More specifically, we first implement MLG-based spectral clustering to cluster superpixels. We then combine the features of all pixels within the same cluster as the regrouped features. Finally, we update the features of each pixel by the regrouped features of its cluster and input the new features for classification. Here, we apply SVM to classify the regrouped features. The concept of our single-resolution HSI segmentation (MLG-SRC) is illustrated by Fig. 4, and the major steps are described in Algorithm 2.

Fig. 4

Scheme of single-resolution segmentation

The benefits of the proposed MLG-SRC include:

• Against a single resolution of coarse superpixels, the MLG-SRC implements an analysis step over a fine resolution and is capable of capturing detailed features. Against a single resolution of fine superpixels, the MLG-SRC substantially reduces pixel number and enhances robustness of the feature inputs to the classifier. Too many superpixels may make the features less distinctive and over-segment the regions, whereas too few superpixels may lead to boundary ambiguity.

• Traditional graph-based superpixel segmentation only captures a single-layer structure. M-GSP regrouping could reveal additional feature information across the heterogeneous multi-band structures.

• In traditional superpixel segmentation, the distinct regions are usually labeled as different superpixels. However, in MLG-SRC, superpixels from different regions may be grouped to the same cluster. Thus, regrouped features can involve similar pixels that cover a large distance and potentially generate more features.

• MLG-SRC can be easily integrated with other feature extraction or selection algorithms. Dimension reduction techniques such as PCA and ICA can potentially improve the performance when applied on features and feature groups generated by MLG-SRC.

Fig. 5

Scheme of MLG-based supervised HSI classification

Fig. 6

Accuracy of MLG-SRC over different initial resolutions

2.3.2 Multi-resolution of MLG-based HSI segmentation

1) Multi-resolution structure:

Although MLG-based spectral clustering can regroup small superpixels into larger ones and to benefit from both fine and coarse resolutions in MLG-SRC, the initial resolution setting of superpixels still affects final performance (Fig. 5). As Fig. 6 shows, different initial resolutions can lead to different levels of accuracy. It is practically difficult to determine the optimal initial number of superpixels.

Here, we consider a multi-resolution structure of classification (MLG-MRC) shown as Fig. 5. In this framework, we examine several different initial resolutions of superpixels. MLG-SRC is applied to each initial resolution to regroup the superpixels with a same reduction ratio in group numbers, i.e., $70\%$ of initial superpixels. Applying SVM to classifying the multiple regrouped superpixels, we fuse the results from different initial resolutions in final segmentation. The algorithm is described in Algorithm 3. Although the multi-resolution structures have been considered in literature, MLG-MRC exhibits two major distinctions. First, we apply a novel MLG-based clustering algorithm to regroup the superpixels and generate new features for classification. Second, we provide several novel decision fusion strategies, based on both confidence score and graph structures to be discussed below.

2) Decision fusion:

Majority voting (MV) [48] is a widely used fusion scheme for different resolutions. In this method, the label l of a specific pixel is determined by

$$\begin{aligned} l={\arg \max }_i \sum _{j=1}^{C} w_j\cdot \delta (l_j), \end{aligned}$$

(12)

where C is the number of distinct resolutions, $l_j$ is the label of the pixel in resolution j, $w_j$ is the voting strength, and $\delta (l_j)=1$ if $l_j=i$; otherwise, $\delta (l_j)=0$. Note that a basic majority voting based on equal strength $w_j={C}^{-1}$ applies the same strength to different resolutions which would ignore their differences. To improve decision fusion, we introduce several novel strategies for the decision fusion.

• Validation Accuracy (VA): One intuitive way to design decision strength is based on validation accuracy. If the validation accuracy of a certain resolution is larger, it might also have better performance in the test data. Thus, we can assign larger weights to the resolution with better validation accuracy. Here, we can apply the validation accuracy directly as the weighting strength $w_j$ for resolution j to fuse the decision according to Eq. (12).

• Decision Value (DV): As one alternative, the decision probability for each class of pixels can be used as the weight. In multi-class SVM, the predicted label is determined according to the decision value ${\mathbf {p}}\in {\mathbb {R}}^{C}$, where C is the number of classes [49]. If the decision value is larger, it means that SVM has higher classification confidence. Thus, decision values can also describe the confidence levels of classification results. Let ${\mathbf {p}}_{ij}$ be the decision value of pixel i in jth resolution. We set the weight of $l_{ij}$ to

  $$\begin{aligned} w_{ij}=\max _k {\mathbf {p}}_{ij}(k). \end{aligned}$$

  (13)

  Unlike validation accuracy which is the same for all pixels in each resolution, this weight based on decision value may vary even for pixels at the same resolution.

• Graph Total Variation (TV): Graph-based metrics can serve as weights. For a robust setup of superpixels, signals should be smooth and exhibit stable underlying graph structure. To this end, we introduce graph-based total variation to measure smoothness. Given a superpixel segmentation j of a HSI with N superpixels and K spectral frames, we regenerate the features of each superpixel by averaging all pixels within. We then construct a single-layer graph based on Gaussian distance to measure similarity between different superpixels. Defining a Laplacian matrix by ${\mathbf {L}}=\mathbf {D-A}$ where ${\mathbf {D}}$ is the degree matrix and ${\mathbf {A}}$ is the adjacency matrix, the total variation [7] of the feature signal ${\mathbf {s}}_p\in {\mathbb {R}}^{N}$ for the pth band frame over ${\mathbf {L}}$ is

  $$\begin{aligned} \text{ TV}_p=||{\mathbf {s}}_p-\frac{1}{|\lambda _{\max }|}{{\mathbf {L}}}{{\mathbf {s}}}_p||_2^2, \end{aligned}$$

  (14)

  where $\lambda _{\max }$ is the largest eigenvalue of ${\mathbf {L}}$. Total variation describes the propagation differences between two steps. A smaller total variation indicates a more smooth signal. With K frames in total, final smoothness for resolution j is defined as $\text{ SM}_j=\frac{1}{K}\sum _p \text{ TV}_p$. Since we prefer a larger weight for the smooth signal, the final weight of the resolution j is defined as

  $$\begin{aligned} w_j=e^{-SM_j}. \end{aligned}$$

  (15)

• Von Neumann Entropy (VN): The stability of the underlying graph structure can also indicate the confidence level of a specific superpixel resolution. In quantum theory [50], a pure state leads to a zero Von Neumann entropy. The entropy is larger if there are more mixed states in the system. Similarly, in our HSI analysis, since we prefer a stable system or a pure state on the underlying graph, the weight should be larger if the Von Neumann entropy is smaller. Here, we introduce the Von Neumann entropy to evaluate the graph stability [50]. Similar to total variation, a Laplacian matrix ${\mathbf {L}}$ can be defined with adjacency matrix ${\mathbf {A}}=(a_{pq})$ for the jth resolution. First, define $c={1}/({\sum _{p,q}a_{pq}})$ and rescale the Laplacian matrix

  $$\begin{aligned} {\mathbf {L}}_G=c\cdot (\mathbf {D-A}), \end{aligned}$$

  (16)

  We can define the weight for the jth resolution as

  $$\begin{aligned} w_j=e^{-h_j}. \end{aligned}$$

  (17)

  based on the Von Neumann entropy

  $$\begin{aligned} h_j=-{\text {Tr}}[{\mathbf {L}}_G\log _2{\mathbf {L}}_G]. \end{aligned}$$

  (18)

Note that, here we provided several possible alternatives for the weights of decision fusion. The performances of the various proposed fusion strategies will be presented in Sect. 3.

3 Results and discussions

We now test the performance of the proposed unsupervised segmentation and supervised HSI classification approaches in several well-known datasets to demonstrate the efficacy of M-GSP in HSI analysis. We also comparatively test the performance of various different fusion decisions.

Table 1 Statistics of different HSI datasets

3.1 Dataset

We test the performances of the proposed methods based on four public HSI datasets accessible from website¹. The first HSI is Indian Pines (IndianP) scene originally gathered by AVIRIS sensors over an agricultural field. The second HSI dataset is the University of Pavia (PaviaU) acquired by ROSIS sensor. Note that some of the samples in PaviaU contain no information and have to be discarded before analysis. Two other HSIs used in the experiments are the Salinas Scene (Salinas) and Salinas-A Scene (SalinasA) datasets, which were collected by the 224-band AVIRIS sensor over Salinas Valley, California, and exhibit high spatial resolution. For each dataset, we have groundtruth classes for part of samples.

For these HSIs, Table 1 provides vital statistics and we provide visual illustration of the geometric plots in Figs. 7 and 8. Note that, Fig. 8 treats the unlabeled groundtruth samples as backgrounds with the same class label. Interested readers can find more information on the HSI datasets at the website$^1$.

Fig. 7

Mean of HSI over spectral dimension

Fig. 8

Ground truth of class labels

3.2 Unsupervised HSI segmentation

In this part, we first test the performance of unsupervised HSI segmentation. Although our proposed MLG-based method is amenable to various sophisticated clustering approaches, we find it easier to demonstrate it by using a basic spectral clustering scheme. Here, we mainly use comparison of basic methods, such as k-means clustering and GSP-based spectral clustering, to demonstrate the power of M-GSP in processing the HSI datasets.

To validate the performance of different methods, we consider two experimental setups. In the first scenario, we carry out unsupervised segmentation on all data samples and evaluate the overall visualization results of labeled samples. In the second stage, we process all data samples but focus on detecting boundaries (edges) of each cluster, in terms of both visualization results and numerical accuracy.

For fair comparison, we segment the HSI into N superpixels first before applying respective clustering algorithms thereupon. For the GSP-based method, we construct the graph ${\mathbf {W}}\in {\mathbb {R}}^{N\times N}$ using Gaussian distance

$$W_{{ij}} = \left\{ {\begin{array}{*{20}l} {e^{{ - \frac{{||{\mathbf{s}}_{i} - {\mathbf{s}}_{j} ||_{2}^{2} }}{{\sigma ^{2} }}}} ,} \hfill & {\left\| {{\mathbf{s}}_{i} - {\mathbf{s}}_{j} } \right\|_{2}^{2} \le \tau } \hfill \\ {0,} \hfill & {{\text{otherwise}}} \hfill \\ \end{array} } \right.$$

(19)

where ${\mathbf {s}}_i$ denotes the feature of ith superpixel. The threshold $\tau$ is set to the statistical mean of all pairwise distances among superpixels, and $\sigma$ is tunable according to specific datasets. For M-GSP-based methods, we construct the multilayer graph with $M=10$ layers, and calculate the distance based on Eqs. (3) and (4). The parameter p in Eq. (4) is also set to the statistical mean of all pairwise intralayer feature distances, and $q=100$ is used. We select the number of key spectra based on the largest gap of singular values. We summarize the results below.

3.2.1 Visualization of HSI segmentation

By setting $N=500$ for all tested HSIs, Figs. 9, 10, 11, 12 present the visualization results of the three HSI segmentations using different algorithm in comparison to the groundtruth data. These results show that, in general, the MLG-based spectral clustering (M-GSP) displays more stable segmentations than the single layer GSP-based spectral clustering (GSP) as well as the k-means algorithm. Since it is harder to evaluate the details for too many classes, we can focus more on the exemplary Salinas A dataset. In particular, Fig. 8d shows six different groundtruth classes. However, both k-means and GSP-based method failed to detect the class No.6, marked by orange in Fig. 9a and marked by yellow in Fig. 9b. Meanwhile, in Fig. 9c, the M-GSP method successfully identified all six classes and delivered results that are closer to groundtruth. Recall that, from Fig. 3, the MLG-based singular values are more concentrated, which provides the benefits of robustness in spectral clustering. Results from “Indian Pines” and “Pavia University” also show similarly stronger performance for M-GSP. These results collectively demonstrate improved efficiency of M-GSP in unsupervised HSI segmentation.

Fig. 9

Segmented results of Salinas A

Fig. 10

Segmented results of Indian Pines

Fig. 11

Segmented results of Pavia University

Fig. 12

Segmented results of Salinas

3.2.2 Boundary segmentation

Although unsupervised methods may generate meaningful segmentation different from groundtruth, we are still interested in how far the segmented results are from the true labels. Since it is inefficient to match all the clusters to the corresponding true labels, we focus on the boundaries of the segmentation. In the boundary detection, we set $N=100$ and define the accuracy as

$$\begin{aligned} {\text {Acc}}=\sum _{i=1}^T 1(L_i={\hat{L}}_i)/T, \end{aligned}$$

(20)

where T is the number of pixels in the HSI, $L_i$ is the true labels of edges for the ith pixel, ${\hat{L}}_i$ is the estimated labels, and $1(\cdot )$ denotes the indicator function. The boundary results are visualized in Figs. 13, 14, 15. Since we consider the unlabeled samples in the clustering process, there can be more details for the HSIs than their corresponding groundtruth, especially for Pavia University and Indian Pines. However, we can still derive some benefits of M-GSP in boundary detection. As Fig. 15 shows, M-GSP can generate clearer edges, obviously on the top half image, while GSP and k-means appear to over-segment. We also present the accuracy defined in Eq. (20) as Table 2. These results show that M-GSP performs better than k-means and GSP. They demonstrate the efficiency of MLG models in HSI analysis. Note that we do not claim M-GSP to be the best approaches for all scenarios. Without impractically requiring hindsight to fine-tune various parameters for each HSI dataset to generate “the best results,” M-GSP delivers consistently strong and stable segmentation results for various HSI datasets by relying on some basic guidelines for selecting parameters.

Fig. 13

Boundaries in Indian Pines

Fig. 14

Boundaries in Pavia University

Fig. 15

Boundaries in Salinas

Table 2 Accuracy of segmentation boundaries

3.3 Supervised HSI classification

We next test M-GSP in supervised HSI classification.

3.3.1 Overall accuracy

Applying M-GSP-based spectral clustering as feature extraction of HSI, we compare the proposed algorithms with several well-known feature extraction algorithms, including NPE [51], LP-NPE [52], LDA [53], LFDA [54], SPCA [39] and MSPCA [39]. For the proposed MLG-SRC and MLG-MRC, we regroup the superpixels to $70\%$ of the original superpixel number, before extracting features based on M-GSP. Here, we show the results of MLG-MRC according to different decision values (fusion weights). More analysis of different fusion strategies will be illustrated further in Sect. 3.3.2.

The overall accuracy under different numbers of training samples per class (TS/C) is shown in Table 3. In this experiment, parameters of multiple resolutions are tuned for different HSIs. From the test results, the proposed MLG-MRC exhibits a superior overall performance, especially for those scenarios with fewer classes.

Table 3 Overall accuracy of different HSI segmentation

3.3.2 Analysis of different fusion strategies

Now, we analyze the performance of different fusion strategies. Since MSPCA [39] also fuses multiple resolutions of superpixels, we investigate our proposed fusion methods in combination with both MSPCA and MLG-MRC.

We fuse the results from 9 resolutions, i.e., $N_i\in [25,35,50,70,100,140,200,280,400]$, for all comparative methods. The test accuracy is shown in Table 4. As shown, our proposed weight approaches lead to significant improvement over the basic majority voting (VT) for MSPCA. Compared to SVM-based weights, the graph-based weights are better for MSPCA since additional geometric information is considered. For MLG-MRC, the proposed weights show a slight improvement, which suggests that MLG-MRC is less sensitive to different decision strengths. To better understand the effect of different weights, we also present decision strengths over different numbers of superpixels in Fig. 16. In Fig. 16a, b, the graph-based weights favor low resolution to form baselines and use high resolutions to interpolate details. Thus, for HSIs with larger area of segmented groups, such as Salinas, the graph-based weights generate superior performances. Using SVM-based weights, we find no consistent trend among different superpixels, and the results vary for different datasets. Since the MLG-based methods have already incorporated the underlying geometric structures, they continue to display robust results even when using SVM-based weights.

Table 4 Performance of different fusion strategies

Fig. 16

Decision strengths over different numbers of superpixels

3.3.3 Robustness in noisy datasets

We further evaluate the robustness of proposed methods in noisy environment. More specifically, we consider two types of noise models: (1) pixel-dependent noise where pixel noise variance depending on corresponding pixel data value; and (2) non-pixel dependent noise where noise variance is defined by mean of all pixel values. These two different noise models describe two different practical sensing noises. We also test both uniform noise and Gaussian noise. From the test performances shown in Table 5, we find the newly proposed MLG-based methods to be less sensitive to various types of sample noises.

Table 5 Overall accuracy in noisy environment

3.3.4 Complexity

In practice, the M-GSP analysis usually consists of two parts: 1) calculation of the MLG spectra; and 2) implementation of algorithms with the calculated spectra for specific tasks. In this section, we evaluate the complexity of M-GSP analysis compared with traditional graph-based methods and signal processing approaches.

Table 6 Running time of graph construction + spectra calculation (Seconds)

• Graph Construction and Spectrum Calculation: We compare the complexity of spectra calculation for M-GSP with several typical graph models for multilayer feature extraction. Suppose we have an HSI with M clustered bands and N superpixels. The first step is to calculate the graph spectra for interlayer connections (cross-band) and intralayer connections (spatial), respectively. For the single-layer graph model, one method (2-Way GSP [15]) is to consider all spectrum bands with the same graph structure, and each superpixel has the same cross-band interactions. Note that the bands are treated as nodes for cross-band interactions in this model. Now, the HSI can be represented by one $M\times M$ matrix for cross-band interactions and one $N\times N$ matrix for spatial connections. Eigen-decomposition is then implemented for these two matrices, respectively. However, such model assumes the same graph structure for all bands which is unable to represent different spatial connections for different bands. To compensate, one can also construct individual graphs for each band (Individual Graphs). If one also considers the same cross-band connections as 2-way GSP, the HSI can be represented by M $N\times N$ matrices for spatial connections and one $M\times M$ matrix for cross-band interactions. Moreover, if different cross-band connections are considered for different superpixels (Complex Individual Graphs), the HSI could be represented by M $N\times N$ matrices for spatial connections and N $M\times M$ matrices for cross-band connections. Here, we only provide comparison with several typical models, leaving further comparison to future works. In M-GSP, we allow different intra-layer connections for each layer, and flexible interlayer interactions. No matter how the interactions are defined, HSI is always represented by one $M\times N\times M \times N$ tensor. The structures of interactions are represented by the entry values, and the identification of layers/nodes is reflected by the index of each entry in the tensor. We then use HOSVD to obtain all the spectra. To compare the complexity, we test different models in the Pavia University dataset and set $M=10$. We test the model by using Matlab programs on a desktop (Intel Xeon CPU @ 3.50GHz, 32 GB RAM), where eigen-decomposition is implemented for single-layer graphs, and HOSVD in the tensor toolbox² is used for M-GSP. The runtime of spectra calculation with graph construction under different N is shown as Table 6. Since 2-way GSP neglects the different structures for different layers and superpixels, it requires the least complexity but provides less information. It is unable to represent the heterogeneous graph structures for different bands. If different interlayer and intralayer connections are considered, M-GSP has similar or slightly better complexity than graph-based models. Moreover, we only need to use one tensor to represent the HSI, which indicates that M-GSP provides a more general representation for different HSIs and MLGs.

• Implementation of M-GSP Algorithms: Since our proposed M-GSP processing can be flexibly integrated with other dimension reduction methods to reduce complexity, we find it unnecessary to provide evaluation of computation complexity for various setups. In general, the original MLG-MRC has a similar runtime as MSPCA. For example, under the same settings of multi-resolutions as Sect. 3.3.1 in Indian Pines HSI with tuning parameters of SVM among 15 sets, the runtimes for MLG-MRC and MSPCA are 28.21 seconds and 25.37 seconds, respectively. These and other tests indicate similar computation complexity for methods based on M-GSP and PCA.

3.3.5 Impact of regrouping ratio

To provide a better interpretation of M-GSP regrouping, we measure the performance under different regrouping ratio. We compare with SPCA and MPCA as an example, where ${\text {TS}}/C=5$. The majority voting is applied for multi-resolution methods. We do not regroup the features of SPCA and MPCA via M-GSP. The results are shown as Table. 7. From the results, we proposed methods have better performance when the ratio is above 30%. When the ratio is too small, some resolutions may have few clusters which reduce the accuracy of classification. In general, the ratio can be set around 60% to achieve a robust performance.

Table 7 Performance under different regrouping ratio

4 Conclusions

This work introduces M-GSP in hyperspectral imaging processing. To capture heterogeneous underlying structures within different but highly correlated spectrum frames in hyperspectral images (HSIs), we propose to represent HSIs via multilayer graphs. Analyzing singular spectra of adjacency tensor for the multilayer graph, we first develop a MLG-based spectral clustering for unsupervised HSI segmentation. Leveraging features extracted with M-GSP, we propose two algorithms for supervised HSI classification. We also consider several novel decision fusion strategies for multiple resolution superpixel analysis. Our experimental results demonstrate the robustness and efficiency of the proposed methods, successfully showcasing the power of M-GSP in HSI analysis.

Future direction Advances in tensor algebra and multilayer graph theory present more opportunities to explore M-GSP and its applications in hyperspectral imaging. One interesting topic is the construction of multilayer graphs efficiently for HSIs. In this work, we provide an easy-to-implement multilayer graph construction based on Gaussian distance, to explore the power of M-GSP. More efficient multilayer graph construction could lead to further performance gains. Potential approaches include those based on feature similarities and graph spectral properties [55]. Another promising direction is the use of M-GSP transformation and spectral filters in HSI analysis. More analytical tools, such as multilayer graph filter banks and sampling theory, can be introduced for HSI processing, including the extraction of HSI’s underlying spatial-frequency relationship. Other interesting future directions include the development of MLG convolutional networks and the integration of M-GSP with deep neural networks.