Prefiltering-based reduced rank multiuser detection in uplink generalized spatial modulation massive MIMO system

Abstract

Exploiting generalized spatial modulation in massive MIMO system, we develop an efficient uplink multiuser detection scheme. Different from the maximum likelihood detection algorithm, the spatial and temporal data in the proposed detection scheme are decoded in a sequential manner: At the front end of base station receiver, a bank of spatial filters is first employed to mitigate the multiuser interference using the benefits of large-scale antenna array. There then, a reduced rank generalized Eigen-value decomposition-based algorithm is proposed to identify the antennas activated at each user terminal. In the second stage, a least-squares-based multiuser detector is employed to decode the multiplexed temporal symbols. Both theoretical analysis and computer simulations are conducted to evaluate system performance in terms of correct identification probability and overall symbol error rate. Simulation results demonstrate that the proposed algorithm outperforms most of the existing algorithms yet the complexity is extensively reduced.

1 Introduction

In recently years, spatial modulation (SM) technique has attracted a lot of attention since it exploits antenna indices in an antenna array system as an additional source of information compared to traditional M-ary communication systems [1,2,3,4,5]. In SM scheme, the data to be transmitted are composed of two information-carrying symbols: spatial and temporal symbols. The spatial symbol determines the antenna(s) at the transmitter antenna array for data transmission. The temporal symbol is mapped to the conventional signal constellation diagram. Hence, the main advantage of SM scheme is the increase in transmitted bits per channel use (bpcu) compared to the conventional modulation scheme. It is plausible to apply SM in MIMO system [5,6,7] since only single RF chain is activated and the complexity is extensively reduced. Generalized spatial modulation (GSM) [8] is a variant of SM, in which more than one antennas are activated at each time slot. Hence, compared to space shift keying (SSK) or SM, the system spectral efficiency is further increased.

It has been shown that multiuser MIMO (MU-MIMO) system has the benefits of increasing the capacity of wireless communication networks [9, 10]. Recently, SM technique has been applied in downlink (DL) MU-MIMO transmission [7, 11,12,13,14,15,16,17]. Alternatively, the realization of SM technique in uplink (UL) MU-MIMO has been proposed in [18,19,20,21,22,23,24]. In the work of [18, 19], Maximum likelihood (ML) detector is employed to decode multiuser data. As the number of users increases, the computation load of ML detector becomes extensive. A suboptimum algorithm named message passing detector is proposed in [20]. Whereas, the complexity grows high in higher-order modulation scheme and/or large-scale antenna systems. A nearest neighbor search (NNS) algorithm is proposed in [21], in which an ordered candidate list of all users’ active antenna indices are first acquired. Nevertheless, as the size of candidate list is huge, the NNS algorithm number becomes computationally prohibitive. A space-alternating and generalized expectation maximization (SAGE) algorithm is proposed in [22]. It recursively performs successively multiuser interference (MUI) cancelation and multiuser data detection. A SAGE aided list-projection (S-LP) algorithm is proposed in [23]. However, as the number of users or array size is large, the computation load of the iterative algorithm is still high. The work of [24] developed a two-stage multiuser detection (MUD) structure to detect the GSM signal in massive MIMO system [25,26,27]. Nevertheless, the eigen-value decomposition (EVD)-based active antennas identification algorithm encounters huge computation load in large-scale antenna system.

One of the most attractive properties of massive MIMO is the asymptotic of “Random matrix theory” [27]. The channel response vectors associated with different UTs tend to be nearly orthogonal when the number of BS antennas is large. In this work, we exploit this property to design a low-complexity MUD to detect GSM signal in UL massive MIMO system. We assume each UT is equipped with N antennas and the array size at BS is L, in which $L \gg N$. At the transmitter side, spatial symbol activates a set of UT antennas for data transmission. At the BS receiver front end, a bank of spatial filters is first exploited to mitigate the MUI based on a priori information of the angle of arrivals (AOAs) of each UT. Moreover, by applying spatial prefiltering, the size of observation vector has been extensively reduced from L to N. The output of each spatial filter is then processed by a two-stage detection scheme: To convert the active antennas’ identification into spatial spectrum peak searching problem, we propose a reduced rank generalized eigen-value decomposition (GEVD) [28] based algorithm. The indices of antennas being activated at each UT are identified at the first stage, and the indices are then demapped to spatial symbols. In the second stage, the least-squares (LS) MUD is utilized to decode the multiplexed temporal symbols. We have compared with existing algorithms toward SM (or GSM) UL MU-MIMO both in the computation load as well as the overall symbol error rate (SER), the advantages of our work include:

• 1. The algorithms developed in [18,19,20,21,22,23] need sophisticated iterations and/or exhausted searches, while the proposed scheme only requires to perform GEVD and linear processing of the data vector with size N.

• 2. Compared with the work of [24], owing to spatial prefiltering, the dimension of the observation vector is extensively reduced from L to N. This comprehensively reduces the complexity and convergence time.

• 3. The overall SER performance is comparable or superior to the works of [18,19,20,21,22,23,24]. Moreover, the proposed algorithm is extremely flexible in that it can be easily extended from SM to GSM without increasing too much complexity.

Furthermore, analytical result toward active antenna correct detection probability is derived and is closely matched to the simulation result.

The remainder of this paper is organized as follows. In Sect. 2, we describe the system and signal model. Section 3 describes and analyzes the proposed prefiltering-based two-stage MUD scheme both in SM and GSM UL massive MIMO system. In Sect. 4, we evaluate the system performance under different scenarios and discuss the numerical results. Concluding remarks are finally made in Sect. 5.

Notation: We represent scalar constants by non-boldface letters (e.g., a or A), vectors by boldface small letters (e.g., a), and matrices by boldface capital letters (e.g., A). ${\mathbf{A}}\left( {i,j} \right)$ denotes the element of the ith row and jth column of matrix A. ${\mathbf{A}}\left( {:,j} \right)$ denotes the jth column vector of matrix A. We use superscripts $\left[ {} \right]^{T}$ and $\left[ {} \right]^{H}$ for matrix (or vector) transpose and Hermitian transpose, respectively. $\left\| {\mathbf{a}} \right\|$ denotes the l₂-norm of vector a. $E\left\{ {} \right\}$ denotes ensemble average of random variable or random process.” A complex normal random variable with mean $\mu$ variance $\sigma^{2}$ reads as $CN\left( {\mu ,\sigma^{2} } \right)$. $\delta ( \cdot )$ is the Dirac delta function. $\left( {\begin{array}{*{20}c} M \\ L \\ \end{array} } \right) = \frac{M!}{{L!\left( {M - L} \right)!}};M \ge L$ stands for the combination of L out of M. “$\left\lfloor x \right\rfloor$” is the floor function that rounds x down and returns the largest number that is less than or equal to x. Re{x} denotes the real part of the complex variable x.

2 Signal and system model

2.1 System model

In this paper, we consider the UL MU-MIMO communication, in which K remote distributed UTs transmit simultaneously to a single BS within a cell. There are N and L antennas at each UT and the BS, respectively. GSM is employed in which D out of N antennas are activated simultaneously for data transmission. The schematic diagram of the UL GSM MU-MIMO system is depicted in Fig. 1. The transmitted symbol is composed of two information carrying blocks, which is referred to as spatial and temporal data, respectively. The spatial data (n bits) of the kth UT, denoted as $\left[ {\begin{array}{*{20}c} {b_{1}^{k} } & {b_{2}^{k} } & \cdots & {b_{n}^{k} } \\ \end{array} } \right]$, are used to activate D out of N antennas for UL transmission and the temporal data (m bits), $\left[ {\begin{array}{*{20}c} {b_{n + 1}^{k} } & {b_{n + 2}^{k} } & \cdots & {b_{n + m}^{k} } \\ \end{array} } \right]$, are mapped to a corresponding symbol chosen from the conventional M-ary signal-constellation diagram. Therefore, $n = \left\lfloor {\log_{2} \left( {\begin{array}{*{20}c} N \\ D \\ \end{array} } \right)} \right\rfloor$ and $m = \log_{2} M$. Specifically, SM is a special case (D = 1) of GSM. It is evident that compared with the SM scheme, GSM increases bpcu from $K\left( {\log_{2} N + \log_{2} M} \right)$ to $K\left( {\left\lfloor {\log_{2} \left( {\begin{array}{*{20}c} N \\ D \\ \end{array} } \right)} \right\rfloor + D\log_{2} M} \right)$.

Fig. 1

Block diagram of the UL GSM MU-MIMO system. Solid lines at each UT represent the activated transmit antennas, while the remaining dashed lines denote the inactive antennas to every receiving antennas at BS

We denote the channel coefficient between the kth UT’s nth antenna and the BS’s lth antenna as $\left\{ {h_{l,n}^{k} } \right\}_{\begin{subarray}{l} l = 1, \ldots ,L \\ k = 1, \ldots ,K \\ n = 1, \ldots ,N \end{subarray} }$. Hence, the MIMO channel matrix between BS and kth UT can be expressed as

$${\mathbf{H}}_{{}}^{k} = \left[ {\begin{array}{*{20}c} {h_{1,1}^{k} } & {h_{1,2}^{k} } & \cdots & {h_{1,N}^{k} } \\ {h_{2,1}^{k} } & {h_{2,2}^{k} } & \cdots & {h_{2,N}^{k} } \\ \vdots & \vdots & \ddots & \vdots \\ {h_{L,1}^{k} } & {h_{L,2}^{k} } & \cdots & {h_{L,N}^{k} } \\ \end{array} } \right]_{L \times N} = \left[ {\begin{array}{*{20}c} {{\mathbf{h}}_{1}^{k} } & {{\mathbf{h}}_{2}^{k} } & \cdots & {{\mathbf{h}}_{N}^{k} } \\ \end{array} } \right];\quad k = {1}, \ldots ,K$$

(1)

where $\left\{ {{\mathbf{h}}_{n}^{k} } \right\}_{\begin{subarray}{l} k = 1, \ldots ,K \\ n = 1, \ldots ,N \end{subarray} }$ represents the channel vector with size L-by-1.

$${\mathbf{h}}_{n}^{k} \equiv \left[ {\begin{array}{*{20}c} {h_{1,n}^{k} } & {h_{2,n}^{k} } & \cdots & {h_{L,n}^{k} } \\ \end{array} } \right]^{T} ;\quad n = {1}, \ldots ,N$$

(2)

$\left\{ {{\mathbf{h}}_{n}^{k} } \right\}_{\begin{subarray}{l} k = 1, \ldots ,K \\ n = 1, \ldots ,N \end{subarray} }$ can be modeled as

$${\mathbf{h}}_{n}^{k} = \sqrt {\gamma_{{}}^{k} } {\mathbf{a}}_{n}^{k}$$

(3)

where $\left\{ {{\mathbf{a}}_{n}^{k} } \right\}_{\begin{subarray}{l} k = 1, \ldots ,K \\ n = 1, \ldots ,N \end{subarray} }$ is the steering vector of the BS array of antennas with respect to the signal coming from the kth UT’s nth antenna. Without loss of generality, we assume that the BS is equipped with uniform linear array (ULA) [29], hence, $\left\{ {{\mathbf{a}}_{n}^{k} } \right\}_{\begin{subarray}{l} k = 1, \ldots ,K \\ n = 1, \ldots ,N \end{subarray} }$ can be modeled as Vandermonde form

$${\mathbf{a}}_{n}^{k} = \frac{1}{\sqrt L }\left[ {\begin{array}{*{20}c} 1 \\ {e^{{ - j\frac{2\pi \eta }{\lambda }\cos \left( {\theta_{n}^{k} } \right)}} } \\ \vdots \\ {e^{{ - j\frac{{2\pi \left( {L - 1} \right)\eta }}{\lambda }\cos \left( {\theta_{n}^{k} } \right)}} } \\ \end{array} } \right]$$

(4)

$\sqrt {\gamma_{{}}^{k} }$ represents large-scale fading for the kth UT. $\eta$ is the antenna spacing at the BS and λ is the signal wavelength. $\theta_{n}^{k} \in \left[ {0,\pi } \right]$ is the angle-of-arrival (AOA) of the kth UT’s nth antenna. In the considered model, we assume that the BS has knowledge of AOAs, while the shadowing parameters, $\left\{ {\gamma_{{}}^{k} } \right\}_{k = 1, \ldots ,K}$, are unknown. According to the signal model of (4), the cross-correlation between ${\mathbf{a}}_{n}^{k} ,{\mathbf{a}}_{{n^{\prime}}}^{{k^{\prime}}}$ can be obtained as

$$\left( {{\mathbf{a}}_{n}^{k} } \right)^{H} {\mathbf{a}}_{{n^{\prime}}}^{{k^{\prime}}} = \frac{1}{L}\sum\limits_{l = 1}^{L} {e^{{ - j\frac{{2\pi \left( {l - 1} \right)\eta }}{\lambda }\left( {\cos \left( {\theta_{n}^{k} } \right) - \cos \left( {\theta_{{n^{\prime}}}^{{k^{\prime}}} } \right)} \right)}} }$$

(5)

It follows that the channel vectors between UTs and the BS become pairwise orthogonal [27] in massive MIMO system, that is, as $L \to \infty ,\theta_{n}^{k} \ne \theta_{{n^{\prime}}}^{{k^{\prime}}}$, then

$$\left( {{\mathbf{a}}_{n}^{k} } \right)^{H} {\mathbf{a}}_{{n^{\prime}}}^{{k^{\prime}}} = \delta \left( {k - k^{\prime},n - n^{\prime}} \right)$$

(6)

However, (6) can hardly be applied in practical scenario since the array size at BS is limited and the N steering vectors, $\left\{ {\theta_{n}^{k} } \right\}_{n = 1, \ldots ,N}$, with respect to the signal coming from the same UT may be highly correlated, i.e., they may be with nearly the same AOA. Hence, in the considered model, (6) is modified as

$$\left( {{\mathbf{a}}_{n}^{k} } \right)^{H} {\mathbf{a}}_{{n^{\prime}}}^{{k^{\prime}}} = \left\{ {\begin{array}{*{20}c} {0;k \ne k^{\prime}\begin{array}{*{20}c} {} & {} & {} \\ \end{array} } \\ {\alpha_{{n,n^{\prime}}}^{k} ;k = k^{\prime},n \ne n^{\prime}} \\ {1;k = k^{\prime},n = n^{\prime}} \\ \end{array} } \right.$$

(7)

where $\left| {\alpha_{{n,n^{\prime}}}^{k} } \right|{ < }1$. Based on (7), we define the cross-correlation matrix as

$$\begin{aligned} {{\varvec{\Gamma}}}_{k}^{{}} & \equiv {\mathbf{A}}_{k}^{H} {\mathbf{A}}_{k}^{{}} = \left[ {\begin{array}{*{20}c} {{\mathbf{a}}_{1}^{k} } & {{\mathbf{a}}_{2}^{k} } & \cdots & {{\mathbf{a}}_{N}^{k} } \\ \end{array} } \right]^{H} \left[ {\begin{array}{*{20}c} {{\mathbf{a}}_{1}^{k} } & {{\mathbf{a}}_{2}^{k} } & \cdots & {{\mathbf{a}}_{N}^{k} } \\ \end{array} } \right] \\ & = \left[ {\begin{array}{*{20}c} 1 & {\alpha_{1,2}^{k} } & \cdots & {\alpha_{1,N}^{k} } \\ {\alpha_{2,1}^{k} } & 1 & \cdots & {\alpha_{2,N}^{k} } \\ \vdots & \ddots & \ddots & \vdots \\ {\alpha_{N,1}^{k} } & \cdots & {\alpha_{N,N - 1}^{k} } & 1 \\ \end{array} } \right]_{N \times N} \\ \end{aligned}$$

(8)

where ${\mathbf{A}}_{k}^{{}} \equiv \left[ {\begin{array}{*{20}c} {{\mathbf{a}}_{1}^{k} } & {{\mathbf{a}}_{2}^{k} } & \cdots & {{\mathbf{a}}_{N}^{k} } \\ \end{array} } \right]$ is the L-by-N steering matrix of the kth UT.

2.2 Signal model

In the SM case (only single antenna is activated), the received signal vector at the BS in the ith symbol duration can be written as

$$\begin{aligned} {\mathbf{y}}\left( i \right) & = \sum\limits_{k = 1}^{K} {\sqrt {P^{k} } {\mathbf{h}}_{{j_{k} }}^{k} s_{k} \left( i \right)} + {\mathbf{v}}\left( i \right) \\ & { = }\sum\limits_{k = 1}^{K} {\sqrt {P^{k} \gamma_{{}}^{k} } {\mathbf{a}}_{{j_{k} }}^{k} s_{k} \left( i \right)} + {\mathbf{v}}\left( i \right);\quad j_{k} \in \left\{ {1,2, \ldots ,N} \right\} \\ \end{aligned}$$

(9)

where ${\mathbf{y}}\left( i \right),{\mathbf{v}}\left( i \right) \in {\mathbb{C}}^{L \times 1}$. ${\mathbf{v}}\left( i \right)$ is the AWGN noise vector received at the BS array, whose covariance matrix is $\sigma^{2} {\mathbf{I}}_{L}$, i.e., ${\mathbf{v}}\left( i \right) \sim N\left( {{\mathbf{0}},\sigma^{2} {\mathbf{I}}_{L} } \right)$. $P^{k}$ is the average transmit energy of the kth UT per symbol. $\left\{ {s_{k} \left( i \right)} \right\}_{\begin{subarray}{l} k = 1, \ldots ,K \\ i = 1, \ldots \end{subarray} }$ is the kth UT’s ith modulated temporal symbol chosen from the M-ary constellation set $\aleph$. Throughout this paper, we assume $E\left\{ {\left| {s_{k} \left( i \right)} \right|^{2} } \right\} = 1$. We denote $\left\{ {j_{k} } \right\}_{k = 1, \ldots ,K} ,j_{k} \in \left\{ {1, \ldots ,N} \right\}$ as the antenna index being activated for data transmission of the kth UT. Thereby, ${\mathbf{h}}_{{j_{k} }}^{k} = {\mathbf{H}}_{{}}^{k} \left( {:,j_{k} } \right)$ is the $j_{k}$th column vector of ${\mathbf{H}}_{{}}^{k}$, ${\mathbf{a}}_{{j_{k} }}^{k} = {\mathbf{A}}_{k}^{{}} \left( {:,j_{k} } \right)$. From (3), we have ${\mathbf{h}}_{{j_{k} }}^{k} = \sqrt {\gamma_{{}}^{k} } {\mathbf{a}}_{{j_{k} }}^{k}$.

In SM scheme, we aim to develop an efficient algorithm that can jointly estimate $\left\{ {j_{k} ,s_{k} \left( i \right)} \right\}_{k = 1, \ldots ,K}$ based on the observations, ${\mathbf{y}}\left( i \right)$.

3 Multiuser receiver design

Upon defining ${\mathbf{A}} \equiv \left[ {\begin{array}{*{20}c} {{\mathbf{a}}_{{j_{1} }}^{1} } & {{\mathbf{a}}_{{j_{2} }}^{2} } & \cdots & {{\mathbf{a}}_{{j_{K} }}^{K} } \\ \end{array} } \right],{\mathbf{A}} \in {\mathbb{C}}^{L \times K}$, we may convert the received vector of (9) into a concise form

$${\mathbf{y}}\left( i \right) = {\mathbf{AD}}^{\frac{1}{2}} {\mathbf{s}}\left( i \right) + {\mathbf{v}}\left( i \right)$$

(10)

where ${\mathbf{D}} \equiv {\text{diag}}\left\{ {\begin{array}{*{20}c} {P^{1} \gamma_{{}}^{1} } & {P^{2} \gamma_{{}}^{2} } & \cdots & {P^{K} \gamma_{{}}^{K} } \\ \end{array} } \right\}$. ${\mathbf{s}}\left( i \right) \equiv \left[ {\begin{array}{*{20}c} {s_{1} \left( i \right)} & {s_{2} \left( i \right)} & \cdots & {s_{K} \left( i \right)} \\ \end{array} } \right]^{T}$. Under AWGN, the maximum likelihood (ML) decision rule is optimum given that both the spatial and temporal data symbols occur with equal probability. Based on the observation vector of (10), the ML detection criterion is the minimum Euclidean distance between y(i) and the set of all possible received signals.

$$\begin{gathered} \left[ {\left\{ {\hat{j}_{k,ML} } \right\}_{k = 1, \ldots ,K} ,\left\{ {\hat{s}_{k,ML} \left( i \right)} \right\}_{k = 1, \ldots ,K} } \right] = \arg \mathop {\min }\limits_{\begin{subarray}{l} \left\{ {\hat{j}_{k} } \right\}_{k = 1, \ldots ,K} \in \left\{ {1, \ldots ,N} \right\} \\ \left\{ {\hat{s}_{k} \left( i \right)} \right\}_{k = 1, \ldots ,K} \in \aleph \end{subarray} } \left\| {{\mathbf{y}}\left( i \right) - {\mathbf{AD}}^{\frac{1}{2}} {\mathbf{s}}\left( i \right)} \right\|^{2} \\ = \arg \mathop {\max }\limits_{\begin{subarray}{l} \left\{ {\hat{j}_{k} } \right\}_{k = 1, \ldots ,K} \in \left\{ {1, \ldots ,N} \right\} \\ \left\{ {\hat{s}_{k} \left( i \right)} \right\}_{k = 1, \ldots ,K} \in \aleph \end{subarray} } \left\{ {2{\text{Re}} \left\{ {{\mathbf{y}}^{H} \left( i \right){\mathbf{AD}}^{\frac{1}{2}} {\mathbf{s}}\left( i \right)} \right\} - {\mathbf{s}}^{H} \left( i \right){\mathbf{D}}^{\frac{1}{2}} {\mathbf{A}}^{H} {\mathbf{AD}}^{\frac{1}{2}} {\mathbf{s}}\left( i \right)} \right\} \\ \end{gathered}$$

(11)

As depicted in (11), it is inevitable to perform exhaustive searches over all possible spatial and temporal data symbols. The computation load of the ML method is extensive since (11) needs to be evaluated by $\left( {NM} \right)^{K}$ times, which is impractical as NM is large.

3.1 Prefiltering-based multiuser SM receiver

To reduce the complexity of the ML algorithm, we develop a two-stage detection process that aims to perform multiuser spatial and temporal symbols’ detection in a sequential manner.

Stage 1: Identification of the active antenna of each UT

Algorithm description

At the front end of BS receiver, the observed data vector ${\mathbf{y}}\left( i \right)$ is first passed through a bank of K spatial filters, $\left\{ {{\mathbf{A}}_{k}^{{}} } \right\}_{k = 1, \ldots ,K}$, and the output of each spatial filter can be obtained as

$$\begin{aligned} {\mathbf{y}}_{k} \left( i \right) & = {\mathbf{A}}_{k}^{H} {\mathbf{y}}\left( i \right) = {\mathbf{A}}_{k}^{H} \left( {\sum\limits_{q = 1}^{K} {\sqrt {P^{q} \gamma_{{}}^{q} } {\mathbf{a}}_{{j_{q} }}^{q} s_{q} \left( i \right)} + {\mathbf{v}}\left( i \right)} \right) \\ & = \sqrt {P^{k} \gamma_{{}}^{k} } \left[ {\begin{array}{*{20}c} {\alpha_{{1,j_{k} }}^{k} } \\ \vdots \\ 1 \\ \vdots \\ {\alpha_{{N,j_{k} }}^{k} } \\ \end{array} } \right]s_{k} \left( i \right) + {\mathbf{v}}_{k} \left( i \right) \\ & = \sqrt {P^{k} \gamma_{{}}^{k} } {{\varvec{\upalpha}}}_{{j_{k} }}^{k} s_{k} \left( i \right) + {\mathbf{v}}_{k} \left( i \right);\quad k = {1}, \, \ldots ,K \\ \end{aligned}$$

(12)

where ${{\varvec{\upalpha}}}_{{j_{k} }}^{k} = {\mathbf{A}}_{k}^{H} {\mathbf{a}}_{{j_{k} }}^{k}$ is the $j_{k}$th column vector of ${{\varvec{\Gamma}}}_{k}^{{}}$. ${\mathbf{v}}_{k} \left( i \right) = {\mathbf{A}}_{k}^{H} {\mathbf{v}}\left( i \right)$ is jointly Gaussian with zero-mean and covariance (correlation) matrix given by

$$E\left\{ {{\mathbf{v}}_{k} \left( i \right){\mathbf{v}}_{k}^{H} \left( i \right)} \right\} = E\left\{ {{\mathbf{A}}_{k}^{H} {\mathbf{v}}\left( i \right){\mathbf{v}}^{H} \left( i \right){\mathbf{A}}_{k}^{{}} } \right\} = \sigma^{2} {\mathbf{A}}_{k}^{H} {\mathbf{A}}_{k}^{{}} = \sigma^{2} {{\varvec{\Gamma}}}_{k}^{{}}$$

(13)

Note that in deriving (12), we have exploited the result of (7). There are two benefits by applying the spatial prefiltering at the front-end of BS multiuser receiver:

• 1. Exploiting the most attractive property that the channel vectors between different users is inherently orthogonal when the number of BS antennas is sufficiently large, the spatial filter can mitigate MUI from other UTs.

• 2. Compared with the work of [24], the size of observation vector has been extensively reduced from L to N by applying spatial prefiltering.

Exploiting (12) and (13), the correlation matrix of the observed data vector can be obtained as

$$\begin{gathered} {\mathbf{R}}_{k} = E\left\{ {{\mathbf{y}}_{k} \left( i \right){\mathbf{y}}_{k}^{H} \left( i \right)} \right\} \\ = P^{k} \gamma_{{}}^{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} E\left\{ {\left| {s_{k} \left( i \right)} \right|^{2} } \right\}\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} + \sigma^{2} E\left\{ {{\mathbf{v}}_{k} \left( i \right){\mathbf{v}}_{k}^{H} \left( i \right)} \right\} \\ = P^{k} \gamma_{{}}^{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} + \sigma^{2} {{\varvec{\Gamma}}}_{k}^{{}} \\ \end{gathered}$$

(14)

Performing GEVD on the matrix pencil $\left\{ {{\mathbf{R}}_{k} ,{{\varvec{\Gamma}}}_{k}^{{}} } \right\}$ yields

$${\mathbf{R}}_{k} {\mathbf{e}}_{u}^{k} = \lambda_{u}^{k} {{\varvec{\Gamma}}}_{k}^{{}} {\mathbf{e}}_{u}^{k} ;u = 1,2, \ldots ,N$$

(15)

where $\lambda_{u}^{k} ;u = 1,2, \ldots ,N$ are the generalized eigenvalues (gevs) of the matrix pencil $\left\{ {{\mathbf{R}}_{k} ,{{\varvec{\Gamma}}}_{k}^{{}} } \right\}$ associated with generalized eigenvectors (GEVs), ${\mathbf{e}}_{u}^{k} ;u = 1,2, \ldots ,N$. Substituting (14) into (15), we have

$$P^{k} \gamma_{{}}^{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{e}}_{u}^{k} = \left( {\lambda_{u}^{k} - \sigma^{2} } \right){{\varvec{\Gamma}}}_{k}^{{}} {\mathbf{e}}_{u}^{k} ;\quad u = 1,2, \ldots ,N$$

(16)

Let the gevs be spread in the order of $\lambda_{1}^{k} \ge \lambda_{2}^{k} \ge \cdots \ge \lambda_{N}^{k}$, then the following facts can be drawn from matrix theory [28].

• 1. $\lambda_{u}^{k} = \sigma^{2}$ is smallest gev with multiplicity N-1, i.e., $\lambda_{2}^{k} = \lambda_{3}^{k} = \cdots = \lambda_{N}^{k} = \sigma^{2}$. According to (16), the corresponding GEVs, ${\mathbf{e}}_{u}^{k} ;u = 2,3, \ldots ,N$, satisfy

  $$P^{k} \gamma_{{}}^{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{e}}_{u}^{k} = 0;\quad u = 2, \ldots ,N$$

  (17)

• 2. ${\mathbf{e}}_{1}^{k}$ (the GEV corresponding to $\lambda_{1}^{k}$) is orthogonal to the vector space ${\mathbf{E}}_{N}^{k} \equiv span\left\{ {{\mathbf{e}}_{2}^{k} ,{\mathbf{e}}_{3}^{k} , \cdots ,{\mathbf{e}}_{N}^{k} } \right\}$, which is referred to as the noise subspace.

• 3. As revealed by (17), ${{\varvec{\upalpha}}}_{{j_{k} }}^{k}$ is orthogonal complement of ${\mathbf{E}}_{N}^{k}$. Define the N-by-(N-1) matrix ${\mathbf{E}}_{N}^{k} \equiv \left[ {{\mathbf{e}}_{2}^{k} ,{\mathbf{e}}_{3}^{k} , \cdots ,{\mathbf{e}}_{N}^{k} } \right]$, thus we have

  $$\left\| {\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{E}}_{N}^{k} } \right\|^{2} = \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{E}}_{N}^{k} \left( {{\mathbf{E}}_{N}^{k} } \right)^{H} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} = \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{P}}_{N}^{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} = 0$$

  (18)

  where ${\mathbf{P}}_{N}^{k} = {\mathbf{E}}_{N}^{k} \left( {{\mathbf{E}}_{N}^{k} } \right)^{H}$ is the projection matrix.

Exploiting the a priori information of $\left\{ {\theta_{n}^{k} } \right\}_{\begin{subarray}{l} k = 1, \ldots ,K \\ n = 1, \ldots ,N \end{subarray} }$, the index of the active antenna can be determined by the following algorithm.

$$\begin{aligned} \hat{j}_{k} & = \arg \mathop {\min }\limits_{{j_{k} \in \left\{ {1, \ldots ,N} \right\}}} \left\| {\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{E}}_{N}^{k} } \right\|^{2} \\ & { = }\arg \mathop {\max }\limits_{{j_{k} \in \left\{ {1, \ldots ,N} \right\}}} \left| {\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{e}}_{1}^{k} } \right|^{2} ;\quad k = {1}, \, \ldots ,K \\ \end{aligned}$$

(19)

However, ${\mathbf{R}}_{k}$ is unknown in practical situation. A typical block-fading wireless communication channel between the UTs and the BS is considered, where the channel coefficients stay constant during the observation interval, and change randomly from block to block. Hence, we may estimate ${\mathbf{R}}_{k}$ by performing time-average on the measurements, $\left\{ {{\mathbf{y}}_{k} \left( i \right)} \right\}_{i = 1,2, \ldots }$.

$${\hat{\mathbf{R}}}_{k} = \frac{1}{J}\sum\limits_{i = 1}^{J} {{\mathbf{y}}_{k} \left( i \right){\mathbf{y}}_{k} \left( i \right)^{H} }$$

(20)

The proposed activated antennas’ identification algorithm can be summarized as

• Step 1: Collect J data vectors $\left\{ {{\mathbf{y}}_{k} \left( i \right)} \right\}_{i = 1, \ldots ,J}$ and use (20) to compute ${\hat{\mathbf{R}}}_{k}$.

• Step 2: Perform GEVD on the matrix pencil $\left\{ {{\hat{\mathbf{R}}}_{k} ,{{\varvec{\Gamma}}}_{k}^{{}} } \right\}$ to obtain gevs (in descending order) $\hat{\lambda }_{1}^{k} \ge \hat{\lambda }_{2}^{k} \ge \cdots \ge \hat{\lambda }_{N}^{k}$, and the corresponding GEVs $\left\{ {{\hat{\mathbf{e}}}_{1}^{k} ,{\hat{\mathbf{e}}}_{2}^{k} , \cdots ,{\hat{\mathbf{e}}}_{N}^{k} } \right\}$.

• Step 3: Using (19) to estimate the index of activated antenna of each UT.

  $$\hat{j}_{k} = \arg \mathop {\max }\limits_{{j_{k} \in \left\{ {1, \ldots ,N} \right\}}} \left| {\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\hat{\mathbf{e}}}_{1}^{k} } \right|^{2} ;\quad k = {1}, \, \ldots ,K$$

• Step 4: Decode $\hat{j}_{k}$ to obtain the spatial data bits, $\left[ {\begin{array}{*{20}c} {\hat{b}_{1}^{k} } & {\hat{b}_{2}^{k} } & \cdots & {\hat{b}_{n}^{k} } \\ \end{array} } \right]$.

The schematic diagram of the proposed prefiltering-based UL GSM MUD is depicted in Fig. 2.

Fig. 2

Schematic diagram of the prefiltering-based uplink GSM multiuser receiver

Performance analysis

It is devised in [30] that the asymptotic distribution of ${\hat{\mathbf{e}}}_{1}^{k}$ is jointly Gaussian with mean equal to the ideal eigenvector, and covariance matrix given by

$$\begin{gathered} {{\varvec{\Sigma}}}_{1}^{k} = E\left\{ {\left( {{\hat{\mathbf{e}}}_{1}^{k} - {\mathbf{e}}_{1}^{k} } \right)\left( {{\hat{\mathbf{e}}}_{1}^{k} - {\mathbf{e}}_{1}^{k} } \right)^{H} } \right\} = \frac{{\lambda_{1}^{k} }}{J}\sum\limits_{u = 2}^{N} {\frac{{\sigma^{2} }}{{\left( {\sigma^{2} - \lambda_{1}^{k} } \right)^{2} }}{\mathbf{e}}_{u}^{k} \left( {{\mathbf{e}}_{u}^{k} } \right)^{H} } \\ { = }\frac{{\lambda_{1}^{k} }}{J}\frac{{\sigma^{2} }}{{\left( {\sigma^{2} - \lambda_{1}^{k} } \right)^{2} }}{\mathbf{E}}_{N}^{k} \left( {{\mathbf{E}}_{N}^{k} } \right)^{H} \\ = \frac{{\sigma^{2} \lambda_{1}^{k} }}{{J\left( {\sigma^{2} - \lambda_{1}^{k} } \right)^{2} }}{\mathbf{P}}_{N}^{k} \\ \end{gathered}$$

(21)

Define the random variables $X_{u} \equiv \left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {\hat{\mathbf{e}}}_{1}^{k} ;u = 1, \ldots ,j_{k} , \ldots ,N$. Since $X_{u}$ is linear combination of Gaussian random variables, thus, it is still Gaussian with mean $m_{u}^{{}} = \left\{ {\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {\mathbf{e}}_{1}^{k} } \right\};u = 1, \ldots ,N$ and variance

$$\begin{aligned} \sigma_{u}^{2} & = E\left\{ {\left( {X_{u} - m_{u}^{{}} } \right)^{2} } \right\} = \left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} E\left\{ {\left( {{\hat{\mathbf{e}}}_{1}^{k} - {\mathbf{e}}_{1}^{k} } \right)\left( {{\hat{\mathbf{e}}}_{1}^{k} - {\mathbf{e}}_{1}^{k} } \right)^{H} } \right\}{{\varvec{\upalpha}}}_{u}^{k} = \left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {{\varvec{\Sigma}}}_{1}^{k} {{\varvec{\upalpha}}}_{u}^{k} \\ & = \frac{{\sigma^{2} \lambda_{1}^{k} }}{{J\left( {\sigma^{2} - \lambda_{1}^{k} } \right)^{2} }}\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {\mathbf{P}}_{N}^{k} {{\varvec{\upalpha}}}_{u}^{k} ;\quad u = {1}, \, \ldots ,N \\ \end{aligned}$$

(22)

Let $Y_{u} = X_{u}^{2}$, we can deduce that the probability density function (pdf) of $Y_{u}$ as

$$f_{{Y_{u} }} \left( {y_{u} } \right) = \frac{1}{{2\sqrt {2\pi \sigma_{u}^{2} y_{u} } }}\left[ {\exp \left( { - \frac{{\left( {\sqrt {y_{u} } - m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right) + \exp \left( { - \frac{{\left( {\sqrt {y_{u} } + m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right)} \right];\quad y_{u} > 0,u = 1, \ldots ,N$$

(23)

Given that $j_{k}$th antenna is activated, the correct detection probability can be calculated as

$$\begin{aligned} P_{{\text{ant correct}}}^{k} & { = }P\left( {\hat{j}_{k} = j_{k} } \right) = P\left( {Y_{{j_{k} }} > \max \left\{ {Y_{1} , \ldots Y_{{j_{k} - 1}} ,Y_{{j_{k} + 1}} , \ldots Y_{N} } \right\}} \right) \\ & { = }P\left( {Y_{{j_{k} }} > Y_{1} ,Y_{{j_{k} }} > Y_{2} , \ldots ,Y_{{j_{k} }} > Y_{N} } \right) \\ & = \int_{0}^{\infty } {f_{{Y_{{j_{k} }} }} } \left( t \right)\prod\limits_{\begin{subarray}{l} u = 1 \\ u \ne j_{k} \end{subarray} }^{N} {P(Y_{u} < t)dt} \\ & \approx \int\limits_{0}^{\infty } {\left\{ {\int\limits_{0}^{t} {\frac{1}{{2\sqrt {2\pi \sigma_{u}^{2} y_{u} } }}\left[ {\exp \left( { - \frac{{\left( {\sqrt {y_{u} } - m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right) + \exp \left( { - \frac{{\left( {\sqrt {y_{u} } + m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right)} \right]dy_{u} } } \right\}^{N - 1} } \\ & \quad \frac{1}{{2\sqrt {2\pi \sigma_{{j_{k} }}^{2} t} }}\left[ {\exp \left( { - \frac{{\left( {\sqrt t - m_{{j_{k} }} } \right)^{2} }}{{2\sigma_{{j_{k} }}^{2} }}} \right) + \exp \left( { - \frac{{\left( {\sqrt t + m_{{j_{k} }} } \right)^{2} }}{{2\sigma_{{j_{k} }}^{2} }}} \right)} \right]dt \\ & = \left( {\frac{1}{{2\sqrt {2\pi } }}} \right)^{N} \int\limits_{0}^{\infty } {\left\{ {\int\limits_{0}^{t} {\frac{1}{{\sqrt {\sigma_{u}^{2} y_{u} } }}\left[ {\exp \left( { - \frac{{\left( {\sqrt {y_{u} } - m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right) + \exp \left( { - \frac{{\left( {\sqrt {y_{u} } + m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right)} \right]dy_{u} } } \right\}^{N - 1} } \\ & \quad \frac{1}{{\sqrt {\sigma_{{j_{k} }}^{2} t} }}\left[ {\exp \left( { - \frac{{\left( {\sqrt t - m_{{j_{k} }} } \right)^{2} }}{{2\sigma_{{j_{k} }}^{2} }}} \right) + \exp \left( { - \frac{{\left( {\sqrt t + m_{{j_{k} }} } \right)^{2} }}{{2\sigma_{{j_{k} }}^{2} }}} \right)} \right]dt \\ \end{aligned}$$

(24)

Stage 2: Detection of the temporal bit stream, $\left[ {\begin{array}{*{20}c} {b_{n + 1}^{k} } & {b_{n + 2}^{k} } & \cdots & {b_{n + m}^{k} } \\ \end{array} } \right]$

As depicted in Fig. 2, owing to the spatial prefiltering, MUI has been mitigated in stage 1. Thereby, the matched filter (or correlator) is optimum for single user detection that maximizes the output signal-to-noise power ratio (SNR). Given that the activated antenna has been correctly detected in stage 1, a simple matched filter can be employed by correlating the spatial prefilter output ${\mathbf{y}}_{k} \left( i \right)$ to ${{\varvec{\upalpha}}}_{{j_{k} }}^{k}$, which yields

$$\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{y}}_{k} \left( i \right) = \sqrt {P^{k} \gamma_{{}}^{k} } \left\| {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right\|^{2} s_{k} \left( i \right) + \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{v}}_{k} \left( i \right)$$

(25)

where the noise term $\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{v}}_{k} \left( i \right)$ is Gaussian with zero-mean and variance $E\left\{ {\left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {\mathbf{v}}_{k} \left( i \right){\mathbf{v}}_{k}^{H} \left( i \right){{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right\} = \sigma^{2} \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k}$. The corresponding SNR at the output of matched filter can be calculated as

$${\text{SNR}}_{k} = \frac{{E\left\{ {\left| {\sqrt {P^{k} \gamma_{{}}^{k} } \left\| {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right\|^{2} s_{k} \left( i \right)} \right|^{2} } \right\}}}{{\sigma^{2} \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} }} = \frac{{P^{k} \gamma_{{}}^{k} \left\| {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right\|^{4} }}{{\sigma^{2} \left( {{{\varvec{\upalpha}}}_{{j_{k} }}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{k} {{\varvec{\upalpha}}}_{{j_{k} }}^{k} }}$$

(26)

After matched-filtering, a standard optimum M-ary demodulator [31] can be employed to extract $s_{k} \left( i \right)$. In what follows, the estimated $s_{k} \left( i \right)$ is then de-mapped to the temporal data bits, $\left[ {\begin{array}{*{20}c} {\hat{b}_{n + 1}^{k} } & {\hat{b}_{n + 2}^{k} } & \cdots & {\hat{b}_{n + m}^{k} } \\ \end{array} } \right]$. For M-ary PSK, the averaged symbol error probability given that activated antenna has been correctly detected can be obtained as

$$P_{{{\text{symbol error}}\left| {\text{ant correct}} \right.}}^{k} \cong 2Q\left( {\sqrt {{\text{SNR}}_{k} } \sin \frac{\pi }{M}} \right)$$

(27)

where the approximation becomes extremely tight, for fixed M, as $SNR_{k}$ is increased.

It is shown in the work of [32] that the overall probability of symbol demodulation error is bounded by

$$P_{{\text{overall error}}}^{k} \le \left( {1 - P_{{\text{ant correct}}}^{k} } \right) + P_{{{\text{symbol error}}\left| {\text{ant correct}} \right.}}^{k} P_{{\text{ant correct}}}^{k}$$

(28)

Substituting (24) and (27) into (28), we have

$$\begin{aligned} P_{{\text{overall error}}}^{k} & \le 1 - \left( {1 - 2Q\left( {\sqrt {{\text{SNR}}_{k} } \sin \frac{\pi }{M}} \right)} \right) \\ & \times \left( {\frac{1}{{2\sqrt {2\pi } }}} \right)^{N} \int\limits_{0}^{\infty } {\left[ {\int\limits_{0}^{t} {\frac{1}{{\sqrt {\sigma_{u}^{2} y_{u} } }}\left[ {\exp \left( { - \frac{{\left( {\sqrt {y_{u} } - m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right) + \exp \left( { - \frac{{\left( {\sqrt {y_{u} } + m_{u} } \right)^{2} }}{{2\sigma_{u}^{2} }}} \right)} \right]dy_{u} } } \right]^{N - 1} } \\ & \frac{1}{{\sqrt {\sigma_{{j_{k} }}^{2} t} }}\left[ {\exp \left( { - \frac{{\left( {\sqrt t - m_{{j_{k} }} } \right)^{2} }}{{2\sigma_{{j_{k} }}^{2} }}} \right) + \exp \left( { - \frac{{\left( {\sqrt t + m_{{j_{k} }} } \right)^{2} }}{{2\sigma_{{j_{k} }}^{2} }}} \right)} \right]dt \\ \end{aligned}$$

(29)

Complexity analysis

In this subsection, we evaluate and compare the complexity of the proposed prefiltering-based and existing algorithms. Since both the NNS [21] and SAGE [22] algorithms are applied only in SM, hence, to treat the comparison fairly, we consider the scenario that only single antenna is activated. The number of complex multiplications required to implement the SAGE, NNS, ML, EVD-based [24] and the proposed algorithms are listed in Table 1, where the parameters of K, L, N, M, J denote the number of UTs, BS receiving antennas, UT transmitting antennas, temporal signal constellation size and window size, respectively. Note that the parameter α for the SAGE and NNS in Table 1 represents the number of cycles executed until the iterative algorithm converges. It has been demonstrated in [22] that the SAGE detector requires in average α = 3 under SNR = 20 dB. Note also that the computation load of $N^{3} ,N^{2}$ in the first row of Table 1 is resulted from the GEVD (Step 2) and identification of active antenna’s index (Step 3), respectively. It is worthy to note that thanks to the spatial prefiltering, the dimension of the observation vector is extensively reduced from L to N, which in term, comprehensively reduces the complexity as well as the convergence time.

Table 1 Computation load of the proposed, EVD-based, SAGE, NNS and ML detection algorithms

As depicted in Table 1, the computation load of the ML detection algorithm is extensive and thus impractical. Figure 3 presents the number of complex multiplications with respect to the number of active users for the proposed and existing suboptimum detection algorithms of UL MU-MIMO SM scheme. We set the parameters as α = 3, J = 12, M = 16, N = 8 and the cases for L = 40 and 80 are provided for comparison. As shown in Fig. 3, the computation load increases for larger K and/or L. It is also verified that the proposed algorithm is much simpler than EVD-based and NNS algorithms and is comparable to the SAGE algorithm. Nevertheless, as will be depicted in next section, the proposed algorithm is much more flexible than SAGE in that it can be easily applied to GSM scheme (multiple antennas are activated simultaneously) without increasing much computation load.

Fig. 3

Number of complex multiplications with respect to the number of active users for EVD-based, SAGE, NNS, and the proposed prefiltering-based algorithms

3.2 Prefiltering-based GSM multiuser receiver

In this subsection, we extend the proposed transmission and detection scheme to GSM, in which D (D > 1) out of N antennas are activated simultaneously at each UT. Moreover, multiplexing can be employed and each antenna carries independent temporal symbol.

For notational convenience, let the first D antennas of each UT are activated and each antenna is dedicated to send independent temporal symbol. Thereby, the received data vector at the BS can be written as

$${\mathbf{y}}\left( i \right) = \sum\limits_{k = 1}^{K} {\sqrt {P^{k} \gamma_{{}}^{k} } {\mathbf{A}}_{D}^{k} {\mathbf{s}}_{k} \left( i \right)} + {\mathbf{v}}\left( i \right)$$

(30)

where ${\mathbf{A}}_{D}^{k} \in {\mathbb{C}}^{L \times D} ,{\mathbf{s}}_{k} \left( i \right) \in {\mathbb{C}}^{D \times 1} ,{\mathbf{A}}_{D}^{k} \equiv \left[ {\begin{array}{*{20}c} {{\mathbf{a}}_{1}^{k} } & {{\mathbf{a}}_{2}^{k} } & {...} & {{\mathbf{a}}_{D}^{k} } \\ \end{array} } \right],{\mathbf{s}}_{k} \left( i \right) \equiv \left[ {\begin{array}{*{20}c} {s_{k,1} \left( i \right)} & {s_{k,2} \left( i \right)} & {...} & {s_{k,D} \left( i \right)} \\ \end{array} } \right]^{T}$. After prefiltering, the output of each spatial filter can be obtained as

$$\begin{aligned} {\mathbf{y}}_{k} \left( i \right) = {\mathbf{A}}_{k}^{H} {\mathbf{y}}\left( i \right) & = {\mathbf{A}}_{k}^{H} \left( {\sum\limits_{q = 1}^{K} {\sqrt {P^{q} \gamma_{{}}^{q} } {\mathbf{A}}_{D}^{q} {\mathbf{s}}_{q} \left( i \right)} + {\mathbf{v}}\left( i \right)} \right) \\ & = \sqrt {P^{k} \gamma_{{}}^{k} } {{\varvec{\Gamma}}}_{D}^{k} {\mathbf{s}}_{k} \left( i \right) + {\mathbf{v}}_{k} \left( i \right);\quad k = {1}, \, \ldots ,K \\ \end{aligned}$$

(31)

where ${{\varvec{\Gamma}}}_{D}^{k} \in {\mathbb{C}}^{N \times D} ,{{\varvec{\Gamma}}}_{D}^{k} \equiv \left[ {\begin{array}{*{20}c} {{{\varvec{\upalpha}}}_{1}^{k} } & {{{\varvec{\upalpha}}}_{2}^{k} } & {...} & {{{\varvec{\upalpha}}}_{D}^{k} } \\ \end{array} } \right]$. Thereby, the correlation matrix of ${\mathbf{y}}_{k} \left( i \right)$ is

$$\begin{aligned} {\mathbf{R}}_{k} & = E\left\{ {{\mathbf{y}}_{k} \left( i \right){\mathbf{y}}_{k}^{H} \left( i \right)} \right\} \\ & = P^{k} \gamma_{{}}^{k} {{\varvec{\Gamma}}}_{D}^{k} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} + \sigma^{2} {{\varvec{\Gamma}}}_{k}^{{}} \\ \end{aligned}$$

(32)

Performing GEVD on the matrix pencil $\left\{ {{\mathbf{R}}_{k} ,{{\varvec{\Gamma}}}_{k}^{{}} } \right\}$, after some manipulations, we have

$$P^{k} \gamma_{{}}^{k} {{\varvec{\Gamma}}}_{D}^{k} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {\mathbf{e}}_{u}^{k} = \left( {\lambda_{u}^{k} - \sigma^{2} } \right){{\varvec{\Gamma}}}_{k}^{{}} {\mathbf{e}}_{u}^{k} ;\quad u = 1,2, \ldots ,N$$

(33)

The following important facts can be drawn.

• 1. $\lambda_{u}^{k} = \sigma^{2}$ is smallest gev with multiplicity N-D, i.e., $\lambda_{D + 1}^{k} = \lambda_{D + 2}^{k} = \cdots = \lambda_{N}^{k} = \sigma^{2}$. And the corresponding GEVs, ${\mathbf{e}}_{u}^{k} ;u = D + 1,D + 2, \ldots ,N$, satisfy

  $$P^{k} \gamma_{{}}^{k} {{\varvec{\Gamma}}}_{D}^{k} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {\mathbf{e}}_{u}^{k} = 0;\quad u = D + 1, \ldots ,N$$

  (34)

• 2. From (33), we have that the column space of ${{\varvec{\Gamma}}}_{D}^{k}$,$span\left\{ {{{\varvec{\upalpha}}}_{1}^{k} ,{{\varvec{\upalpha}}}_{2}^{k} , \ldots ,{{\varvec{\upalpha}}}_{D}^{k} } \right\}$, is orthogonal complement of the vector space $span\left\{ {{\mathbf{e}}_{D + 1}^{k} ,{\mathbf{e}}_{D + 2}^{k} , \cdots ,{\mathbf{e}}_{N}^{k} } \right\}$, which is referred to as the noise subspace.

• 3. Define ${\mathbf{E}}_{N}^{k} \equiv \left[ {{\mathbf{e}}_{D + 1}^{k} ,{\mathbf{e}}_{D + 2}^{k} , \cdots ,{\mathbf{e}}_{N}^{k} } \right]$, then

  $$\left( {{{\varvec{\upalpha}}}_{j}^{k} } \right)^{H} {\mathbf{E}}_{N}^{k} = {\mathbf{0}};\quad \forall j = 1, \ldots ,D$$

  (35)

According to (35), the active antennas can be decided by projecting $\left\{ {{{\varvec{\upalpha}}}_{j}^{k} } \right\}_{j = 1, \ldots ,N}$ onto noise subspace.

$$\begin{aligned} f\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right) & \equiv \left\| {\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {\mathbf{E}}_{N}^{k} } \right\|^{2} = \left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {\mathbf{E}}_{N}^{k} \left( {{\mathbf{E}}_{N}^{k} } \right)^{H} {{\varvec{\upalpha}}}_{u}^{k} \\ & = \left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {\mathbf{P}}_{N}^{k} {{\varvec{\upalpha}}}_{u}^{k} ;\quad u = {1}, \, \ldots ,N \\ \end{aligned}$$

(36)

In ideal case, $f\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right) = 0$ provided that the uth antenna is active. On the other hand, if the uth antenna is inactive, then $f\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right){ > }0$.

In summary, the proposed active antennas’ identification algorithm for GSM scheme is as following:

• Step 1: Collect J data vectors $\left\{ {{\mathbf{y}}_{k} \left( i \right)} \right\}_{i = 1, \ldots ,J}$ and use (20) to compute ${\hat{\mathbf{R}}}_{k}$.

• Step 2: Perform GEVD on the matrix pencil $\left\{ {{\hat{\mathbf{R}}}_{k} ,{{\varvec{\Gamma}}}_{k}^{{}} } \right\}$ to obtain gevs (in descending order) $\hat{\lambda }_{1}^{k} \ge \hat{\lambda }_{2}^{k} \ge \cdots \ge \hat{\lambda }_{N}^{k}$, and the corresponding GEVs $\left\{ {{\hat{\mathbf{e}}}_{1}^{k} ,{\hat{\mathbf{e}}}_{2}^{k} , \cdots ,{\hat{\mathbf{e}}}_{N}^{k} } \right\}$.

• Step 3: Construct ${\hat{\mathbf{E}}}_{N}^{k} \equiv \left[ {{\hat{\mathbf{e}}}_{D + 1}^{k} ,{\hat{\mathbf{e}}}_{D + 2}^{k} , \cdots ,{\hat{\mathbf{e}}}_{N}^{k} } \right]$, and calculate ${\hat{\mathbf{P}}}_{N}^{k} = {\hat{\mathbf{E}}}_{N}^{k} \left( {{\hat{\mathbf{E}}}_{N}^{k} } \right)^{H}$.

• Step 4: Using (36) to calculate $\left\{ {\hat{f}\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)} \right\}_{u = 1, \ldots ,N}$

  $$\hat{f}\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right) = \left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)^{H} {\hat{\mathbf{P}}}_{N}^{k} {{\varvec{\upalpha}}}_{u}^{k} ;\quad u = {1}, \, \ldots ,N$$

• Step 5: Choosing among $\left\{ {\hat{f}\left( {{{\varvec{\upalpha}}}_{u}^{k} } \right)} \right\}_{u = 1, \ldots ,N}$ D smallest values with indices $\left\{ {\hat{u}_{d}^{k} } \right\}_{d = 1, \ldots ,D}$, respectively.

• Step 6: Decode $\left\{ {\hat{u}_{d}^{k} } \right\}_{d = 1, \ldots ,D}$ to the spatial data bits $\left[ {\begin{array}{*{20}c} {\hat{b}_{1}^{k} } & {\hat{b}_{2}^{k} } & \cdots & {\hat{b}_{n}^{k} } \\ \end{array} } \right]$.

In the second stage, we attempt to extract the multiplexed temporal symbol information, $\left\{ {\begin{array}{*{20}c} {s_{k,1} \left( i \right)} & {s_{k,2} \left( i \right)} & \cdots & {s_{k,D} \left( i \right)} \\ \end{array} } \right\}$, carried by the D active antennas. Revisit the linear model as depicted in (31)

$${\mathbf{y}}_{k} \left( i \right) = \sqrt {P^{k} \gamma_{{}}^{k} } {{\varvec{\Gamma}}}_{D}^{k} {\mathbf{s}}_{k} \left( i \right) + {\mathbf{v}}_{k} \left( i \right)$$

Since in the first stage, we have acquired ${{\varvec{\Gamma}}}_{D}^{k}$, hence, Least-squares (LS)-based method (or the decorrelating detector [33]) can be applied to extract ${\mathbf{s}}_{k} \left( i \right)$ while eliminating the multi-symbol interference.

$${\hat{\mathbf{s}}}_{k,LS} \left( i \right) = \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {\mathbf{y}}_{k} \left( i \right)$$

(37)

Substituting (31) into (37), we have

$${\hat{\mathbf{s}}}_{k,LS} \left( i \right) = \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {\mathbf{y}}_{k} \left( i \right) = \sqrt {P^{k} \gamma_{{}}^{k} } {\mathbf{s}}_{k} \left( i \right) + {\mathbf{w}}_{k} \left( i \right)$$

(38)

where ${\mathbf{w}}_{k} \left( i \right) \equiv \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {\mathbf{v}}_{k} \left( i \right)$ is a D-by-1 Gaussian random vector with zero-mean and covariance matrix given by

$$\begin{gathered} E\left\{ {{\mathbf{w}}_{k} \left( i \right){\mathbf{w}}_{k}^{H} \left( i \right)} \right\} \equiv \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} E\left\{ {{\mathbf{v}}_{k} \left( i \right){\mathbf{v}}_{k}^{H} \left( i \right)} \right\}{{\varvec{\Gamma}}}_{D}^{k} \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \\ = \sigma^{2} \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{k} {{\varvec{\Gamma}}}_{D}^{k} \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \\ \end{gathered}$$

(39)

Therefore, the SNR for the dth symbol can be obtained as

$$SNR_{k,d} = \frac{{P^{k} \gamma_{{}}^{k} }}{{\sigma^{2} \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} \left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{k} {{\varvec{\Gamma}}}_{D}^{k} \left[ {\left( {{{\varvec{\Gamma}}}_{D}^{k} } \right)^{H} {{\varvec{\Gamma}}}_{D}^{k} } \right]^{ - 1} {(}d,d)}};d = 1, \ldots ,D$$

(40)

And the corresponding symbol error rate for M-ary PSK modulated signal can be obtained from (27), $2Q\left( {\sqrt {SNR_{k,d} } \sin \frac{\pi }{M}} \right)$, accordingly.

4 Performance evaluation

In this section, we aim at evaluating performances of the proposed prefiltering-based GSM multiuser receiver under various scenario and compare with existing algorithms. Unless otherwise mentioned, the array sizes for BS are set as L = 64. Each point of the Monte Carlo simulation is resulted from the average of 100,000 independent trials. The parameter of SNR for evaluation is defined as ${\text{SNR}} \equiv 10\log_{10} \frac{P}{{\sigma^{2} }}\left( {{\text{dB}}} \right)$. Since identification of active antennas is essential in SM, hence, we first verify the correct detection probability of the active antenna, $P_{{\text{ant correct}}}^{k} { = }P\left( {\hat{j}_{k} = j_{k} } \right)$. Figure 4 presents $P_{{\text{ant correct}}}^{k}$ with respect to the window size (J) that is used to estimate the correlation matrix. The SNR is fixed to be 6 dB, and J varies from 4 to 16. Both the simulation and the analytical results (as derived in (24)) of the proposed as well as the EVD-based [24] algorithms are provided for comparison. From the results of Fig. 4, several observations can be made:

• 1. $P_{{\text{ant correct}}}^{k}$ increases in accordance with J. This arises from the fact that larger J leads to better estimation accuracy of the correlation matrix.

• 2. The convergence speed of the proposed algorithm is much faster than the EVD-based algorithm. This is owing to the spatial prefiltering that largely reduce the rank of the correlation matrix. Moreover, we verify that for fixed J, the proposed scheme outperforms the EVD-based algorithm.

• 3. The simulation and the analytical results of the proposed algorithm are closely matched.

Fig. 4

$P_{{\text{ant correct}}}^{k}$ with respect to the window size (J), where the analytical and simulation results of the proposed GEVD algorithm and the EVD-based algorithm are provided for comparison

The second simulation highlights the impact of the array size (N) at the UT on $P_{{\text{ant correct}}}^{k}$. Figure 5 presents $P_{{\text{ant correct}}}^{k}$ with respect to SNR, where we set L = 64, J = 12 and the scenarios for N = 6, 12, and 18 are provided for comparison. As depicted in Fig. 5, larger N corresponds to better performance at the low SNR region, yet $P_{{\text{ant correct}}}^{k}$ approaches 1 as SNR exceeds 5 dB. The third simulation evaluates the proposed algorithm in GSM scheme. Figure 6 presents $P_{{\text{ant correct}}}^{k}$ with respect to SNR, where the cases for SM (D = 1), GSM (D = 2), and GSM (D = 3) are provided for comparison. The window size is set to be J = 10. As shown in Fig. 6, $P_{{\text{ant correct}}}^{k}$ degrades for larger number of active antennas in the low SNR scenario, nevertheless, it approaches to 1 for all three cases as SNR exceeds 8 dB. Exploiting Eq. (28), we evaluate the upper bound of overall symbol error rate, $P_{{\text{overall error}}}^{k}$, with respect to SNR for D = 1, 2, 3 and the simulation result is depicted in Fig. 7. The temporal symbol constellation size is set to be M = 8. Though Fig. 7 reveals that $P_{{\text{overall error}}}^{k}$ increases for more active antennas, nevertheless, the benefits for spectral efficiency enhancement for GSM overwhelm the slightly degradation in overall SER.

Fig. 5

$P_{{\text{ant correct}}}^{k}$ with respect to SNR for the proposed algorithm in SM scheme, where different UT’s array size (N = 6, 12, 18) are provided for comparison

Fig. 6

Comparison of the active antennas correct detection probability for the cases of single, two, and three active antennas, J = 10

Fig. 7

The symbol error rate with respect to SNR for the SM and GSM (two and three antennas are activated) schemes

In the last simulation, we compare $P_{{\text{overall error}}}^{k}$ of the proposed scheme with existing algorithms, e.g., EVD-based [24], SAGE [22], ${\text{NNS}}\left( {1,1} \right)$ and ${\text{NNS}}\left( {3,3} \right)$ [21], in which 16 QAM is employed. The window size, numbers of UT, BS receiving antennas, UT transmitting antennas are set as J = 12, K = 10, L = 64, and N = 8, respectively. Since both the NNS and SAGE are applied only in SM (D = 1), to treat the comparison fairly, we apply the proposed and the EVD-based algorithms only on the SM scheme. Please note that there are two important parameters in NNS algorithm, namely $K_{S}$ and $N_{S}$:$K_{S}$ denotes the maximum number of entries which can be changed simultaneously. $N_{S}$ is the number of nearest neighbors. These two parameters determine the size of the candidate list as well as the computation load of the algorithm. In the simulation, we set ${\text{NNS}}\left( {K_{S} ,N_{S} } \right)$ as ${\text{NNS}}\left( {1,1} \right)$ and ${\text{NNS}}\left( {3,3} \right)$, respectively, for comparison. As shown in Fig. 8, the proposed prefiltering-based method is superior to EVD-based, ${\text{NNS}}\left( {1,1} \right)$ and even ${\text{NNS}}\left( {3,3} \right)$ while slightly worse than the SAGE algorithm in high SNR region. Nevertheless, the proposed algorithm is flexible than SAGE and NNS in that it can be easily extended to the GSM scheme.

Fig. 8

Comparison of the proposed prefiltering-based detector with the EVD-based, SAGE and NNS algorithms in terms of SER performance

5 Conclusions

In this paper, we have proposed a prefiltering-based reduced rank multiuser GSM detection scheme for uplink massive MIMO system. Benefited from the large-scale number of antennas at BS, MUI elimination is fulfilled by spatial prefiltering at the front end of BS receiver. After pre-filtering, we proposed a reduced rank GEVD-based algorithm to identify the active antennas of each UT. In the second stage, the LS-based MUD is utilized to decode the multiplexed temporal symbols at the activated antennas. We have demonstrated in Fig. 4 that the proposed prefiltering-based algorithm can work reliably for small window size, which is essential for the low latency requirement in current and future network. We have also verified in Fig. 4 and Fig. 8 that it is close or better than the existing algorithms, yet our proposed method is with low complexity and more flexible.