Partial interference cancellation with maximum likelihood sequence detection in FBMC spatial multiplexing system

Abstract

Intrinsic interference in filter bank multicarrier (FBMC) modulation prevents the maximum likelihood (ML) detection in spatial multiplexing (SM) system. This intrinsic interference is caused by the transmultiplexer impulse response in time-frequency domain. Solutions based on interference cancellation are not always effective because they may introduce error propagation. In this paper, we propose to study some receivers based on reducing the interference concerned by the cancellation. These solutions can be seen as a trade-off between the whole Viterbi (ML) detection and the whole interference cancellation. The principle of the proposed receivers is to partially estimate and cancel the interference, and the rest of the interference is processed by a low complexity Viterbi detector. We show in this work that the receiver performance depends on the transmultiplexer impulse response and on the choice of the partial Viterbi detector.

1 Introduction

Nowadays, the most well-known multicarrier modulation is the orthogonal frequency division multiplex (OFDM) modulation. Thanks to the cyclic prefix (CP) insertion, OFDM technique provides a high robustness against the multipath fading channels. However, since OFDM uses CP and also rectangular pulse shape filtering that causes some disadvantages concerning the spectrum control and efficiency, to tackle this issue, filter bank multicarrier systems with offset quadrature amplitude modulation (FBMC/OQAM) was proposed as an alternative to CP-OFDM. Many works on the comparison between FBMC/OQAM and OFDM can be found in the literature such as [1]. One of the features of FBMC/OQAM is the possibility of shaping subcarrier signals with well-localized prototype filter in time and frequency axis [1]. That is, sidelobes steeply decay and allow a better spectrum control and offer an important resistance against time and frequency misalignment. Thus, FBMC/OQAM enables asynchronous multiple access and reduces intercarrier interference since each subcarrier signal is spectrally confined in a band and has negligible interference to other bands. In cognitive radios, the filtering capability of FBMC systems makes them the perfect choice for filling in the spectrum holes [1]. In the other hand, thanks to its spectral confinements, FBMC-based systems can reduce their guard band at the frequency boundaries, thereby increasing the spectral efficiency. Moreover, the absence of CP and any guard interval in time domain also contributes to increase the spectral efficiency. Many research works have been carried out in the issues related to doubly spread channels, and it has been shown that FBMC modulations are far better choices when compared to OFDM [2–4]. For instance, the author in [1] clearly shows the outperformance of the FBMC over OFDM in terms of the resilience to the doubly dispersive channels under the same assumptions of channel mobility and symbol density.

In FBMC/OQAM, each subcarrier is modulated with an OQAM transmitting pulse-amplitude modulation (PAM) symbols at each half a period (T/2). The orthogonality condition is considered only in the real field [5]. Indeed, the data at the receiver side is carried only by the real components of the signal, and the imaginary parts appear as interference terms. This interference is caused by the data symbols transmitted in the neighborhood area in the time-frequency domain. Since the interference is orthogonal to the useful data symbols, data detection is easily performed when the channel is flat fading or lowly selective [6]. However, in some situations like when the channel is highly selective, the intrinsic interference may cause performance degradation. Several works based on sub-channel equalization were carried out to tackle to this issue as in [7, 8] where multi-tap equalization is used. In [9], the authors have performed a minimum mean square error (MMSE) per subcarrier equalization. Whereas in [10, 11] the authors propose to implement a per-subcarrier decision-feedback equalizer (DFE). The extension of the latter to the multiple-input multiple output (MIMO) spatial multiplexing (SM) system was addressed in [12]. The per-subchannel equalization in FBMC with MIMO systems was also studied in [13, 14]. Some other works [15–20] address the design of MIMO precoding and decoding techniques for the FBMC modulation. For low MIMO selective channels, linear equalizations such as zero forcing (ZF) or MMSE can be applied to FBMC as shown in [21].

The presence of the inherent interference causes problems when combining FBMC/OQAM with some MIMO techniques such as Alamouti space-time block coding (STBC) scheme and SM systems with maximum likelihood (ML) detection. Regarding the Alamouti scheme, its application in a straightforward manner to the FBMC makes an inherent interference appear that cannot be easily removed [22]. Many works have been carried out on this topic such as [22] where the authors show that Alamouti coding can be performed but only when it is combined with code division multiple access (CDMA). A pseudo-Alamouti scheme was introduced in [23] where it is combined with single-carrier FBMC using the cyclic prefix (CP). Another solution was proposed by Renfors et al. in [24] where the Alamouti coding is performed in a block-wise manner. We have proposed in [25, 26] an iterative Alamouti scheme for FBMC based on intrinsic interference cancellation. On the other side, ML detection in SM system, which is supposed to offer a diversity order equal to the number of the receive antennas [27], cannot be applied straightforwardly with FBMC due to the presence of the interference.

Full ML receivers, in principle, offer the best possible performance but require an impractically high complexity when the impulse response is long. Moreover, the intrinsic interference in FBMC is two-dimensional which further complicates the detection task in maximum likelihood sense. Interference cancellation approaches generally offer the possibility of removing interference with low complexity increase and without enhancing the level of noise already present in the received signal [28]. Interference cancellation schemes are essentially based on using preliminary decisions to estimate and cancel the interference. In [29], we have addressed the possibility to cancel the interference before applying a local ML detection and proposed some receiver schemes. However, the obtained performance was limited and far from the optimum due to the error propagation. We have also proposed in [26, 30] to modify the FBMC/OQAM modulation by transmitting QAM symbols at each period of T instead of transmitting PAM symbols at each T/2. We referred to this modulation as FBMC/QAM. This modulation allows to reduce the intrinsic interference power but at the expense of the orthogonality [26]. Error propagation problem in iterative inter-symbol interference (ISI) cancellation has been addressed in many research fields. In the general case, the authors in [31] have established conditions under which the cancellation scheme is effective. The idea is to consider the interference as the sum of two terms. The first one is cancelled by using tentative decisions, and the second uncancelled one is considered by a maximum likelihood sequence equalizer (MLSE).

In this work, in order to counteract the error propagation in SM-FBMC and make the cancellation scheme effective, we were inspired by [31]. We apply the theory developed in [31] and propose some receiver structures that satisfy the conditions of the interference cancellation effectiveness. The receivers are based on partial interference cancellation followed by a Viterbi detector instead of an ML detector. The tentative detector is first used to only partially cancel the intrinsic interference. We have presented a part of this work in [32] where we have only focused on FBMC/OQAM with one prototype filter. In this paper, both FBMC/OQAM and FBMC/QAM modulations are addressed, and their performances are compared. Furthermore, each of the both modulations are analyzed with two different prototype filters.

The paper is organized as follows. We start in Section 2 by giving an overview on FBMC/OQAM modulation highlighting the issue of FBMC when it is combined with SM-ML detection. Then, in Section 3, we give a background on the use of tentative decisions to cancel the interference, and we present the principle of partial interference cancellation. Sections 4 and 5 are devoted to the analysis and adaptation of the partial interference cancellation to the FBMC/OQAM and FBMC/QAM, respectively. Simulation results of the different proposed receivers are presented in Section 6. Then, we finish by a conclusion in Section 7.

2 The FBMC/OQAM modulation

2.1 System model

At the transmitter side, the discrete time of the equivalent baseband FBMC signal is written as follows [5]:

$$ s[\!m] = \sum_{k=0}^{M-1}\sum_{n \in \textbf{Z}} a_{k,n}g[\!m-nM/2]e^{j\frac{2\pi}{M} k\left(m-\frac{D}{2}\right)}e^{j\phi_{k,n}}, $$

((1))

where M is an even number of subcarriers, g[ m] is the prototype filter taking values in real field, and $\frac {D}{2}$ is the delay term which depends on the length (L _g ) of g[ m]. For the sake of simplification, we can set, as in [5], D=L _g −1 and L _g =K M, where K is the overlapping factor. The transmitted symbols a _k,n are real-valued symbols which are the real or the imaginary parts of QAM symbols. The additional phase term ϕ _k,n is given by $\phi _{k,n}=\frac {\pi }{2}(n+k)-\pi nk$. We can rewrite (1) in a simple manner

$$ s[\!m] = \sum_{k=0}^{M-1}\sum_{n \in \textbf{Z}} a_{k,n}g_{k,n}[\!m], $$

((2))

where g _k,n [ m] are the shifted versions of g[ m] in time and frequency. In the case of a perfect channel without noise, the demodulated symbol over the k ^′th subcarrier and the n ^′th instant is determined using the inner product of s[ m] and $g_{k',n'}[\!m]\phantom {\dot {i}\!}$

$$\begin{array}{*{20}l} {} r_{k^{\prime},n^{\prime}} = \langle s,g_{k^{\prime},n^{\prime}}\rangle &= \sum_{m=-\infty}^{+\infty} s[\!m]g_{k^{\prime},n^{\prime}}^{*}[\!m] \\ &=\sum_{m=-\infty}^{+\infty}\sum_{k=0}^{M-1}\sum_{n \in \textbf{Z}} a_{k,n} g_{k,n}[\!m]g_{k^{\prime},n^{\prime}}^{*}[\!m]. \end{array} $$

((3))

The transmultiplexer impulse response can be derived assuming null data except at one time-frequency position (k ₀,n ₀) where a unit impulse is applied. Then, Eq. (3) becomes

$$\begin{array}{*{20}l} r_{k^{\prime},n^{\prime}} &= \sum_{m=-\infty}^{+\infty} g_{k_{0},n_{0}}[\!m]g_{k^{\prime},n^{\prime}}^{*}[\!m]\\ &=\sum_{m=-\infty}^{+\infty}g[\!m]g\left[\!m-\Delta n M/2\right]e^{j\frac{2\pi}{M} \Delta k\left(\frac{D}{2}-m\right)} \\ & \qquad e^{j\pi \left(\Delta k+k_{0}\right)\Delta n}e^{-j\frac{\pi}{2}\left(\Delta k+ \Delta n\right)}=\Gamma_{\Delta k, \Delta n}^{\left(k_{0}\right)}, \end{array} $$

((4))

where Δ n=n ^′−n ₀, Δ k=k ^′−k ₀ and $\Gamma _{\Delta k, \Delta n}^{(k_{0})}$ is then the transmultiplexer impulse response coefficients. It is worth noticing that the impulse response of the transmultiplexer depends on k ₀. Indeed, because the multiplicative factor $\phantom {\dot {i}\!}e^{j\pi k_{0}\Delta n}=(-1)^{k_{0}\Delta n}$, the sign of some impulse response coefficients (with Δ n odd) depends on the parity of k ₀.

The prototype filters are designed in a way that they provide a well-localized spectrum and spread over only a few adjacent subcarriers. Several pulse shaping prototype filters g[ m] can be used according to their properties. A survey on prototype filters can be found in [33]. In this paper, we consider two different pulse shapes with overlapping factor set to K=4. The first one is that which was adopted in the European Physical Layer for Dynamic Access (PHYDYAS) project on FBMC [34] and is referred to as the PHYDYAS prototype filter. The second is the one based on the so-called isotropic orthogonal transform algorithm (IOTA) function [3]. IOTA filter is designed according to the energy concentration criterion, it is obtained by orthogonalizing the Gaussian function to prevent interference to neighboring points in the lattice [2]. Thus, it targets to preserve the optimum concentration property of Gaussian filter and fulfills the Nyquist criterion and can yield isotropic response in time and frequency [33]. As for PHYDYAS filter, it is designed according to the criterion of rapid decaying of the sidelobes. It is shown in [35] that, for an overlapping factor of K=4, the minimum stop-band attenuation is 39.86 dB, and the sidelobes fall at |ω|⁻³ where ω is the angular frequency. Although the decaying rate and the stop-band attenuation increase with increasing the overlapping factor K, the choice of K=4 is considered as a good trade-off between the stop-band attenuation and the system delay caused by the time spread of the filter [36]. Tables 1 and 2 depict the transmultiplexer impulse response coefficients $\left (\Gamma _{\Delta k, \Delta n}^{(k)}\right)$ for even subcarriers k of the PHYDYAS and IOTA filter, respectively. We should note that the transmultiplexer impulse response is independent from the subcarrier number (M) as long as M is sufficiently high, M≥8. All the prototype filters g[ m] are designed to satisfy the real orthogonality condition given by [5]

$$ \mathfrak{Re}\left\{ \sum_{m=-\infty}^{\infty}g_{k^{\prime},n^{\prime}}[\!m]g_{k,n}^{*}[\!m] \right\} =\delta_{k,k^{\prime}}\delta_{n,n^{\prime}}. $$

((5))

Table 1 FBMC/OQAM transmultiplexer impulse response (main part) using PHYDYAS filter

Table 2 FBMC/OQAM transmultiplexer impulse response using IOTA filter

Let us consider the SISO FBMC transmission. When passing through the radio channel and adding the zero-mean and circular white Gaussian noise contribution b[ m], Eq. (3) becomes [37]

$$ {{}\fontsize{8.98pt}{6pt}\selectfont{\begin{aligned} r_{k^{\prime},n^{\prime}}=h_{k^{\prime},n^{\prime}}a_{k^{\prime},n^{\prime}} &+\underbrace{\sum_{(k,n)\neq(k^{\prime},n^{\prime})}h_{k,n}a_{k,n}\sum_{m=-\infty}^{m=\infty}g_{k,n}[\!m]g_{k^{\prime},n^{\prime}}^{*}[\!m]}_{I_{k^{\prime},n^{\prime}}} \\ &+\underbrace{\sum_{m=-\infty}^{+\infty} b[\!m]g_{k^{\prime},n^{\prime}}^{*}[\!m]}_{\gamma_{k^{\prime},n^{\prime}}}, \end{aligned}}} $$

((6))

where $h_{k',n'}\phantom {\dot {i}\!}$ is the channel coefficient at subcarrier k ^′ and time index n ^′ and the term $\phantom {\dot {i}\!}I_{k',n'}$ is defined as an intrinsic interference. Since the prototype filters have a finite support in time and we assume that they are spectrally well-localized, then the energy of the impulse response is confined in a finite set around the considered symbol [6]. Consequently, we can consider that the intrinsic interference term depends only on this restricted set (denoted by $\phantom {\dot {i}\!}\Omega _{k',n'}$). Moreover, for the sake of derivation simplicity, we consider that the channel is sufficiently low-selective such that we can assume that the channel coefficient h _k,n is approximately constant at least over the summation set Ω _k,n . Hence, we can write as in [37]

$$ {{}\fontsize{7.98pt}{6pt}\selectfont{\begin{aligned} r_{k^{\prime},n^{\prime}}\approx h_{k^{\prime},n^{\prime}} \left(a_{k^{\prime},n^{\prime}}+\underbrace{\sum_{(k,n)\in \Omega_{k^{\prime},n^{\prime}}} a_{k,n} \sum_{m=-\infty}^{\infty}g_{k,n}[\!m]g_{k^{\prime},n^{\prime}}^{*}[\!m]}_ {\hat{I}_{k^{\prime},n^{\prime}}} \right)+\gamma_{k^{\prime},n^{\prime}}. \end{aligned}}} $$

((7))

According to (5) and because a _k,n is real-valued, the intrinsic interference $\hat {I}_{k',n'}=ju_{k',n'}$ is pure imaginary. Thus, we can write the demodulated signal as

$$\begin{array}{*{20}l} r_{k^{\prime},n^{\prime}}\approx h_{k^{\prime},n^{\prime}}\left(a_{k^{\prime},n^{\prime}}+ju_{k^{\prime},n^{\prime}}\right)+\gamma_{k^{\prime},n^{\prime}}. \end{array} $$

((8))

It is worth noting that the noise term $\gamma _{k',n'}\phantom {\dot {i}\!}$ defined in Eq. (6) is colored, and its autocorrelation is given by

$$\begin{array}{*{20}l} \mathbb{E}\left\{\gamma_{k^{\prime},n^{\prime}}\gamma_{k,n}^{*}\right\}&=\sigma^{2}\sum_{m=-\infty}^{\infty}g_{k,n}[\!m]g_{k^{\prime},n^{\prime}}^{*} [\!m]\\ &= \sigma^{2}\Gamma_{k^{\prime}-k,n^{\prime}-n}^{(k)}. \end{array} $$

((9))

The last equality is obtained according to Eq. (4).

2.2 Problem statement

In MIMO spatial multiplexing context, when N _t antennas are used to transmit N _t real data symbols and N _r antennas are used to collect the transmitted signals, the FBMC demodulated signal at the tth receive antenna and at a given time-frequency position (k,n) can be expressed by

$$ r_{k,n}^{(t)}=\sum_{i=1}^{N_{t}}h_{k,n}^{(ti)}\left(a_{k,n}^{(i)}+ju_{k,n}^{(i)}\right)+\gamma_{k,n}^{(t)}. $$

((10))

We should recall that the above expression is only valid under the assumption of the low-selectivity of the channels, as assumed for (7), between the transmit and receive antennas. Finally, the matrix formulation of the system can be expressed as follows:

$$ \mathbf{r}_{k,n}=\mathbf{H}_{k,n}\left(\mathbf{a}_{k,n}+j\mathbf{u}_{k,n}\right)+\boldsymbol{\gamma}_{k,n}, $$

((11))

where H _k,n is an (N _r ×N _t ) channel matrix with the element $h_{k,n}^{(ti)}$ at the tth row and ith column, $\mathbf {r}_{k,n}=\left [r_{k,n}^{(1)}, r_{k,n}^{(2)},\ldots,r_{k,n}^{(N_{r})}\right ]^{T}$ is the received vector, $\mathbf {a}_{k,n}=\left [a_{k,n}^{(1)}, a_{k,n}^{(2)},\ldots,a_{k,n}^{(N_{t})}\right ]^{T}$ is the transmitted symbol vector, $\mathbf {u}_{k,n}=\left [u_{k,n}^{(1)}, u_{k,n}^{(2)},\ldots,u_{k,n}^{(N_{t})}\right ]^{T}$ is the vector of the interference values, and $\boldsymbol {\gamma }_{k,n}=\left [\gamma _{k,n}^{(1)}, \gamma _{k,n}^{(2)},\ldots,\gamma _{k,n}^{(N_{r})}\right ]^{T}$ is the noise vector.

Linear equalization as ZF and MMSE in FBMC/OQAM context can be performed as described in [21] where a virtually transmitted vector c _k,n =a _k,n +j u _k,n is considered instead of the effective one. The vector r _k,n represents the input of the equalizer having as output the equalized virtually transmitted vector $\tilde {\mathbf {c}}_{k,n}$

$$ \tilde{\mathbf{c}}_{k,n}=\mathbf{G}_{k,n}^{H}\mathbf{r}_{k,n}, $$

((12))

where G _k,n is the equalization matrix based on the ZF or MMSE criterion. Then, the real part retrieval of $\tilde {\mathbf {c}}_{k,n}$ yields the real equalized data vector $\tilde {\mathbf {a}}_{k,n}$ [21].

As for maximum likelihood (ML) detection, the presence of the interference vector term u _k,n in (11) prevents the application of ML separately at each time-frequency position (k,n). This is because the interference terms take their values in a large set and depend on the transmitted data symbols in the neighborhood around the considered position (k,n). The global ML detector is the one that considers all the transmitted data symbols within a frame. Obviously, such a receiver implementation is by far impractical due to its huge complexity.

3 Overview on partial interference cancellation

In [31], the authors have considered the channel model depicted in Fig. 1, where f ₀(a _k ,a _k−1,…,a _k−δ+1) is a function of δ data symbols and represents the target response expected by the receiver. f ₁(a _k+γ ,a _k+γ−1,…,a _k−λ+1) is a function of γ+λ data symbols and represents a small channel perturbation.

Fig. 1

Channel model

It should be noted that, in general, both f ₀ and f ₁ may be nonlinear functions. The samples of the signal at the input of the receiver are

$$\begin{array}{*{20}l} r_{k} &= \,f_{0}\left(a_{k}, a_{k-1}, \ldots, a_{k-\delta+1}\right) \\ & \quad + f_{1}\left(a_{k+\gamma}, a_{k+\gamma-1}, \ldots, a_{k-\lambda+1}\right) + w_{k}, \end{array} $$

((13))

where w _k is the noise contribution. The receiver is composed by a tentative detector producing tentative decisions and a main Viterbi detector which assumes that the channel is described anly by f ₀. Before performing the main Viterbi detector, the tentative decisions are used only to cancel the remaining ISI (RISI) represented by f ₁. The receiver scheme is depicted in Fig. 2.

Fig. 2

Receiver scheme with ISI cancellation using tentative decisions

Given the correct data sequence {a _k }, let us define a wrong sequence $\left \{a_{k}^{(\epsilon)}\right \}$ where the wrong symbols $a_{k}^{(\epsilon)}$ are determined by a given error event ε with $a_{k}^{(\epsilon)} = a_{k} + \epsilon _{k}$. That is, for a value of k, if ε _k =0, then the symbol $a_{k}^{(\epsilon)}$ in the sequence $\left \{a_{k}^{(\epsilon)}\right \}$ is correct. Let us define also

$$\begin{array}{*{20}l} \Delta_{k}^{(\epsilon)} &= f_{0}\left(a_{k}^{(\epsilon)}, a_{k-1}^{(\epsilon)}, \ldots, a_{k-\delta+1}^{(\epsilon)}\right) \\ & \quad- f_{0}\left(a_{k}, a_{k-1}, \ldots, a_{k-\delta+1}\right), \end{array} $$

((14))

$$\begin{array}{*{20}l} \Phi(\epsilon)=\left[\Delta_{0}^{(\epsilon)}, \Delta_{1}^{(\epsilon)}, \ldots, \Delta_{K-1}^{(\epsilon)}\right]^{T}, \end{array} $$

((15))

where K is assumed to be the total number of transmitted symbols. The authors in [31] classified the error events in terms of their distance d ₀(ε) in the absence of RISI (f ₁ =0), which is given, in the presence of correlated noise, by

$$\begin{array}{*{20}l} d_{0}(\epsilon)=\frac{\|\Phi(\epsilon)\|^{2}}{\sqrt{\Phi^{H}(\epsilon)\mathbf{R}\Phi(\epsilon)}}, \end{array} $$

((16))

where $\mathbf {R}=\mathbb {E}\{\mathbf {ww}^{H}\}/{\sigma _{w}^{2}}$ is the normalized noise autocovariance matrix, with ${\sigma _{w}^{2}}$ is the noise variance and w=[w ₀,w ₁,…,w _K−1]^T. The events whose distance d ₀(ε) is the minimum are called “first-order” error events. Similarly, events whose distance is the second smallest are called “second-order” error events and so on [31].

The conditions for which RISI cancellation is satisfying are summarized as follows [31]:

• 1. Errors affecting the main (Viterbi detector) and the tentative detector must be statically independent.

• 2. The RISI (described by f ₁) must be small enough to guarantee that the main Viterbi detector can make relatively reliable decisions even when the tentative detector makes a decision error and such that the tentative detector also makes relatively reliable decisions in spite of the ISI.

• 3. The distance of second-order and higher-order error events that could cause error propagation must be significantly larger than that of first-order error events.

4 Application to MIMO-SM FBMC/OQAM

Now that the conditions for effective RISI cancellation are summarized, we will attempt to apply them to FBMC. Hence, the problem is, essentially, how to select the functions f ₀ and f ₁ such that the three conditions cited above are fulfilled. The intrinsic interference in FBMC is seen as a two-dimensional intersymbol interference (2D-ISI). An extension of the works of Agazzi and Seshardi [31] to 2D-ISI channels was treated in [38] assuming that the noise samples are uncorrelated (which is not the case in FBMC). Hence, in general, the target response f ₀ may also represent a 2D-ISI channel. Then, a 2D-Viterbi algorithm is required to match with f ₀. Designing a 2D-Viterbi is quite challenging. Therefore, for simplicity reasons, we opted to set the additional constraint that the target response f ₀ must be one-dimensional and that f ₁ covers the rest of 2D-ISI.

Obviously, the receiver complexity depends essentially on the complexity of the Viterbi detector. Therefore, we have to choose a configuration with the least complex Viterbi detector that meets the conditions for effective RISI cancellation. We will select three configurations with different sizes of the target response f ₀. According to the second condition, f ₀ must contain the largest coefficients $\left (\Gamma _{\Delta k,\Delta n}^{(k)}\right)$ in each configuration. Hence, from Table 1, the selected target responses are [32]

$$\begin{array}{*{20}l} &f_{0}^{(1)}(a_{k,n})=a_{k,n}, \end{array} $$

((17))

$$\begin{array}{*{20}l} &f_{0}^{(2)}(a_{k,n},a_{k,n-1})=a_{k,n}+\Gamma_{0,-1}^{(k)}\times a_{k,n-1}, \end{array} $$

((18))

and

$$\begin{array}{*{20}l} {}f_{0}^{(3)}(a_{k,n+1},a_{k,n},a_{k,n-1}) &= \Gamma_{0,1}^{(k)} a_{k,n+1}+a_{k,n} \\ &\quad +\Gamma_{0,-1}^{(k)} a_{k,n-1}. \end{array} $$

((19))

The first configuration $\left (f_{0}^{(1)}\right)$ corresponds to the whole ISI cancellation which has been studied in [29, 39]. Since, in FBMC, we have $\sum _{p,q}|\Gamma _{p,q}^{(k)}|^{2} = 2$ [6], it is easy to calculate the power of the RISI (represented by f ₁) for each configuration.

Regarding the first condition, it is easily satisfied when the tentative detector is different from the main one (Viterbi) [31]. We recall that we consider the case of spatial multiplexing system. Then, we chose the MIMO-MMSE equalizer as the tentative detector.

The third condition concerns the spectrum distances d ₀(ε) defined by (16). Hence, for each configuration $\left (f_{0}^{(i)}\right.$, $\left.{\vphantom {f^{(i)}_{0}}}i\in \{1,2,3\}\right)$, we compare the non-minimum distances to the minimum one. Then, according to (16), we have first to determine the matrix R. Since we consider that the target response are one-dimensional and Viterbi algorithm is performed on each subcarrier “ k”, the noise vector w is then w= [w _k,n ,w _k,n+1,…,w _k,n+K−1]^T. Hence, the matrix $\mathbf {R}=\mathbb {E}\{\mathbf {ww}^{H}\}/{\sigma _{w}^{2}}$ is only composed by the coefficients $\Gamma _{0,q}^{(k)}$, $q\in \mathbb {Z}$ and is given by

$$ {{}\fontsize{7.9pt}{6pt}\selectfont{\begin{aligned} \mathbf{R}&=\frac{1}{{\sigma_{w}^{2}}} \left[\begin{array}{cccc} \mathbb{E}\{w_{k,n}w_{k,n}^{*}\} &\mathbb{E}\{w_{k,n}w_{k,n+1}^{*}\} &\mathbb{E}\{w_{k,n}w_{k,n+2}^{*}\} &\cdots\\ \mathbb{E}\{w_{k,n+1}w_{k,n}^{*}\} & \mathbb{E}\{w_{k,n+1}w_{k,n+1}^{*}\} &\mathbb{E}\{w_{k,n+1}w_{k,n+2}^{*}\} &\cdots\\ \mathbb{E}\{w_{k,n+2}w_{k,n}^{*}\} &\mathbb{E}\{w_{k,n+2}w_{k,n+1}^{*}\} & \mathbb{E}\{w_{k,n+2}w_{k,n+2}^{*}\} &\cdots\\ \vdots &\vdots &\vdots &\ddots \end{array}\right]_{K\times K} \\ &= \left[\begin{array}{cccc} 1 &\Gamma_{0,1}^{(k)} &\Gamma_{0,2}^{(k)} &\cdots\\ \Gamma_{0,-1}^{(k)} & 1 &\Gamma_{0,1}^{(k)} &\cdots\\ \Gamma_{0,-2}^{(k)} &\Gamma_{0,-1}^{(k)} & 1 &\cdots\\ \vdots &\vdots &\vdots &\ddots \end{array}\right]_{K\times K,} \end{aligned}}} $$

((20))

where the last equality is obtained thanks to Eq. (9).

In Tables 3 and 4, we summarize the values of the first- and second-order distances obtained by using (16) and also the power of the RISI for the three considered configurations and both the PHYDYAS and IOTA filters, respectively.

Table 3 Spectrum distances and RISI power for FBMC/OQAM with the PHYDYAS filter

Table 4 Spectrum distances and RISI power for FBMC/OQAM with the IOTA filter

Let us denote by ε ₁ and ε ₂ the first- and second-order error events, respectively. We also define w(ε) as the number of error position in the error event ε. The values of w(ε) are also shown in the table. We remark that the difference between the second-order and the first-order distances is almost the same for all the configurations (0.8±0.03), so we consider (as considered in [31]) that the higher-order distances are sufficiently larger than the minimum distance for each configuration. Hence, condition 3 is fulfilled for the three configurations.

Now, we have only to determine the configuration(s) for which the second condition is satisfied. Unfortunately, the determination of the RISI power for which the cancellation starts to be effective (or equivalently, error propagation ceases) is not trivial and depends also on the noise variance σ ² [31]. We will show by simulations that only the third configuration $\left (f_{0}^{(3)}\right)$ allows to obtain effective RISI cancellation.

As for the receiver complexity, it strongly depends on that of the Viterbi detector. When we consider a spatial multiplexing system with N _t transmit antennas, the Viterbi detector has to compute $q^{i\times N_{t}}\phantom {\dot {i}\!}$ branch metrics, where q is the number of all possible symbols a _k,n (constellation size) and i∈{1,2,3} is the number of the taps in $f_{0}^{(i)}$. In order to reduce the receiver complexity, we can replace the Viterbi detection algorithm by the M-Algorithm [40] which keeps only a fixed number (J) of inner states instead of all the inner states $\left (q^{(i-1)\times N_{t}}\right)\phantom {\dot {i}\!}$. Hence, the M-algorithm has to compute only the $\phantom {\dot {i}\!}J\times q^{N_{t}}$ branch metrics. In the rest of the paper, we call the proposed receivers “PaIC/Viterbi” (for Partial Interference Cancellation with Viterbi detector) and can be followed by an index to indicate the considered configuration.

5 Application of PaIC/Viterbi receivers in FBMC/QAM

Aiming to reduce the intrinsic interference power, we consider in this section the FBMC system transmitting complex QAM symbols instead of real ones (OQAM) [30]. That is, instead of transmitting real symbols a _k,n at each half a period (T/2), complex symbols s _k,n are transmitted at each period (T). Therefore, the equivalent baseband discrete time FBMC/QAM signal can be deduced from Eq. (1) and given by

$$ s[\!m] = \sum_{k=0}^{M-1}\sum_{n \in \textbf{Z}} s_{k,n}g[\!m-nM]e^{j\frac{2\pi}{M} k\left(m-\frac{D}{2}\right)}e^{j\theta_{k,n}}, $$

((21))

where $\theta _{k,n} = \phi _{k,2n}=\pi n + \frac {\pi }{2}k$. Consequently, the transmultiplexer impulse response coefficients $\acute {\Gamma }_{p,q}$ can be also derived from Eq. (4) and expressed as

$$ {{}\fontsize{8.9pt}{6pt}\selectfont{\begin{aligned} \acute{\Gamma}_{\Delta k, \Delta n} = \sum_{m=-\infty}^{+\infty}g[\!m]g[\!m-\Delta n M]e^{j\frac{2\pi}{M} \Delta k\left(\frac{D}{2}-m\right)}e^{-j\frac{\pi}{2}(\Delta k+ 2\Delta n)}. \end{aligned}}} $$

((22))

It is worthy to note that, in contrast to ${\Gamma }_{\Delta k, \Delta n}^{(k)}$, the coefficients $\acute {\Gamma }_{\Delta k, \Delta n}$ do not depend on the subcarrier index k. The impulse response coefficients $\acute {\Gamma }_{\Delta k, \Delta n}$ for FBMC/QAM using the PHYDYAS and IOTA filters are depicted in Tables 5 and 6, respectively.

Table 5 FBMC/QAM transmultiplexer impulse response using the PHYDYAS filter

Table 6 FBMC/QAM transmultiplexer impulse response using the IOTA filter

The main principle of the PaIC/Viterbi receiver remains the same as for conventional FBMC presented in Section 4. However, the configuration of the PaIC/Viterbi receiver has to be adapted to the new FBMC/QAM context. Indeed, we have seen that the function f ₀ that matches the Viterbi detector must contain the largest coefficients $\acute {\Gamma }_{\Delta k,\Delta n}$. According to the transmultiplexer impulse responses depicted in Tables 5 and 6, the three largest coefficients ({1,±0.239j} for PHYDYAS and {1,±0.4411j} for IOTA) in each table are aligned with the direction of the frequency axis. Therefore, we can select the following three functions $f_{0}^{(i)}$, i=1,2,3 that determine the Viterbi target responses

$$\begin{array}{*{20}l} &f_{0}^{(1)}\left(s_{k,n}\right)=s_{k,n}, \end{array} $$

((23))

$$\begin{array}{*{20}l} &f_{0}^{(2)}\left(s_{k,n},s_{k-1,n}\right)=s_{k-1,n}+\acute{\Gamma}_{-1,0}\times s_{k,n-1}, \end{array} $$

((24))

and

$$\begin{array}{*{20}l} {}f_{0}^{(3)}\left(s_{k+1,n},s_{k,n},s_{k-1,n}\right)= \acute{\Gamma}_{1,0} s_{k+1,n}+s_{k,n}+\acute{\Gamma}_{-1,0} s_{k-1,n}, \end{array} $$

((25))

where $f_{0}^{(1)}$ obviously corresponds to the IIC-ML receiver [26] where the whole interference cancellation is performed. Hence, the advantage for PaIC/Viterbi receivers in FBMC/QAM lies in the fact that the Viterbi algorithms are performed in the frequency axis direction. Whereas in FBMC/OQAM, they are performed in the time axis direction. Consequently, from an implementation point of view, the PaIC/Viterbi receivers are less complicated for implementation in FBMC/QAM than in the conventional FBMC, because only one Viterbi algorithm has to be performed for each one received multicarrier symbol, whereas in FBMC/OQAM, we need to perform a Viterbi detector simultaneously for each subcarrier.

We have seen in Section 4 that the performance of PaIC/Viterbi receivers depend on the power of the residual ISI which is concerned by the interference cancellation and also depend on the values of the spectrum distances d ₀(ε) given by Eq. (16). To calculate the values of d ₀(ε), we need first to determine the normalized autocovariance matrix of the noise R. Since the Viterbi algorithm is performed in the direction of the frequency axis, the noise vector w is then w=[w _0,n ,w _1,n ,…,w _M−1,n ]^T. Hence, the matrix $\mathbf {R}=\mathbb {E}\{\mathbf {ww}^{H}\}/{\sigma _{w}^{2}}$ is only composed by the coefficients $\acute {\Gamma _{p,0}}$, $p\in \mathbb {Z}$ and is given by

$$ {{}\fontsize{7.7pt}{6pt}\selectfont{\begin{aligned} \mathbf{R}&=\frac{1}{{\sigma_{w}^{2}}} \left[\begin{array}{cccc} \mathbb{E}\{w_{0,n}w_{0,n}^{*}\} &\mathbb{E}\{w_{0,n}w_{1,n}^{*}\} &\cdots &\mathbb{E}\{w_{0,n}w_{M-1,n}^{*}\} \\ \mathbb{E}\{w_{1,n}w_{0,n}^{*}\} & \mathbb{E}\{w_{1,n}w_{1,n}^{*}\} &\cdots &\mathbb{E}\{w_{1,n}w_{M-1,n}^{*}\} \\ \vdots &\vdots &\ddots &\vdots \\ \mathbb{E}\{w_{M-1,n}w_{0,n}^{*}\} &\mathbb{E}\{w_{M-1,n}w_{1,n}^{*}\} &\cdots & \mathbb{E}\{w_{M-1,n}w_{M-1,n}^{*}\} \end{array}\right]_{M\times M} \\ &= \left[\begin{array}{ccccc} 1 &\acute{\Gamma}_{1,0} &\acute{\Gamma}_{2,0} &\cdots &\acute{\Gamma}_{M-1,0}\\ \acute{\Gamma}_{-1,0} & 1 &\acute{\Gamma}_{1,0} &\cdots &\acute{\Gamma}_{M-2,0}\\ \acute{\Gamma}_{-2,0} &\acute{\Gamma}_{-1,0} & 1 &\cdots &\acute{\Gamma}_{M-3,0}\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ \acute{\Gamma}_{1-M,0} &\acute{\Gamma}_{2-M,0} &\acute{\Gamma}_{3-M,0} &\cdots &1 \end{array}\right]_{M\times M,} \end{aligned}}} $$

((26))

where M is the number of subcarriers. Then, the values of d ₀(ε) can be calculated for each single error event ε and for each one of the chosen receiver configurations (according to $f_{0}^{(i)}$, i=1,2,3). We remind that the error events are classified according to their spectrum distance d ₀(ε). The error events corresponding to the smallest d ₀(ε) are called first-order error events ε ₁, and those corresponding to the second smallest d ₀(ε) are called second-order error events ε ₂. Tables 7 and 8 give the different values of d ₀(ε) of the first- and second-order error events for each receiver configuration, respectively, for the PHYDYAS and IOTA filters. We also show in the tables the corresponding power of the residual interference $\left (\sigma _{\text {RISI}}^{2}\right)$ and the number of error positions (w(ε)) within the first and second error events.

Table 7 Spectrum distances and RISI power for FBMC/QAM with the PHYDYAS filter

Table 8 Spectrum distances and RISI power for FBMC/QAM with the IOTA filter

It is worth noticing that in the third configuration using the IOTA filter, the values of the two smallest distances are practically the same. Moreover, the first- and second-order error events contain, respectively, 3 and 4 error positions (w(ε ₁)=3 and w(ε ₂)=4), whereas in the other cases, they only contain 1 and 2 errors, respectively. Consequently, even if the residual interference is perfectly removed, the bit error rate (BER) performance would be poor because the most likely error events are those that contain 3 or 4 errors.

Finally, we note that if we assume imperfect channel estimation, the performance of the proposed detectors (for both FBMC/OQAM and FBMC/QAM) will be affected. Indeed, channel estimation error will lead additional noise to the detected signals. However, according to the considered system model given by Eqs. (8) or (10), the channel estimation error is uncorrelated with the intrinsic interference. Hence, since the proposed configurations $f_{0}^{(i)}$, i∈{1,2,3} are only related to the FBMC intrinsic interference, then they will not be changed in the context of imperfect channel estimation. Nevertheless, the metric used in the Viterbi algorithm might be changed to improve the detection performance under imperfect channel knowledge assumption. It is worthwhile to note that a generalized robust (against channel estimation errors) ML decoder that yields near optimal BER performance was proposed in [41].

6 Simulation results

In the following, we provide the simulation results concerning the three configurations of the PaIC/Viterbi receivers for both FBMC/OQAM and FBMC/QAM schemes with the PHYDYAS and IOTA filters. Since the motivation of this work is to address the problem of optimum detection in spatial multiplexing with FBMC, we have considered the simple 2×2 spatial multiplexing scheme. We assume perfect channel knowledge at the receiver side, and the four Rayleigh sub-channels are spatially non-correlated and follow the Veh-A channel model [42]. The parameters of the Veh-A channel are [42]: delays = [0 300 700 1100 1700 2500] ns, powers = [0 −1 −9 −10 −15 −20] dB, and the velocity is v=60 km/h. For the simulation settings, we considered that the sampling period is T _s =100 ns, the carrier frequency is f _c =1 GHz, and the number of subcarriers is M=512. Therefore, the maximum Doppler spread is given by $f_{\mathrm {D}} = \frac {v f_{c}}{c}$, where c=3×10⁸ m/s is the light speed, thus f _D≈55.56 Hz. We should notice that with the above parameters, the approximation considered in Eq. (10) is still valid. The data symbols are offset quadrature phase-shift keying (OQPSK) modulated for FBMC/OQAM, and quadrature phase-shift keying (QPSK) modulated for FBMC/QAM. That is, the number of bits per data symbol is q=2 for both FBMC/OQAM and FBMC/QAM. The system performance is assessed in terms of BER as function of the signal-to-noise ratio (SNR) per bit E _b /N ₀ and is compared to that of the conventional CP-OFDM with ML detector. The CP duration for CP-OFDM is set to T _CP=32T _s =3.2 μs. The considered SNR per bit for FBMC is given by

$$\begin{array}{*{20}l} E_{b}/N_{0} = N_{t}\frac{E_{s}/q}{{\sigma_{n}^{2}}/2} = \frac{2E_{s}}{{\sigma_{n}^{2}}}, \end{array} $$

((27))

where $E_{s} = \mathbb {E}\{\left |s[\!m]\right |^{2}\}$ is the transmitted signal energy in each transmit antenna, N _t =2 is the number of transmit antennas, and ${\sigma _{n}^{2}}$ is the noise power in each receive antenna. As for CP-OFDM, we have

$$\begin{array}{*{20}l} E_{b}/N_{0} = N_{t}\frac{2E_{s}}{q{\sigma_{n}^{2}}}\frac{\text{MT}_{s} + T_{\text{CP}}}{\text{MT}_{s}} = \frac{17E_{s}}{8{\sigma_{n}^{2}}}. \end{array} $$

((28))

Therefore, the performance curves of CP-OFDM exhibit an SNR loss of about 10 log10(17/16)≈0.26 dB.

6.1 FBMC/OQAM

Figure 3 depicts the performance of the MMSE equalizer (which is our tentative detector) and of the PaIC/Viterbi for the three considered configurations in FBMC/OQAM using the PHYDYAS filter. We clearly notice that PaIC/Viterbi-3 exhibits almost the same performance as CP-OFDM/ML, and that the RISI cancellation is effective. Hence, the value of the RISI power given in Table 3 for the third configuration is sufficiently small so that condition 2 is satisfied.

Fig. 3

Performance of PaIC/Viterbi receivers for 2×2 spatial multiplexing using FBMC/OQAM modulation with the PHYDYAS filter

As for the first and second configurations, the performance degradation compared to CP-OFDM/ML begins from about 12 dB. This relatively high degradation is due to the high values of the corresponding RISI powers causing error propagation.

Now, we consider only the third configuration $\left (f_{0}^{(3)}\right)$ since the RISI cancellation is effective, and we assess the BER performance when using M-Algorithm instead of the Viterbi one in order to reduce the complexity. In Fig. 4, we show the obtained performance of the M-Algorithm with two different values of J (J=2 and J=4). We notice that with J=2, we have an SNR loss of about 2.5 dB compared to PaIC/Viterbi-3. This SNR loss is due to the suboptimality of the M-Algorithm when J is small. Moreover, PaIC/M-Algo with J=2 provides a performance worse than the one provided by the tentative detector (MMSE) as long as the SNR is less than 12 dB. However, with J=4, we can observe that PaIC/M-Algo exhibits almost the same performance as PaIC/Viterbi-3 but with much lower algorithm complexity (4 inner states instead of 16).

Fig. 4

Performance of PaIC/M-Algo receiver for 2×2 spatial multiplexing using FBMC/OQAM modulation with the PHYDYAS filter

In the case where the IOTA filter is used, the BER performance curves are plotted in Fig. 5. We remark that no configuration among the three ones can provide the same BER performance as CP-OFDM/ML. We can observe, for all configurations, an SNR loss with respect to CP-OFDM/ML performance in the range where SNR is greater than 15 dB. This bad performance results are explained by the high residual interference powers presented in Table 4 that prevent the convergence to the optimal performance.

Fig. 5

Performance of PaIC/Viterbi receiver for 2×2 spatial multiplexing using FBMC/OQAM modulation with the IOTA filter

6.2 FBMC/QAM

Figure 6 depicts the BER performance of the proposed receivers in FBMC/QAM using the PHYDYAS filter. We notice that all the three PaIC/Viterbi configurations exhibit performance that converge to the CP-OFDM/ML performance. This is because the conditions of effective interference cancellation are fulfilled, namely the RISI powers are sufficiently small, and the first-order error events are separated enough from the other error events by their spectrum distance.

Fig. 6

Performance of PaIC/Viterbi receivers for 2×2 spatial multiplexing using FBMC/QAM modulation with the PHYDYAS filter

In the case where the IOTA filter is used, we can observe in Fig. 7 that PaIC/Viterbi-3 receiver with the IOTA filter suffers from an SNR loss of about 1.2 dB with respect to CP-OFDM/ML. This SNR loss is due to the errors caused by the Viterbi algorithm itself. Indeed, we have seen in Table 8 that the most likely error events are those that contain 3 or 4 errors, and the RISI power is negligible. That is, the BER performance curve is shifted up. In the other hand, since the effect of erroneous tentative decisions are practically negligible because the variance of the residual interference is very small $\left (\sigma _{\text {RISI}}^{2}\approx 0.007\right)$, the tentative detector (MMSE equalizer) is not useful and can be withdrawn.

Fig. 7

Performance of PaIC/Viterbi receivers for 2×2 spatial multiplexing using FBMC/QAM modulation with the IOTA filter

As for PaIC/Viterbi-2 and PaIC/Viterbi-1 receivers, we can notice that with the IOTA filter the BER performance does not reach the CP-OFDM/ML one. We obtain an SNR loss at BER=10⁻² with respect to CP-OFDM/ML of about 2.25 dB for PaIC/Viterbi-2, and 6.25 dB for PaIC/Viterbi-1. Hence, both latter configurations suffer from error propagation and the interference cancellation is not effective. This performance limitation is explained by the fact that the residual interference variances are not sufficiently small; according to Table 8, we have $\sigma _{\text {RISI}}^{2}\approx 0.2$ and $\sigma _{\text {RISI}}^{2}\approx 0.4$ for PaIC/Viterbi-2 and PaIC/Viterbi-1, respectively.

Because the orthogonality condition is lost in FBMC/QAM, we can observe the effect of the interference by comparing the BER-performance of the MMSE equalizer in both cases where the PHYDYAS or IOTA filter is used. We notice in Fig. 6 that the MMSE equalizer reaches the BER value of 10⁻¹ at SNR=15.5 dB, whereas in Fig. 7, we have BER=10⁻¹ at SNR=9.5 dB. This is due to the fact that the BER floor in the IOTA case is much higher than the BER floor in the case where the PHYDYAS filter is used.

7 Conclusions

The intrinsic interference in FBMC is a 2D-ISI in the time-frequency plan. In order to avoid a full 2D-Viterbi detector, we have proposed in this paper to study a trade-off between a whole interference cancellation receiver and a full 2D-Viterbi detector. The proposed receiver is composed by a tentative detector giving decisions which serve to partially cancel the interference, followed by a Viterbi detector matching to the non-cancelled interference. Three configurations were treated. The first one is called PaIC/Viterbi-1 and correspond to the whole interference cancellation. The second one is PaIC/Viterbi-2, where the Viterbi detector matches with the two largest coefficients of the transmultiplexer impulse response. The third one is PaIC/Viterbi-3 and the Viterbi detector matches with the three largest coefficients. We have shown that the performance of the PaIC/Viterbi receivers essentially depends on the power of the residual interference (RISI) which is not concerned by the Viterbi detector. When the RISI power is not sufficiently low, the receivers suffer from error propagation.

We have studied the proposed receivers in both FBMC/OQAM and FBMC/QAM. This latter can offer the best performance in some situation since the global intrinsic interference is reduced. For FBMC/OQAM, we have shown by simulations that only PaIC/Viterbi-3 with the PHYDYAS filter can provide satisfactory performance compared to CP-OFDM/ML one. Whereas for FBMC/QAM, all the three PaIC/Viterbi configurations with the PHYDYAS filter exhibit the same performance as CP-OFDM/ML. However, the interference coefficients of the IOTA filter do not allow any PaIC/Viterbi configuration to reach the optimal BER performance. Indeed, in spite of the lowest RISI power for PaIC/Viterbi-3 with the IOTA filter, the BER performance is limited because of the fact that the most likely error events in the Viterbi detector contain more than 3 errors.

Finally, since the Viterbi detection increases the computational complexity, we can replace the Viterbi detector by another one based on M-Algorithm in order to reduce the receiver complexity without performance deterioration.

As a future work, one can consider extending the present work to the context of imperfect channel state information. As we have aforementioned, in order to minimize the BER performance degradation, we should adapt the Viterbi algorithm and investigate an efficient Viterbi metric capable of achieving near optimal performance.