Joint window and filter optimization for new waveforms in multicarrier systems

Abstract

One of the demands of the next-generation wireless communication systems is being supportive to asynchronous traffic types. In order to meet this demand, many waveform candidates for next-generation wireless communication systems have low out-of-subband emissions (OOSBE) of the transmitted signal, and the popular ones deploy either filtering or windowing at the transmitter. In this paper, a joint windowing and filtering multi-carrier waveform based on a generalized orthogonal frequency division multiplexing (OFDM) system is proposed. A joint window and filter optimization problem to minimize OOSBE is also formulated. The optimization problem is first divided into two solvable subproblems using interior-point methods, and then an iterative algorithm is proposed to obtain the optimal window and filter pair. Simulation results suggest that the proposed joint windowing and filtering waveform has an advantage in suppressing OOSBE or enhancing spectral efficiency, and is more robust to frequency asynchronism compared to waveforms deploying only either filtering or windowing.

1 Introduction

New transmission waveforms beyond orthogonal frequency division multiplexing (OFDM) have been studied to a great extent recently, in order to meet the demands of next-generation communication systems [1]. Many new waveform proposals aim at suppressing sidelobe power or out-of-subband emissions (OOSBE) in generalized OFDM systems, since waveforms with this property may provide advantages such as low interference among asynchronous users accessing adjacent frequencies, spectral efficiency increase through reuse of guard frequencies, and enabling multi-service systems using mixed numerologies [2–7].

Among the many waveform proposals, filter bank multi-carrier (FBMC) systems [8] possess perhaps the best sidelobe suppression performance when their associated pulse-shaping filters are properly designed. However, filters with this great property usually have a long impulse response, typically in the order of several symbol durations. This causes severe inter-symbol interference (ISI) and results in difficulties in receiver designs especially when the signal is transmitted over multi-path channels [9]. With a similar transmitter structure, a windowing-based waveform, commonly known as weighted overlap-and-add (WOLA) [4], has been proposed to suppress sidelobe powers without using long pulse shaping. Nevertheless, an extended guard interval, such as a cyclic prefix (CP), is required for eliminating ISI caused by multi-path channels and the window of WOLA [10, 11], which leads to reduced spectral efficiency. In [12, 13], windowing schemes that require no guard interval extension were proposed to suppress sidelobe powers. However, the receiver structure in [12] causes signal-to-noise ratio (SNR) loss and leads to a degraded data reception performance, whereas the receiver structure in [13] leads to inter-carrier interference (ICI) between subcarriers for arbitrary window coefficients with multi-path fading channels.

In [14–16], precoding-based waveforms were proposed to suppress OOSBE by designing a precoder for transmitting data symbols. Generalized frequency division multiplexing (GFDM) [14] is able to suppress sidelobe power by using a circularly pulse-shaped precoder. In general, the GFDM precoder is represented by a non-unitary matrix[17]. The non-unitariness of such precoding matrix could lead to noise enhancement at the receiver, and thus, it degrades the performance of data reception. Spectrally precoded OFDM [15] and a power leakage-suppressing precoder [16] both possess unitary precoding matrices so that there is no noise enhancement at the receiver. However, the precoding-based waveforms require a number of null subcarriers, called virtual carriers (VCs), to be inserted in each subband, which will degrade spectral efficiency.

Recently, some new filtering-based waveforms were proposed with the objective to shorten the impulse response of the pulse shaping filters while maintaining a similar level of OOSBE suppression [18–25]. These waveforms take advantage of per-subband filtering to obtain a much shorter filter than FBMC, which executes per-subcarrier filtering. Nevertheless, all filtering-based waveforms increase the delay spread of the equivalent channel and inevitably suffer from either a decreased spectral efficiency due to guard interval extension [21] or a receiver with a larger complexity for equalization of ISI effects. To trade a better OOSBE suppression performance, inserting a small number of VCs between adjacent subbands as additional guard bands is also considered [2, 22–24]. However, such insertion also leads to decreased spectral efficiency.

In this paper, a new waveform that jointly utilizes windowing in [12] and filtering at the transmitter is proposed. To suppress OOSBE, a subband-based joint window and filter optimization method is proposed. The proposed waveform using the proposed optimization method has an advantage in OOSBE suppression in contrast with previous filtering-based and windowing-based waveforms. Alternatively, the proposed waveform is more spectrally efficient for suppressing OOSBE to a specific level because it requires less VCs. Simulation results confirm the above advantages of the proposed waveform.

1.1 Related waveforms

The waveforms related to this research, including the aforementioned windowing-based and filtering-based waveforms, are further presented in the following.

Windowing-based waveforms: WOLA deploys non-memoryless windowing at either the transmitter or the receiver [4]. To mitigate OOSBE, WOLA performed at the transmitter is introduced. A Nyquist window (e.g., raised-cosine) is usually adopted by current WOLA systems. However, an extended CP is required for ISI elimination. A waveform using memoryless windowing at the transmitter was proposed in [12]. This waveform has low OOSBE compared to the conventional OFDM and needs no extra CP. The objective of optimizing the window coefficients in [12] is to minimize the power leakage out of a subcarrier. Nevertheless, the receiver that assures orthogonality between subcarriers induces SNR loss caused by noise enhancement. A memoryless windowing-based waveform was also proposed in [13], which adopted a different receiver structure from that in [12]. Specifically, it gives up on assuring orthogonality between subcarriers for multi-path fading channels; thus, it results in ICI and endurable ISI.

Filtering-based waveforms: Universal-filtered multi-carrier (UFMC) [18–20] was first proposed with the idea of using per-subband filtering to suppress OOSBE. Such subband-based filtering incurs mitigated ISI compared to FBMC. A UFMC system with an extended CP was proposed in [21] to further eliminate the ISI. With the similar idea of UFMC, filtered-OFDM (f-OFDM) [22, 23] considers a longer filter (up to a half symbol duration) to trade off even better OOSBE suppression at the expense of some manageable ISI. A UFMC system that employs WOLA-based windowing and specifically targets at low-latency applications has been proposed in [25].

1.2 Contributions

The major contributions of this paper are as follows:

• A per-subband-based waveform that exploits both windowing and filtering operations at the transmitter is proposed. This waveform is suitable for performing joint processing because there are two sets of design parameters. Unlike [25] which utilizes WOLA-based windowing, the windowing structure used in this paper is inspired by that in [12], which helps the CP length to be further reduced in contrast with the previous filtering-based waveforms. The proposed waveform exploits a subband-based filtering and windowing design, whereas a per-subcarrier design is executed in [12]. This design makes the proposed waveform more suitable for multi-subband systems (e.g., in [2]).

• A joint window and filter optimization method is proposed to suppress OOSBE of the proposed waveform. The proposed method enables the proposed waveform to possess advantages in spectral efficiency and OOSBE suppression over the previous filtering-based and windowing-based waveforms. The proposed method also facilitates the proposed waveform to induce much less SNR loss than the windowing-based waveform in [12] and to allow the use of a shorter CP for ISI elimination compared to WOLA and the previous filtering-based waveform to suppress OOSBE to a specific level.

1.3 Organization and notations

The remainder of this paper is organized as follows. Section 2 presents the system model. Section 3 addresses the joint window and filter design problem. The proposed optimization method is then described in Section 4. Simulation results are provided in Section 5, and Section 6 offers a conclusion.

Notations: Boldface lowercase letters represent column vectors; boldface uppercase letters represent matrices. The superscripts (·)^∗, (·)^T, and (·)^H denote the conjugate, transpose, and transpose-conjugate operations, respectively. The expected value operation is denoted as E{·}. The vector d(θ) of an M-length vector $\boldsymbol {\theta } \triangleq [ \ \theta _{1} \ \cdots \ \theta _{M}\ ]^{T}$ is [ cosθ₁ ⋯ cosθ_M sinθ₁ ⋯ sinθ_M ]^T. Operations ∥·∥ and diag(·) denote the vector norm and vector/matrix diagonalization, respectively. Hadamard and Kronecker products between vectors v₁, v₂ are denoted as v₁∘v₂ and v₁⊗v₂, respectively. An identity matrix of size M is denoted as I_M; a zero matrix of size M×N is denoted as 0_M×N. An N-length vector with all of its entries be equal to one is denoted as 1_N; a zero column vector of size M is denoted as 0_M. A column vector v_L(z) of length L is written as v_L(z)=[ 1 z ⋯ z^L−1 ]^T. An M×M circulant matrix C whose first column is [ c₀ c₁ ⋯ c_M−1 ]^T is written as

$$\begin{array}{*{20}l} \mathbf{C} = \left[\begin{array}{cccc} c_{0} & c_{M-1} & \cdots & c_{1} \\ c_{1} & c_{0} & \cdots & c_{2} \\ \vdots & \vdots & \ddots & \vdots \\ c_{M-1} & c_{M-2} & \cdots & c_{0} \end{array} \right]. \end{array} $$

A normalized discrete Fourier transform (DFT) matrix of size M is W_M. The (m,n)th entry of the DFT matrix is $[\textbf {W}_{M}]_{m,n} = \frac {1}{\sqrt {M}}e^{-j\frac {2\pi }{M}(m-1)(n-1)}$. The domain of function f is denoted as dom f. The sets of non-negative and positive vectors of size M are denoted as $\mathsf {R}_{+}^{M}$ and $\mathsf {R}_{++}^{M}$, respectively.

2 System model

2.1 Transmitter structure

Consider a multi-carrier system with M subcarriers. Suppose that a communication device is allocated to a subband consisting of P adjacent subcarriers. Figure 1 depicts the block diagram of the transmitter. Let s_P[ n] denote the transmit symbol vector and assume

$$\mathrm{E}\left\{\mathbf{s}_{P}[\!n] \mathbf{s}_{P}^{H}[\!m]\right\} = \left\{\begin{array}{ll} E_{s} \textbf{I}_{P}, & n=m \\ \boldsymbol{0}_{P \times P}, & \text{else} \end{array} \right. $$

where E_s denotes the symbol energy. Now, the transmit symbol vector is precoded by an IDFT matrix $\textbf {W}_{M}^{H}$, resulting in a vector $\textbf {u}_{M}[\!n] = \textbf {W}_{M}^{H} \textbf {s}[\!n]$ where $\textbf {s}[\!n] = \left [\boldsymbol {0} \ \textbf {s}_{P}^{T}[\!n] \ \boldsymbol {0} \ \right ]^{T}$.

Fig. 1

Block diagram of the transmitter

To eliminate ISI and suppress the OOSBE, the vector u[ n] is obtained by adding a CP of length L to u_M[ n] and multiplying a matrix consisting of window coefficients. Specifically, the windowed signal vector is expressed as u[ n]=diag([w_−L ⋯ w₀ ⋯ w_M−1])Ru_M[ n], where w_m,m=−L,⋯,M−1 are the window coefficients, and matrix R, which represents adding the CP, is written as

$$\begin{array}{*{20}l} \mathbf{R} = \left[\begin{array}{c} \boldsymbol{0}_{L\times (M-L)} \ \ \mathbf{I}_{L} \\ \mathbf{I}_{M} \end{array} \right]. \end{array} $$

Now, define

$$ W\left(e^{j\omega}\right)=\sum_{m=-L}^{M-1}w_{m} e^{-j (m+L)\omega} $$

(1)

as the Fourier transform of the window coefficients, then the power spectral density (PSD) of signal u[ n], after parallel-to-serial (P/S) conversion of vector u[ n], is [26]

$$ \mathcal{S}_{u}\left(e^{j\omega}\right) = \frac{E_{s}}{N}\sum_{k\in \mathcal{K}} \left|W\left(e^{j(\omega-\Omega_{k})}\right)\right|^{2} $$

(2)

where N=M+L, $\Omega _{k} = \frac {2\pi k}{M}$ and $\mathcal {K}$ denotes the set of subcarriers that are actually used in a subband. Let K denote the index of the subcarrier at the left of the allocated subband, then

$$\begin{array}{*{20}l} \mathcal{K} \triangleq \{ K, K+1, \cdots, K+P-1\}. \end{array} $$

In conventional OFDM systems, the window coefficients are usually chosen as w_m=1,∀m=−L,⋯,M−1 referred to as a rectangular window. In this paper, the coefficients are considered to be more general, with a constraint that the power of the signal using any arbitrary coefficients is equal to that using the rectangular window. That is,

$$ \sum_{m = -L}^{M-1} |w_{m}|^{2} = N. $$

(3)

Note that the slope of sidelobe power decay of conventional OFDM signals is approximately proportional to the frequency distance to the center of the mainlobe [27]. The slow decay of sidelobe power in conventional OFDM systems results in large OOSBE.

An effective way of suppressing OOSBE is deploying per-subband filtering at the transmitter, which was first proposed in [18]. Subsequently, the windowed signal u[ n] is filtered by an FIR filter F(z), which is defined as

$$\begin{array}{*{20}l} F(z) = \sum_{n=0}^{L_{f}-1} f[\!n]z^{-n} = \mathbf{v}_{L_{f}}^{T}\left(z^{-1}\right) \mathbf{f} \end{array} $$

where f[ n] is the impulse response and L_f denotes the filter length, which is assumed to be much smaller than the number of subcarriers, i.e., L_f≪M, and f=[ f[ 0] f[ 1] ⋯ f[L_f−1]]^T. Recall that the definition of $\textbf {v}_{L_{f}}(z)$ is given in Section 1.3. Then, the PSD of the transmitted signal x[ n] is written as

$$ \mathcal{S}_{x}\left(e^{j\omega}\right) = \mathcal{S}_{u}\left(e^{j\omega}\right) \left|F\left(e^{j\omega}\right)\right|^{2}. $$

(4)

The filter function f[ n],n=0,⋯,L_f−1, is chosen such that the expected value of the energy of the filtered signal x[ n] is equivalent to that of the windowed signal u[ n]. Based on Parseval’s relation and (3),

$$ \int_{-\pi}^{\pi} \mathcal{S}_{x}\left(e^{j\omega}\right) \frac{\mathrm{d}\omega}{2\pi} = \int_{-\pi}^{\pi} \mathcal{S}_{u}\left(e^{j\omega}\right) \frac{\mathrm{d}\omega}{2\pi} = P E_{s}. $$

(5)

2.2 Effective channel

As illustrated in Fig. 1, the filtered signal x[ n] is sent over the channel H(z). It is assumed that H(z) is an FIR channel with a maximum order L_h, i.e.,

$$ H(z) = \sum_{n=0}^{L_{h}}h[\!n] z^{-n} $$

(6)

where h[ n] denotes the impulse response of the FIR channel. By assuming that the transmitter and the receiver are perfectly synchronized, the signal u[ n] can be viewed as passing through an effective channel, whose impulse response is written as

$$\begin{array}{*{20}l} c[\!n] = \sum_{l=0}^{L_{f}-1}f[\!l]h[\!n-l], n = 0, 1, \cdots, L_{c}-1 \end{array} $$

where L_c=L_f+L_h is the length of the effective channel. Here, the CP length is assumed to be no smaller than the effective channel length, i.e., L≥L_c, so that the ISI can be eliminated at the receiver [21].

2.3 Receiver structure

The modified zero-forcing receiver presented in [12] is adopted. Specifically, channel equalization in this paper is only performed on the subcarriers in the allocated subband, rather than on all the subcarriers, as presented in [12]. The block diagram of the receiver is depicted in Fig. 2. As suggested in [21], receiver filtering in the original filtering-based waveform is not considered here in order to scale down the complexity of the receiver.

Fig. 2

Block diagram of the receiver

The received signal after serial-to-parallel (S/P) and CP removal is multiplied by the DFT matrix W_M, and then channel equalization for signals on the used subcarriers is performed by multiplying the inverse of the diagonal matrix $\boldsymbol {\Lambda }(\mathcal {K})$. After that, the equalized signal is multiplied by the de-windowing matrix $\textbf {F}(\mathcal {K})$ in order to restore the orthogonality between subcarriers in the subband. In [12], it has been proven that the de-windowing matrix $\textbf {F}(\mathcal {K})$ is independent of the effective channel if the window satisfies the cyclic-prefixed property: w_m=w_m+M,m=−1,−2,⋯,−L. Based on this property, the windowed signal vector u[ n] is mathematically equivalent to:

$$ \mathbf{u}[\!n] = \mathbf{R} \mathbf{D} \mathbf{u}_{M}[\!n] $$

(7)

where D=diag(w) with $\textbf {w} \triangleq \left [\ w_{0} \ w_{1} \ \cdots \ w_{M-1} \ \right ]^{T}$. Without regarding the signals of users allocated to other subbands, it can be shown that the vector y_M[n], after serial-to-parallel conversion and CP removal, is

$$ \mathbf{y}_{M}[n] = \mathbf{C} \mathbf{D} \mathbf{u}_{M}[n] + \mathbf{e}_{M}[n] $$

(8)

where C is an M×M circulant matrix whose first column is $\left [c[0] \ c[1] \ \cdots \ c[L_{c}-1] \ \boldsymbol {0}_{(M-L_{c}-1)} \right ]^{T}$ and $\textbf {e}_{M}[\!n] \sim CN\left (\boldsymbol {0}_{M}, \sigma _{e}^{2} \textbf {I}_{M}\right)$ is the additive white Gaussian noise vector [28]. The circulant matrix can be diagonalized by the DFT matrix:

$$ \mathbf{C} = \mathbf{W}_{M}^{H} \mathbf{\Lambda} \mathbf{W}_{M} $$

(9)

where $\mathbf {\Lambda } = \text {diag}\left (\left [\ C\left (e^{j0}\right) \ C\left (e^{j\frac {2\pi }{M}}\right) \ \cdots \ C\left (e^{j\frac {2\pi }{M}(M-1)}\right) \ \right ]\right)$ with C(e^jω) denoting the effective channel frequency response. Since the user is allocated to the subband consisting of subcarriers in $\mathcal {K}$, equalization for the effective channel is only performed on the allocated subcarriers. Assuming that the receiver has perfect knowledge of effective channel responses of the allocated subcarriers, the equalized vector z_P[ n] is expressed as

$$ \mathbf{z}_{P}[\!n] = \boldsymbol{\Lambda}^{-1}(\mathcal{K}) \mathbf{E}(\mathcal{K}) \mathbf{W}_{M} \mathbf{y}_{M}[\!n] $$

(10)

where $\textbf {E}(\mathcal {K}) = [\!\begin {array}{ccc}\boldsymbol {0}_{P\times (K-1)} & \textbf {I}_{P} & \boldsymbol {0}_{P\times (M-P-K+1)} \end {array}]$, and $\boldsymbol {\Lambda }(\mathcal {K})$ is the P×P diagonal matrix whose diagonal elements are the effective channel frequency responses of subcarriers in $\mathcal {K}$. That is, the diagonal matrix is written as $\boldsymbol {\Lambda }(\mathcal {K}) = \text {diag}\left (\left [ \ C\left (e^{j\frac {2\pi }{M}K}\right) \ C\left (e^{j\frac {2\pi }{M}(K+1)}\right) \ \cdots \ C\left (e^{j\frac {2\pi }{M}(K+P-1)}\right)\right ] \right)$.

After equalization, to restore the orthogonality between subcarriers in the subband, de-windowing is performed so that the decoded symbol vector $\hat {\mathbf {s}}_{P}[\!n]$ is written as

$$ \hat{\mathbf{s}}_{P}[\!n] = \mathbf{F}(\mathcal{K}) \mathbf{z}_{P}[\!n]. $$

(11)

where $\textbf {F}(\mathcal {K}) \triangleq \textbf {E}(\mathcal {K})\textbf {F}\textbf {E}^{T}(\mathcal {K})$. The matrix F is designed in a zero-forcing sense, which can be interpreted as transforming the equalized signal into a time-domain signal, by multiplying a diagonal matrix whose diagonal elements are the inverse of window coefficients, and then taking DFT on the product [12], i.e.,

$$ \mathbf{F} = \mathbf{W}_{M} \mathbf{D}^{-1} \mathbf{W}_{M}^{H}. $$

(12)

Such de-windowing operation will cause SNR loss at the receiver with an arbitrary w other than a constant-modulus window, e.g., w=1_M.

A different receiver structure that adopts de-windowing before channel equalization was proposed in [13]. This structure is not considered here because the adopted one assures orthogonality between subcarriers while data reception is executed at the receiver, and ISI is eliminated given that L≥L_c. Moreover, the adopted structure is more suitable for performing window optimization because SNR loss caused by noise enhancement can be quantified explicitly, which will be presented in Section 2.4.

2.4 Noise analysis

In this subsection, the SNR loss caused by de-windowing is derived, which plays an important role in window optimization. The derived SNR loss can be approximated as the same as that quantified in [12]. By substituting (8), (9), and $\textbf {F}(\mathcal {K})$ in (11) into (10), the correlation matrix of vector z_P[ n], after CP removal, DFT, and channel equalization, is written as

$$\begin{array}{*{20}l} \mathbf{R}_{z} &= E\left\{\mathbf{z}_{P}[\!n]\mathbf{z}_{P}^{H}[\!n]\right\} \\ &= \mathbf{E}(\mathcal{K}) \mathbf{W}_{M} \mathbf{D} E\left\{\mathbf{u}_{M}[\!n] \mathbf{u}_{M}^{H}[\!n]\right\} \mathbf{D} \mathbf{W}_{M}^{H} \mathbf{E}^{T}(\mathcal{K}) + \mathbf{\Sigma}_{z} \end{array} $$

where $\mathbf {\Sigma }_{z} = \text {diag}\left (\left [ \ \sigma _{z_{1}}^{2} \ \sigma _{z_{2}}^{2} \ \cdots \ \sigma _{z_{P}}^{2}\right ]\right)$ is the correlation matrix of noise after DFT and channel equalization. The ith diagonal element of Σ_z is written as

$$ \sigma_{z_{i}}^{2} = \frac{\sigma_{e}^{2}}{\left| C\left(e^{j\frac{2\pi}{M}(i+K-1)}\right) \right|^{2}} = \frac{\sigma_{i}^{2}}{\left| F\left(e^{j\frac{2\pi}{M}(i+K-1)}\right) \right|^{2}} $$

(13)

where $\sigma _{i}^{2} \triangleq \sigma _{e}^{2}/ | H\left (e^{j\frac {2\pi }{M}(i+K-1)}\right) |^{2}$ is the resulting noise variance of the conventional OFDM system. After de-windowing is performed as in (11), the correlation matrix of $\hat {\textbf {s}}_{P}[\!n]$ is written as $\textbf {R}_{\hat {s}} = E_{s}\textbf {I}_{P} + \textbf {F}(\mathcal {K}) \mathbf {\Sigma }_{z} \textbf {F}^{H}(\mathcal {K}).$ The output noise power of an entry in $\hat {s}_{P}[n]$ is the corresponding diagonal element of the latter term. After the de-windowing operation in (12), the total output noise power is written as

$$ \mathcal{N}_{\text{WF}} = \sum_{i=1}^{P}\sigma_{z_{i}}^{2} \frac{1}{M} \sum_{m=0}^{M-1} \frac{1}{|w_{m}|^{2}}. $$

(14)

As a special case of the proposed waveform, filtering-based waveforms apply rectangular windowing. The resulting total output noise power is $\mathcal {N}_{\mathrm {F}} = \sum _{i=1}^{P} \sigma _{z_{i}}^{2}$, which is obtained by substituting w_m=1,∀m∈{0,⋯,M−1} into (14). By letting $F\left (e^{j\frac {2\pi }{M}(i+K-1)}\right) = 1, \forall i \in \{ 1, \cdots, P \}$ and adopting rectangular windowing, the total output noise of the conventional OFDM system is obtained as $\mathcal {N}_{\mathrm {C}} = \sum _{i=1}^{P} \sigma _{i}^{2}$.

To quantify the noise enhancement effect caused by de-windowing at the receiver compared to the conventional OFDM system, the SNR loss is defined as

$$ \sigma(\mathbf{w}) = \frac{\mathcal{N}_{\text{WF}}}{\mathcal{N}_{\mathrm{C}}} = \frac{\mathcal{N}_{\text{WF}}/\mathcal{N}_{\mathrm{F}}}{\mathcal{N}_{\mathrm{C}}/\mathcal{N}_{\mathrm{F}}} = \frac{1}{M} \sum_{m=0}^{M-1} \frac{1}{|w_{m}|^{2}} \frac{\mathcal{N}_{\mathrm{F}}}{\mathcal{N}_{\mathrm{C}}}. $$

(15)

As suggested in [20], a large variation of $|F\left (e^{j\Omega _{k}}\right)|^{2}, k \in \mathcal {K}$ will degrade the bit error rate (BER) performance. Therefore, the passband ripple of transmit filtering is constrained tightly so that $|F(e^{j\Omega _{k}})|^{2} \approx 1, \forall k \in \mathcal {K}$. Accordingly, the ratio of total noise power of filtering-based waveforms to the total noise power of the conventional OFDM is approximated as

$$ \mathcal{R} = \frac{\mathcal{N}_{\mathrm{F}}}{\mathcal{N}_{\mathrm{C}}} \approx 1 $$

(16)

since $\sigma _{i}^{2} \approx \sigma _{z_{i}}^{2}, i = 1, \cdots, P$. As a result, the SNR loss in (15) is approximated as

$$ \sigma(\mathbf{w}) \approx \frac{1}{M} \sum_{m=0}^{M-1} \frac{1}{|w_{m}|^{2}}. $$

(17)

In the sequel, it is assumed that the approximation in (16) holds given a tight passband ripple constraint. The reasons are that (i) this approximation is accurate for most of the channel responses and (ii) the approximation error does not affect the BER performance of the proposed waveform, as will be shown later in Sections 5.3 and 5.7.

3 Problem statement

The goal of this paper is to jointly design the filter f and the window w so that the OOSBE can be suppressed with a controllable SNR loss, and the passband ripple caused by filtering is restrained. The design is based on a minimax criterion. In [29], it has been pointed out that a wide transition bandwidth (TBW) is beneficial to suppressing stopband magnitude for minimax-based filter design with specified filter length and passband ripple. This phenomenon suggests that the OOSBE suppression performance of a filtering-based waveform is limited with a fixed number of VCs, filter length, and passband ripple. Here, windowing is introduced to further improve the OOSBE suppression performance over the filtering-based waveforms.

Suppose a user is allocated with a subband $\left (\omega _{\mathrm {p}}^{(l)}, \omega _{\mathrm {p}}^{(r)}\right)$. Then, the specification of the transmit PSD is illustrated in Fig. 3. Let the stopband and the passband of the filter and the window be defined as $B_{\mathrm {s}} \triangleq \left [-\pi, \omega _{\mathrm {s}}^{(l)}\right ] \cup [\omega _{\mathrm {s}}^{(r)}, \pi)$ and $B_{\mathrm {p}} \triangleq \left (\omega _{\mathrm {p}}^{(l)}, \omega _{\mathrm {p}}^{(r)}\right)$, respectively. Neighboring subbands allocated to other users lie in the stopband such that the ICI between users can be mitigated. Define $B_{\mathrm {t}} \triangleq \left (\omega _{\mathrm {s}}^{(l)}, \omega _{\mathrm {p}}^{(l)}] \cup [\omega _{\mathrm {p}}^{(r)}, \omega _{\mathrm {s}}^{(r)}\right)$ as the transition band. The subcarriers in the transition band are treated as VCs. Let η denote the one-side normalized TBW, which is expressed as half the TBW over the subcarrier spacing (i.e., 2π/M), then the set of left and the right edges of the stopband are

$$ \left(\omega_{\mathrm{s}}^{(l)}, \omega_{\mathrm{s}}^{(r)}\right) = \left(\omega_{\mathrm{p}}^{(l)} - \frac{(2\eta+1)\pi}{M}, \omega_{\mathrm{p}}^{(r)} + \frac{(2\eta+1)\pi}{M}\right), $$

(18)

Fig. 3

An illustration of problem statement

where $\left (\omega _{\mathrm {p}}^{(l)}, \omega _{\mathrm {p}}^{(r)}\right)=(\omega _{\mathrm {c}} - P\pi /M, \omega _{\mathrm {c}} + P\pi /M)$. For convenience, η can be thought of as the number of VCs on one side of the transition band.

Our design objective is to minimize the maximum (weighted) PSD in the stopband. The minimization is subject to constraints of SNR loss and passband ripple. In addition, for fair comparison, the signals after windowing and filtering should not have any power amplification effect. Therefore, the constraints (3) and (5) should be satisfied. Based on the above requirements, the optimization problem is expressed as

$$\begin{array}{*{20}l} \underset{\mathbf{w}, \mathbf{f}} \min &\ \underset{\omega \in B_{\mathrm{s}}}\max \ G\left(e^{j\omega}\right) \mathcal{S}_{x}\left(e^{j\omega}\right) \end{array} $$

(19a)

$$\begin{array}{*{20}l}[.5pt] \text{subject to} &\ 1/\alpha \leq |F\left(e^{j\Omega_{k}}\right)|^{2} \leq \alpha, \forall k \in \mathcal{K} \end{array} $$

(19b)

$$\begin{array}{*{20}l}[.5pt] &\int_{-\pi}^{\pi} \mathcal{S}_{x}\left(e^{j\omega}\right) \frac{\mathrm{d}\omega}{2\pi} = \int_{-\pi}^{\pi} \mathcal{S}_{u}\left(e^{j\omega}\right) \frac{\mathrm{d}\omega}{2\pi} \end{array} $$

(19c)

$$\begin{array}{*{20}l} &\ \sigma(\mathbf{w}) \leq \gamma \end{array} $$

(19d)

$$\begin{array}{*{20}l}[.5pt] &\ \sum_{m = -L}^{M-1} |w_{m}|^{2} = N \end{array} $$

(19e)

where α and γ denote controllable parameters of passband ripple and SNR loss, respectively, and G(e^jω) is the weighting function of the stopband in order to meet a specified spectrum mask restriction. For simplicity, it is assumed that G(e^jω)=1,∀ω∈B_s in the following derivations. When a more general G(e^jω) is considered, however, it may not impose too much difficulty in generalizing the derivations.

Here, problem (19) is formulated as a more compact form. Based on the cyclic-prefix window property, the Fourier transform of window coefficients in (1) can be written as $W\left (e^{j\omega }\right) = \textbf {v}_{N}^{H}\left (e^{j\omega }\right) \textbf {R} \textbf {w}$. Consequently, the PSD of signal u[ n] is expressed as a quadratic form:

$$ \mathcal{S}_{u}\left(e^{j\omega}\right) = \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w} $$

(20)

where the positive semidefinite matrix B(e^jω) is

$$ \mathbf{B}\left(e^{j\omega}\right) = \frac{E_{s}}{N} \mathbf{R}^{T} \sum_{k\in\mathcal{K}} \mathbf{v}_{N}\left(e^{j(\omega-\Omega_{k})}\right) \mathbf{v}_{N}^{H}\left(e^{j(\omega-\Omega_{k})}\right)\mathbf{R} $$

(21)

With this formulation, the original problem (19) is written as

$$\begin{array}{*{20}l} \underset{\mathbf{w}, \mathbf{f}} \min &\ \underset{\omega \in B_{\mathrm{s}}}\max \ \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w} \left| \mathbf{v}_{L_{f}}^{H}\left(e^{j\omega}\right) \mathbf{f} \right|^{2} \end{array} $$

(22a)

$$\begin{array}{*{20}l} \text{subject to} &\ 1/\alpha \leq \left| \mathbf{v}_{L_{f}}^{H}\left(e^{j\Omega_{k}}\right) \mathbf{f} \right|^{2} \leq \alpha, \forall k \in \mathcal{K} \end{array} $$

(22b)

$$\begin{array}{*{20}l} &\int_{-\pi}^{\pi} \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w} \left| \mathbf{v}_{L_{f}}^{H}\left(e^{j\omega}\right) \mathbf{f} \right|^{2} \frac{\mathrm{d}\omega}{2\pi} = P E_{s} \end{array} $$

(22c)

$$\begin{array}{*{20}l} &\ \sigma(\mathbf{w}) \leq \gamma \end{array} $$

(22d)

$$\begin{array}{*{20}l} &\ \|\mathbf{R} \mathbf{w} \|^{2} = N. \end{array} $$

(22e)

This problem is nonconvex due to, e.g., constraints (22b), (22c), or (22e) [30]. Therefore, the well-developed convex optimization techniques [30] can not be applied directly to solve this problem. Instead of optimizing w and f simultaneously, a method is proposed to optimize w and f iteratively.

4 Proposed method

In this section, an alternative optimization method to solve problem (19) is proposed. First, the problem (22) is divided into two subproblems, which are (i) optimization of filter: optimize the filter function f, subject to filter constraints with a given window, and (ii) optimization of window: optimize the cyclic-prefixed window coefficients w_m,m=−L,⋯,M−1, subject to window constraints with a given filter. Then, an iterative algorithm is proposed to solve the original optimization problem. The convergence and computational complexity of the proposed method, and the complexity of the transceiver are analyzed.

4.1 Optimization of filter function

The filter design method presented in [20] is used by relaxing the nonconvex problem into a convex one using spectral factorization [31]. The relaxed problem can be solved by using an interior-point method (IPM) in [30]. Given any w that satisfies (22d) and (22e), problem (22) becomes a filter optimization problem written as

$$\begin{array}{*{20}l} \underset{\mathbf{f} }\min &\ \underset{\omega \in B_{\mathrm{s}}}\max \ \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w}\left|\mathbf{v}_{L_{f}}^{H}\left(e^{j\omega}\right) \mathbf{f} \right|^{2} \end{array} $$

(23a)

$$\begin{array}{*{20}l} \text{subject to} &\ 1/\alpha \leq \left|\mathbf{v}_{L_{f}}^{H}\left(e^{j\Omega_{k}}\right) \mathbf{f} \right|^{2} \leq \alpha, k \in \mathcal{K} \end{array} $$

(23b)

$$\begin{array}{*{20}l} &\ \int_{-\pi}^{\pi} \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w} \left| \mathbf{v}_{L_{f}}^{H}\left(e^{j\omega}\right) \mathbf{f} \right|^{2} \frac{\mathrm{d}\omega}{2\pi} = P E_{s}. \end{array} $$

(23c)

This problem is nonconvex due to the lower-bounded constraint (23b) and the equality constraint (23c). This issue is overcome by using a technique presented in [32]. Let the auto-correlation function of f[ n] be defined as

$$\begin{array}{*{20}l} r_{f}(m) = \sum_{n=0}^{L_{f}-1} f[\!n]f^{*}[n+m]. \end{array} $$

Then, the squared magnitude of F(e^jω) can be written as

$$ |F\left(e^{j\omega}\right)|^{2} = \mathbf{a}^{T}\left(e^{j\omega}\right)\mathbf{r}_{f} \geq 0 $$

(24)

where r_f=[r_f(−L_f+1) ⋯ r_f(0) ⋯ r_f(L_f−1)]^T and

$$ \mathbf{a}\left(e^{j\omega}\right) = \left[e^{j\left(L_{f}-1\right)\omega} \ \cdots \ 1 \ \cdots \ e^{-j\left(L_{f}-1\right)\omega} \right]^{T}. $$

(25)

Note that r_f(m) is conjugate-symmetric, i.e., $r_{f}(m) = r_{f}^{*}(-m)$. With (24), the PSD in (4) can be rewritten as

$$ \mathcal{S}_{x}\left(e^{j\omega}\right) = \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w} \mathbf{a}^{T}\left(e^{j\omega}\right) \mathbf{r}_{f}. $$

(26)

With this equation and some manipulations, the problem (23) is modified as

$$\begin{array}{*{20}l} \underset{\mathbf{r}_{f}}\min &\ \underset{\omega \in B_{\mathrm{s}}}\max \ \mathbf{b}^{T}\left(e^{j\omega}\right)\mathbf{r}_{f} \end{array} $$

(27a)

$$\begin{array}{*{20}l} \mathrm{subject \ to} &\ 1/\alpha \leq \mathbf{a}^{T}\left(e^{j\Omega_{k}}\right)\mathbf{r}_{f} \leq \alpha, k \in \mathcal{K} \end{array} $$

(27b)

$$\begin{array}{*{20}l} &\ \mathbf{q}^{T} \mathbf{r}_{f} = PN \end{array} $$

(27c)

$$\begin{array}{*{20}l} &\ \mathbf{a}^{T}\left(e^{j\omega}\right)\mathbf{r}_{f} \geq 0, \forall \omega \in [-\pi, \pi) \end{array} $$

(27d)

$$\begin{array}{*{20}l} &\ r_{f}(m) = r_{f}^{*}(-m), m = 0, 1, \cdots, L_{f}-1 \end{array} $$

(27e)

where b(e^jω)=w^HB(e^jω)wa^T(e^jω) and the vector q is given by [q]_i=w^HR^TQ_iRw with

$$ [\mathbf{Q}_{i}]_{m,n} =\left\{\begin{array}{ll} \sum_{k\in \mathcal{K}}e^{j(n-m)\Omega_{k}}, & m-n+L_{f}-i=0 \\ 0, & \text{else}. \end{array}\right. $$

(28)

Details of the derivation of constraint (27c) from (23c) are described in Appendix 1. The problem (27) is semi-infinite, which means that there are infinite number of constraints. A straightforward way of relaxing this problem is to discretize ω by sampling a finite set of frequencies in [−π,π) such that

$$ -\pi \leq \check{\omega}_{0} < \cdots < \check{\omega}_{J-1} < \pi $$

(29)

where J denotes the number of samples. This paper adopts uniformly spaced sampling, i.e., $\check {\omega }_{i} = -\pi +2\pi i/J, i = 0, 1, \cdots, J-1$. A previous work [33] recommended that J≈30L_f is sufficiently large to approximate the problem (27). With this discretization, the filter optimization problem (27) is modified as

$$\begin{array}{*{20}l} \underset{\mathbf{r}_{f}}\min &\ \underset{\check{\omega}_{i} \in \check{B}_{s}}\max \ \mathbf{b}^{T}\left(e^{j\check{\omega}_{i}}\right)\mathbf{r}_{f} \end{array} $$

(30a)

$$\begin{array}{*{20}l} \mathrm{subject \ to} &\ 1/\alpha \leq \mathbf{a}^{T}\left(e^{j\Omega_{k}}\right)\mathbf{r}_{f} \leq \alpha, k \in \mathcal{K} \end{array} $$

(30b)

$$\begin{array}{*{20}l} &\ \mathbf{q}^{T} \mathbf{r}_{f} = PN \end{array} $$

(30c)

$$\begin{array}{*{20}l} &\ \mathbf{a}^{T}\left(e^{j\check{\omega}_{i}}\right)\mathbf{r}_{f} \geq 0, \forall \check{\omega}_{i} \in \check{B}_{\mathrm{a}} \end{array} $$

(30d)

$$\begin{array}{*{20}l} &\ r_{f}(m) = r_{f}^{*}(-m), m = 0, 1, \cdots, L_{f}-1 \end{array} $$

(30e)

where $\check {B}_{\mathrm {s}} = \left \{\check {\omega }_{i} \big | \check {\omega }_{i} = -\pi +{2\pi i}/{J}, i = 0, 1, \cdots, J-1,\right. \left. \text {and} \ \check {\omega }_{i} \in B_{\mathrm {s}}\right \}$ and $\check {B}_{\mathrm {a}} = \left \{\check {\omega }_{i} \big |\check {\omega }_{i} = -\pi +{2\pi i}/{J}, i = 0,\right. \left. 1, \cdots, J-1 \right \}$. This problem can be solved efficiently by existing toolboxes (e.g., cvx [34]). To obtain f from the optimal r_f, spectral factorization in [31] can be applied using an efficient implementation based on inverse fast Fourier transform (IFFT) and fast Fourier transform (FFT).

4.2 Optimization of window coefficients

A per-subband optimization method is proposed that minimizes the maximum stopband PSD subject to window constraints in (22), whereas per-subcarrier optimization was proposed in [12]. Specifically, the window coefficients are restricted as real, and then the window optimization problem is relaxed as a convex one. By adopting the technique presented in Section 4.1, problem (22) is reformulated as

$$\begin{array}{*{20}l} \underset{\mathbf{w}, \mathbf{r}_{f}} \min &\ \underset{\omega \in B_{\mathrm{s}}}\max \ \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w} \mathbf{a}^{T}\left(e^{j\omega}\right) \mathbf{r}_{f} \end{array} $$

(31a)

$$\begin{array}{*{20}l} \text{subject to} &\ 1/\alpha \leq \mathbf{a}^{T}\left(e^{j\Omega_{k}}\right) \mathbf{r}_{f} \leq \alpha, \forall k \in \mathcal{K} \end{array} $$

(31b)

$$\begin{array}{*{20}l} &\int_{-\pi}^{\pi} \mathbf{w}^{H} \mathbf{B}\left(e^{j\omega}\right) \mathbf{w} \mathbf{a}^{T}\left(e^{j\omega}\right) \mathbf{r}_{f} \frac{\mathrm{d}\omega}{2\pi} = P E_{s} \end{array} $$

(31c)

$$\begin{array}{*{20}l} &\ \mathbf{a}^{T}\left(e^{j\omega}\right)\mathbf{r}_{f} \geq 0, \forall \omega \in [-\pi, \pi) \end{array} $$

(31d)

$$\begin{array}{*{20}l} &\ r_{f}(m) = r_{f}^{*}(-m), m = 0, 1, \cdots, L_{f}-1 \end{array} $$

(31e)

$$\begin{array}{*{20}l} &\ \sigma(\mathbf{w}) \leq \gamma \end{array} $$

(31f)

$$\begin{array}{*{20}l} &\ \|\mathbf{R} \mathbf{w} \|^{2} = N. \end{array} $$

(31g)

Given any fixed r_f that satisfies (31b), (31d), and (31e). The constraint (31c) is removed and the problem (22) is rewritten as

$$\begin{array}{*{20}l} \underset{\mathbf{w} }\min &\ \underset{\omega \in B_{\mathrm{s}}} \max \ \mathbf{w}^{H} \mathbf{U}\left(e^{j\omega}\right) \mathbf{w} \end{array} $$

(32a)

$$\begin{array}{*{20}l} \text{subject to} &\ \sigma(\mathbf{w}) \leq \gamma \end{array} $$

(32b)

$$\begin{array}{*{20}l} &\ \|\mathbf{R} \mathbf{w} \|^{2} = N. \end{array} $$

(32c)

where U(e^jω)=B(e^jω)a^T(e^jω)r_f. The reasons that the constraint (31c) is removed are that (31c) is nonconvex in w in general, and the relaxation does not prevent us from finding the optimal w and f for problem (22), as will be explained in Section 4.3.

In the literature, a common way of dealing with a complex value optimization problem is to reformulate it as a real-valued one via separating the real part and imaginary part of the parameters (e.g., [35, 36]). Define $\hat {\textbf {w}} = \left [\text {Re}\left \{ \textbf {w}^{T} \right \} \ \text {Im}\left \{ \textbf {w}^{T} \right \}\right ]^{T},$ then the SNR loss in (32b) is expressed as a function of $\hat {\textbf {w}}$:

$$ \hat{\sigma}(\hat{\textbf{w}}) = \frac{1}{M} \sum_{m=1}^{M} \frac{1}{\hat{w}_{m}^{2} + \hat{w}_{m+M}^{2} } $$

(33)

where $\hat {w}_{m} \triangleq [\! \hat {\textbf {w}}]_{m}$. Subsequently, problem (32) is reformulated via separating the real and imaginary parts of w, and relaxing (32c) as an inequality constraint so that this problem becomes a convex one. Then,

$$\begin{array}{*{20}l} \underset{\hat{\mathbf{w}}} \min &\ \underset{\omega \in B_{\mathrm{s}}} \max \ \hat{\mathbf{w}}^{T} \hat{\mathbf{U}}\left(e^{j\omega}\right) \hat{\mathbf{w}} \end{array} $$

(34a)

$$\begin{array}{*{20}l} \text{subject to} &\ \hat{\sigma}(\hat{\mathbf{w}}) \leq \gamma \end{array} $$

(34b)

$$\begin{array}{*{20}l} &\ \|\hat{\mathbf{R}} \hat{\mathbf{w}} \|^{2} \leq N \end{array} $$

(34c)

where $\hat {\textbf {R}} \triangleq \boldsymbol {1}_{2} \otimes \textbf {R}$ and

$$\begin{array}{*{20}l} \hat{\mathbf{U}}(e^{j\omega}) = \left[\begin{array}{cc} \text{Re}\{ \mathbf{U}\left(e^{j\omega}\right) \} & -\text{Im}\{ \mathbf{U}\left(e^{j\omega}\right) \} \\ \text{Im}\{ \mathbf{U}\left(e^{j\omega}\right) \} & \ \ \text{Re}\{ \mathbf{U}\left(e^{j\omega}\right) \} \end{array} \right]. \end{array} $$

The convexity of this problem depends on constraint (34b), since the objective function and constraint (34b) are convex. The convexity of $\hat {\sigma }(\hat {\textbf {w}}), \text {dom}\,\hat {\sigma } \subseteq \mathsf {R}_{+}^{2M}$ is shown by the following theorem.

Theorem 1

Convexity of $\hat {\sigma } (\hat {\textbf {w}})$:

• The function $\hat {\sigma }(\hat {\textbf {w}})$ in (33) is convex if $\text {dom}\,\hat {\sigma } = \mathsf {D}(\boldsymbol {\theta }) \triangleq \left \{(\boldsymbol {1}_{2} \otimes \boldsymbol {\tau }) \circ \textbf {d}(\boldsymbol {\theta }) \big |\boldsymbol {\tau } \in \mathsf {R}_{++}^{M}\right \}$ for any θ=[θ₁ ⋯ θ_M ]^T that satisfies θ₁,⋯,θ_M∈[−π,π).

• The function $\hat {\sigma }$ is nonconvex if $\mathsf {dom}\,\hat {\sigma } = \mathsf {R}_{++}^{2M}$.

Proof

See Appendix 2. □

Theorem 1 suggests that problem (34) is convex if $\text {dom}\, \hat {\sigma } = \mathsf {D}(\boldsymbol {\theta })$ for any θ₁,⋯,θ_M∈[−π,π). For simplicity, restricting θ=0_M is adopted, in which case the window coefficients are positive-valued. This restriction implies that the original SNR loss function $\sigma (\textbf {w}), \text {dom}\,\sigma = \mathsf {R}_{++}^{M}$, is convex, which is suggested by the proof of Theorem 1(a). By applying frequency discretization as in (29) and (30), problem (34) is approximated as

$$\begin{array}{*{20}l} \underset{\mathbf{w} }\min &\ \underset{\check{\omega}_{i} \in \check{B}_{\mathrm{s}}} \max \ \mathbf{w}^{T} \mathbf{U}\left(e^{j\check{\omega}_{i}}\right) \mathbf{w} \end{array} $$

(35a)

$$\begin{array}{*{20}l} \text{subject to} &\ \sigma(\mathbf{w}) \leq \gamma \end{array} $$

(35b)

$$\begin{array}{*{20}l} &\ \mathbf{w}^{T} \mathbf{R}^{T} \mathbf{R} \mathbf{w} \leq N \end{array} $$

(35c)

where $\textbf {w} \in \mathsf {R}_{++}^{M}$. This problem is a convex optimization problem with finite constraints, which can be solved by using cvx.

4.3 Proposed iterative method for joint optimization

An iterative method that optimizes window and filter is proposed. In an iteration of the proposed method, the window optimization and filter optimization presented in the previous subsections are performed based on given filter function and given window coefficients, respectively. Specifically, window optimization is performed based on the optimal filter’s auto-correlation function obtained in the previous iteration, and then filter optimization is performed based on the optimal window of the current iteration.

A detailed description of the proposed method is shown in Algorithm 1. In each iteration, the window optimization in (35) and the filter optimization (30) are performed sequentially. Let w(t) and r_f(t) denote the optimal solutions of (35) and (30) in iteration t, respectively. The squared differences of the optimal filter’s correlation function vector and optimal window between iterations t and t−1 are, respectively, written as ξ_w(t)=∥w(t)−w(t−1)∥² and ξ_r(t)=∥r_f(t)−r_f(t−1)∥². Since the proposed method is convergent (the proof will be given in Section 4.4), the squared differences ξ_w(t) and ξ_r(t) will decrease as the number of iterations increases. Therefore, the stopping criteria of the proposed iterative algorithm is that both ξ_w(t) and ξ_r(t) are smaller than pre-determined tolerances ε_w and ε_r, respectively. Let $\hat {t}$ denote the termination iteration index. Then, the optimal window is $\textbf {w}(\hat {t})$, and the optimal filter is obtained by performing spectral factorization [31] on $\textbf {r}_{f}(\hat {t})$. For convenience, the starting point is given as $\textbf {r}_{f}(0)= \left [\boldsymbol {0}_{L_{f}-1}^{T} \ 1 \ \boldsymbol {0}_{L_{f}-1}^{T}\right ]^{T}$, which means that no transmit filtering is deployed. Other starting points have been tried, and the resulting performances in OOSBE suppression are close to or worse than that of the current one. Finding the optimal starting point is an open problem for alternative optimization algorithms [37, 38], which is left for our future work.

In any iteration of the algorithm, the pair (w(t),r_f(t)) must satisfy the constraint (27c) after filter optimization. Accordingly, the constraint (31c) is not necessary to be met by the window w(t) based on r_f(t−1). Therefore, it is removed from (31) in window optimization (32).

4.4 Convergence analysis of the proposed algorithm

It is necessary to characterize whether the proposed algorithm is convergent, since it is operated iteratively. To justify that the convergence of the proposed algorithm is guaranteed, it is sufficient to show that the iteration of updating w and r_f converges and the objective is lower-bounded. A similar argument can be found in [37, 38]. In each iteration, the optimal w is first obtained by solving (35). The optimal r_f is then computed by solving (30) based on the determined w. Consequently, the objective function $\max _{\check {\omega }_{i} \in \check {B}_{\mathrm {s}}} \ \textbf {w}^{H} \textbf {B}\left (e^{j\check {\omega }_{i}}\right) \textbf {w} \textbf {a}^{T}\left (e^{j\check {\omega }_{i}}\right) \textbf {r}_{f}$ decreases in each iteration. Clearly, the objective function is lower-bounded because the stopband PSD is non-negative. Therefore, the proposed algorithm is convergent.

4.5 Complexity analysis of the proposed algorithm

The computational complexity of the IPM [30] for problem (30) is expressed as $O\left (L_{f}^{3.5}\right)$ [36]. The complexity order of spectral factorization for obtaining the optimal filter from its auto-correlation function using an FFT-based implementation is O(L_f log2L_f) [31]. Similar to filter optimization, the computational complexity of the IPM for window optimization is expressed as O(M^3.5) [39]. As a result, the computational complexity in each iteration of the proposed iterative algorithm is $O\left (M^{3.5}+ L_{f}^{3.5}+L_{f}\log _{2}L_{f} \right)$, which is equivalent to O(M^3.5) since L_f≪M. Observe that the number of iterations for satisfying ξ_w(t)≤ε_w and ξ_f(t)≤ε_r is independent from M. Therefore, the complexity order of the proposed method can also be expressed as O(M^3.5).

4.6 Complexity analysis of the transceiver

At the transmitter, the number of arithmetic operators of cyclic-prefixed windowing is expressed as O(M) because the windowed signal can be represented by (7); the number of arithmetic operators of filtering is expressed as O(NL_f)=O(N²) since L_f is usually an integer equal to N divided by a power of two and can possibly be reduced to O(N log2N) [40]. De-windowing at the receiver in (11) can be implemented by using FFT, IFFT, and M additional multiplications. Therefore, the computational complexity of receiver with de-windowing is O(3M log2M+M)=O(M log2M), which is of the same order as the receiver of conventional OFDM.

5 Results and discussion

In this section, some numerical results are provided for demonstrating the advantages of the proposed waveform.

5.1 Parameter settings

The parameters used in our simulations are listed below:

• Total number of subcarriers: M=128

• Number of subcarriers in a subband: P=12

• Maximum channel order: L_h=M/8=16

• Filter length: L_f=17

• CP length: L=M/4=32

The values of the channel impulse response in (6) are assumed to be i.i.d. complex Gaussian random variables:

$$\begin{array}{*{20}l} h[\!n]\sim CN(0, 1), n = 0, 1, \cdots, L_{h}. \end{array} $$

There are 10⁷ channel realizations used in our simulations. It is assumed that all subcarriers in the subband are used, i.e., $\mathcal {K} = \{0, 1, \cdots, P-1 \}$. The center frequency of the allocated subband is ω_c=(P−1)π/M. The passband set $\left (\omega _{\mathrm {p}}^{(l)}, \omega _{\mathrm {p}}^{(r)}\right)$, and the pair stopband edges, $\omega _{\mathrm {s}}^{(l)}$ and $\omega _{\mathrm {s}}^{(r)}$, are computed as Eq. (18) with arbitrary normalized TBW η. In the following simulations, the proposed waveform is referred to as “joint windowing and filtering” (joint W-F).

5.2 Convergence of the proposed algorithm

Figure 4 verifies the convergence of the proposed optimization method under different configurations. The normalized TBW used in this figure is η=3.5. The left and right plots demonstrate that the objective function in (22) decreases and converges rapidly in each iteration after window optimization and filter optimization, respectively.

Fig. 4

Convergence of the proposed method

5.3 Approximation of SNR loss

To examine the accuracy of the approximation in (17) with respect to various passband ripple constraints, the histogram of the ratio $\mathcal {R}$ in (16), based on 10⁷ arbitrary channel realizations, is depicted in Fig. 5, and the probability of the ratio falling in a range is shown in Table 1. The normalized TBW used here is also η=3.5. One can observe that the ratios with α=0.3 dB and α=0.5 dB both have peak probabilities very close to $\mathcal {R}=1$. In addition, over 96% of the ratios with α=0.3 dB and α=0.5 dB distribute in the range of (−0.93,1.07). These observations suggest that the approximation is tight for most of the channel realizations, and the approximation error imposes a slight affect on the BER performance, as will be shown in Section 5.7. On the contrary, the ratio with α=1 dB leads to inaccurate approximation since its distribution is scattered in a much wider range and does not have a peak probability close to $\mathcal {R}=1$. In summary, α=0.5 dB is chosen for the following simulations because it results in an accurate SNR loss approximation and has a better OOSBE suppression performance than the case with α=0.3 dB.

Fig. 5

The histogram of the ratio $\mathcal {R}$ in (16)

Table 1 Probability of the ratio $\mathcal {R}$ falls in a range

5.4 Related waveforms for comparison

To highlight the advantages of the proposed joint W-F waveform, the following related waveforms that use either windowing or filtering are introduced for comparison in our simulations.

• Pure filtering: The waveform deploys optimum filtering in [20] at the transmitter with rectangular windowing.

• Pure windowing: The waveform deploys optimum windowing without filtering at the transmitter. The optimal window is obtained through solving (35).

• WOLA at the transmitter: Transmitter windowing of WOLA is performed on some of the CP symbol and an additional cyclic suffix (CS), and then the weighted CS is overlapped with the weighted CP of the next symbol. The transceiver structure in ([26], [Ch. 9]) is adopted for simulations. The raised-cosine (R-C) window is usually used by current WOLA systems, which is expressed as $\textbf {w}_{\text {rc}} = \left [ \begin {array}{ccccccc} p_{0} & \cdots & p_{\nu -1} & \boldsymbol {1}_{N-\nu }^{T} & p_{\nu } & \cdots & p_{2\nu -1} \end {array} \right ]^{T}$ where

  $$\begin{array}{*{20}l} {\kern-.4cm}p_{n} = 0.5 \left(1 -\cos\left(\frac{n+1 }{\nu}\pi \right) \right), n = 0, \cdots, \nu-1, \end{array} $$

  and p_n=1−p_n−ν,n=ν,⋯,2ν−1 [41] with ν denoting the CS length. The receiver of WOLA is the same as that of the conventional OFDM system. To eliminate ISI introduced at the receiver, the CP length of WOLA should satisfy L≤L_h+ν. This paper assumes that ν=L_f for fair comparison.

5.5 Normalized PSD

Here, the PSDs of the transmitted signals modulated by the aforementioned waveforms are demonstrated. The PSD is normalized so that its value on an allocated subband approaches 0 dB. The normalized PSD is written as

$$\begin{array}{*{20}l} \bar{\mathcal{S}}_{x}\left(e^{j\omega}\right) = \frac{1}{N E_{s}} \mathcal{S}_{x}\left(e^{j\omega}\right). \end{array} $$

Figures 6 and 7 depict the normalized PSDs of the proposed and the other waveforms. In Fig. 6, the normalized TBWs of the pure filtering, pure windowing, and the proposed waveforms are η=8.5, which is as wide as that of WOLA. The resulting maximum stopband PSD of the proposed waveform with SNR loss γ=1.2 dB is about −67 dB, which is the best performance compared to the others. With a little bit less SNR loss, the proposed waveform with γ=0.5 dB has a maximum stopband PSD performance of approximately 4 dB-larger than that of the one with γ=1.2 dB, and it still outperforms other waveforms in suppressing maximum stopband PSD. In Fig. 7, the normalized TBWs of the proposed, pure filtering, and pure windowing waveforms are chosen as η=3.5 so that less VCs are required. The proposed waveform using different SNR losses also outperforms other waveforms in suppressing maximum stopband PSD. Although WOLA has good stopband suppression performance for frequencies far from the passband, it will cause larger PSD at some frequencies near the stopband. Therefore, WOLA using R-C window will require more VCs to avoid possible ICI from other subbands compared to the proposed waveform.

Fig. 6

PSD of various types of waveforms (η=8.5)

Fig. 7

PSD of various types of waveforms (η=3.5)

5.6 Bandwidth efficiency

The proposed joint W-F waveform is more spectrally efficient than the other waveforms due to its capability of suppressing OOSBE to a specific level with a narrower guard band. As depicted in Fig. 3, the TBW is treated as the guard band between two adjacent subbands and is shared by two different users that are allocated to these subbands separately. Accordingly, the bandwidth efficiency is defined as

$$ \mathcal{E}(\eta)=\frac{P}{P+\eta}. $$

(36)

Note that η denotes the normalized TBW over the bandwidth of a subcarrier, and P denotes the number of subcarriers in a subband.

The bandwidth efficiency performance of the various waveforms is listed in Table 2. These waveforms possess similar BER performances, which will be shown later. It is observed that the proposed waveform outperforms all the other waveforms at the price of approximately 0.5 dB SNR loss. Moreover, the proposed and the pure filtering waveforms can obtain better performances by relaxing the passband ripple constraint. These observations not only show the superiority of the proposed waveform in suppressing OOSBE spectral-efficiently but also suggest that the proposed optimization method offers more flexibility in designing the transmission waveform.

Table 2 Bandwidth efficiency comparison

5.7 Single-user BER simulations

BER is considered as a metric for the performance of data transmission and reception in our simulations. QPSK or 16QAM without channel coding [42] is adopted as the modulation scheme with gray-coded symbols. Here, the simulations are executed based on the following cases.

• ISI-free case: The CP length is sufficient to eliminate ISI, and the transmitter is perfectly synchronized with the receiver. In this case, the proposed waveform, compared to OFDM, WOLA, and the pure filtering waveform, has a 0.5 dB BER disadvantage, which is roughly equal to the SNR loss. This phenomenon suggests that the proposed waveform trades a slight SNR loss for the advantageous bandwidth efficiency as shown in Table 2.

• ISI-incurring case: The CP length is insufficient so that there is ISI incurred at the receiver, and the transmitter is perfectly synchronized with the receiver. In this case, the proposed waveform is less prone to ISI compared to the pure filtering waveform (and WOLA) because it requires a shorter filter to achieve the same OOSBE suppression level. Therefore, the proposed waveform can use a shorter CP to mitigate ISI, which makes it more spectrally efficient than the pure filtering waveform and WOLA.

The results of the ISI-free case are shown in Figs. 8 and 9. In the simulations, the normalized TBW is set as η=3.5. It is observed that the pure filtering waveform and WOLA have BER performances similar to that of OFDM. The pure windowing waveform has an exact 1.2 dB disadvantage. Although the proposed waveform has an approximately 0.5 dB disadvantage (approximately equal to the SNR loss), it possesses the best OOSBE suppression performance as shown in Figs. 6 and 7. This advantage will be further demonstrated in Section 5.8.

Fig. 8

ISI-free BER performances (QPSK)

Fig. 9

ISI-free BER performances (16QAM)

In Figs. 10 and 11, the results of the ISI-incurring case are demonstrated. Here, the CP length is set as L=23. The filter lengths of the pure filtering and the proposed waveforms are set as L_f=L=23 and L_f=17, respectively. They are chosen this way so that the two waveforms possess a similar maximum stopband PSD of approximately −44 dB. The CS length of WOLA is ν=17, and the resulting OOSBE suppression performance is −26.56 dB, and that of the pure windowing waveform with 1.2 dB SNR loss is −29.83 dB. In both plots, one can observe that the proposed waveform outperforms the pure filtering waveform and WOLA in the presence of ISI at the receiver. Although the BER performance of the pure windowing waveform is not be affected by ISI, its performance in suppressing OOSBE is poor, as demonstrated in Figs. 6 and 7. These results not only suggest that the proposed waveform requires a shorter filter length than the previous filtering-based waveform to achieve similar OOSBE suppression performances, but also indicate that the proposed waveform is less prone to ISI while possessing good capability of suppressing OOSBE compared to WOLA and the pure filtering waveform.

Fig. 10

ISI-incurring BER performances (QPSK)

Fig. 11

ISI-incurring BER performances (16QAM)

5.8 Asynchronous multi-user BER simulations

Here, BER simulations are performed based on an asynchronous uplink scenario with multiple users modulated by the same waveform. The proposed waveform has the best performance especially in this scenario.

As suggested in [1–3], the requirement of relaxed frequency synchronization is based on the necessity of deploying a waveform with reduced OOSBE compared to conventional OFDM. Therefore, frequency asynchronism caused by carrier frequency offsets (CFOs) is introduced in our simulations. In [2], it was pointed out that the CFO of a user can be estimated and compensated at the receiver without requiring any information feedback from the user. Some example methods of estimating CFOs can be found in [43, 44]. However, ICI will still occur because various CFOs of users are unable to be compensated for at the same time.

Suppose there are three users transmitting data symbols simultaneously, as illustrated in Fig. 3. The CFO of the user allocated to subband $\mathcal {K}$ is assumed to be perfectly estimated and compensated for at the receiver, while the other two users are allocated to neighboring subbands and are asynchronous to the receiver, i.e., the CFOs of the neighboring users are not compensated for. Specifically, the two neighboring users, indexed user 1 and user 2, are allocated to subbands $\mathcal {K}_{1} \triangleq \{ -P-\eta, -P-\eta -1, \cdots, -1-\eta \}$ and $\mathcal {K}_{2} \triangleq \{ P+\eta, P+\eta +1, \cdots, 2P+\eta -1 \}$, respectively. The data symbols of the two users are denoted as $\textbf {s}_{P}^{(1)}[\!n]$, and $\textbf {s}_{P}^{(2)}[\!n]$, respectively. The precoded data symbols of user l before adding CP are denoted as $\textbf {u}_{M}^{(l)}[\!n] = \textbf {W}_{M}^{H} \textbf {E}^{T}(\mathcal {K}_{l})\textbf {s}_{P}^{(l)}[\!n], l = 1, 2$. The window coefficients deployed by each user are the same; the filter functions used by user 1 and user 2, denoted f⁽¹⁾[ n] and f⁽²⁾[ n], respectively, are the frequency-shifted versions of the optimal filter function to the center frequencies of the allocated subbands, i.e.,

$$\begin{array}{*{20}l} &f^{(1)}[\!n] = f[\!n] e^{-j\frac{2\pi}{M}(P+\eta)n}, n = 0, 1, \cdots, L_{f}-1 \\ \text{and} \ \ &f^{(2)}[\!n] = f[\!n] e^{j\frac{2\pi}{M}(P+\eta)n}, n = 0, 1, \cdots, L_{f}-1. \end{array} $$

Let h_l[n]∼CN(0,1),n=0,1,⋯,L_h denote the channel impulse response of user l, then the resulting effective channel is c_l[n]=h_l[n]∗f^(l)[n] where ∗ denotes the operation of convolution. For simplicity, the CP length is assumed to be sufficient to eliminate ISI. In the presence of CFO, the received signal vector after serial-to-parallel conversion and CP removal is expressed as

$$\begin{array}{*{20}l} \mathbf{y}_{M}[n] = \mathbf{C} \mathbf{D} {\mathbf{u}_{M}}[n] + {\sum}_{l=1}^{2} \mathbf{\Phi}({\varepsilon}_{l}) \mathbf{C}_{l} \mathbf{D} \mathbf{u}^{(l)}_{M}[n] + \mathbf{e}_{M}[n] \end{array} $$

where $\mathbf {\Phi }({\varepsilon }_{l}) \triangleq \text {diag}\left (\left [1{e}^{j\frac {2\pi }{M}{\varepsilon }_{l}} \cdots {e}^{j\frac {2\pi }{M}(M-1){\varepsilon }_{l}} \right ]\right)$ is the matrix representing phase shift incurred by the CFO, ε_l denotes the normalized CFO of user l, and C_l is the circulant matrix consisting the effective channel’s impulse response of user l [44]. The decoding of s_P[n] is the same as performed in (10) and (11).

In Figs. 12 and 13, the BER simulation results of the user allocated to subband $\mathcal {K}$ are demonstrated. Here, the normalized CFOs are expressed as i.i.d. uniformly distributed random variables in the range of [−ϕ,ϕ], as suggested in [44], where ϕ=0.1. The normalized TBW is η=1. The other parameters are set as presented in Section 5.1. The maximum stopband PSD of the proposed waveform (SNR loss γ=0.5 dB), the pure filtering waveform, and the pure windowing waveform are −28.67 dB, −24.51 dB, and −20.87 dB, respectively. One can observe that the proposed waveform outperforms other waveforms significantly when $E_{b}/\sigma _{e}^{2}$ is larger than 12.5 dB. This is because the proposed waveform has the best capability of suppressing OOSBE so that ICI introduced by CFOs can be greatly mitigated. Due to rectangular windowing and inferior capabilities in OOSBE suppression, the BER performances of OFDM, WOLA, and pure filtering waveform are similar. The pure windowing waveform has the worst BER performance since it will cause severe ICI due to destroying the orthogonality between subcarriers. In regimes of lower transmitting energy, i.e., $E_{b}/\sigma _{e}^{2} \leq 12.5$ dB, the proposed waveform has a slight performance disadvantage compared to WOLA, the pure filtering waveform, and the conventional OFDM. This is because the BER performance of the zero-forcing receiver is more prone to noise rather than ICI in such regimes. These results suggest that the proposed waveform is more capable of suppressing ICI for asynchronous transmissions than other waveforms.

Fig. 12

Multi-user BER performances with CFOs (QPSK)

Fig. 13

Multi-user BER performances with CFOs (16QAM)

5.9 BER simulations with ETU channel model

BER simulations are executed based on the extended typical urban (ETU) channel model [45]. The results are demonstrated in Figs. 14 and 15. In these figures, the sampling frequency is 1.92 MHz, the velocity of the UE is 3 km/h, the number of subcarriers is M=128, and η=1. The filter length of the proposed joint W-F method is L_f=9, and ν=L_f for WOLA in both figures. In Fig. 14, the CP lengths of all the waveforms are set as L=16, and the filter length of the pure filtering one is L_f=9. One can observe that the joint W-F method also has a 0.5 dB performance disadvantage comparing with OFDM, WOLA, and the pure filtering method, which is the same as what was demonstrated in Fig. 9. In Fig. 15, the CP lengths of all the waveforms are set as L=8, and the filter length of the pure filtering one is L_f=9. The filter lengths are chosen so that the pure filtering and the proposed waveforms possess a similar maximum stopband PSD of approximately −31 dB. In this figure, it is also observed that the proposed waveform outperforms the pure filtering waveform and WOLA in the presence of ISI at the receiver, as demonstrated in Fig. 11.

Fig. 14

BER performance with ETU channel model (16QAM, L=16)

Fig. 15

BER performance with ETU channel model (16QAM, L=8)

6 Conclusions

In this paper, a joint windowing and filtering multi-carrier waveform was proposed. An iterative method for joint optimization was also proposed to minimize the OOSBE with controllable SNR loss at the receiver. In contrast with the previous pure filtering and pure windowing waveforms, the proposed waveform has an advantage in suppressing OOSBE to a lower level with similar BER performances. This advantage indicates that the proposed waveform is more spectrally efficient when suppressing OOSBE than the other waveforms. Moreover, the proposed waveform requires a shorter filter length and induces less SNR loss for achieving an OOSBE suppression specification compared to previous pure filtering and pure windowing waveforms, respectively. The detailed comparison of the proposed waveform with OFDM, the pure filtering waveform, the pure windowing waveform, and WOLA is presented in Table 3. Simulation results support the above advantages of the proposed waveform.

Table 3 Comparison of the waveforms

Channel estimation and peak-to-average power ratio (PAPR) reduction remain as open problems. Extending the proposed waveform to a multiple-input-multiple-output system is also of interest.

7 Appendix 1: Derivation of constraint (27c)

By substituting (24) into (4), the PSD of the transmitted signal x[n] is expressed as

$$ \mathcal{S}_{x}\left(e^{j\omega}\right) = \frac{E_{s}}{N} \mathbf{w}^{H} \mathbf{R}^{T} \mathbf{V}\left(e^{j\omega}\right) \mathbf{R} \mathbf{w} \mathbf{a}^{T}\left(e^{j\omega}\right) \mathbf{r}_{f} $$

(37)

where $\textbf {V}(e^{j\omega })= \sum _{k\in \mathcal {K}} \textbf {v}_{N}\left (e^{j(\omega -\Omega _{k})}\right)\textbf {v}_{N}^{H}\left (e^{j(\omega -\Omega _{k})}\right)$. By integrating (37) from π to −π,

$$ {} \int_{-\pi}^{\pi} \mathcal{S}_{x}\left(e^{j\omega}\right) \frac{\mathrm{d}\omega}{2\pi} \,=\, \frac{E_{s}}{N} \underbrace{\mathbf{w}^{H} \mathbf{R}^{T} \int_{-\pi}^{\pi} \mathbf{V}\left(e^{j\omega}\right) \mathbf{R} \mathbf{w} \mathbf{a}^{T}\left(e^{j\omega}\right) \frac{\mathrm{d}\omega}{2\pi}}_{\mathbf{q}^{T}} \mathbf{r}_{f}. $$

(38)

Next, denote the ith entry of vector a(e^jω) as $\left [\textbf {a}(e^{j\omega })\right ]_{i}=e^{j(L_{f}-i)\omega }$. Then, the ith entry of vector q is written as

$$\begin{array}{*{20}l} [\mathbf{q}]_{i} = \mathbf{w}^{H} \mathbf{R}^{T} \underbrace{\int_{-\pi}^{\pi} \mathbf{V}\left(e^{j\omega}\right) e^{j(L_{f}-i)\omega} \frac{\mathrm{d}\omega}{2\pi}}_{\mathbf{Q}_{i}} \mathbf{R} \mathbf{w}. \end{array} $$

Let the (m,n)th entry of matrix V(e^jω) be $\left [ \textbf {V}\left (e^{j\omega }\right) \right ]_{m,n} = \sum _{k\in \mathcal {K}} e^{j(m-n)(\omega -\Omega _{k})}$. Then, the N×N matrix Q_i is given by

$$\begin{array}{*{20}l} [\mathbf{Q}_{i}]_{m,n} &= \sum_{k\in \mathcal{K}} e^{j(n-m)\Omega_{k}} \int_{-\pi}^{\pi} e^{j(m-n+L_{f}-i)\omega} \frac{\mathrm{d}\omega}{2\pi} \\ &=\left\{\begin{array}{ll} \sum_{k\in \mathcal{K}}e^{j(n-m)\Omega_{k}}, & m-n+L_{f}-i=0 \\ 0, & \text{else}. \end{array}\right. \end{array} $$

By substituting Eq. (38) into (5), constraint (27c) is obtained.

8 Appendix 2: Proof of Theorem 1

• First, prove that D(θ) is a convex set. Let $\hat {\textbf {w}}_{1} = (\boldsymbol {1}_{2} \otimes \boldsymbol {\tau }) \circ \textbf {d}(\boldsymbol {\theta })$ and $\hat {\textbf {w}}_{2} = \left (\boldsymbol {1}_{2} \otimes \boldsymbol {\tau }^{\prime }\right) \circ \textbf {d}(\boldsymbol {\theta })$ be two vectors in D(θ), then

  $${} \lambda \hat{\mathbf{w}}_{1} + (1-\lambda) \hat{\mathbf{w}}_{2} = \left\{ \boldsymbol{1}_{2} \otimes[ \lambda \boldsymbol{\tau} + (1-\lambda) \boldsymbol{\tau}^{\prime} ] \right\} \circ \mathbf{d}(\boldsymbol{\theta}) \in \mathsf{D}(\boldsymbol{\theta}) $$

  for all m=1,⋯,M and λ∈[0,1]. Therefore, D(θ) is a convex set.

  Followed by the argument of $\text {dom}\,\hat {\sigma }$, define

  $$\begin{array}{*{20}l} g(\boldsymbol{\tau}) &= \hat{\sigma}(\hat{\mathbf{w}})\big|_{\hat{\mathbf{w}}= (\boldsymbol{1}_{2} \otimes \boldsymbol{\tau}) \circ \mathbf{d}(\boldsymbol{\theta})} \\ &= \frac{1}{M} \sum_{m=1}^{M} \frac{1}{\tau_{m}^{2}}, \text{dom}\,g = \mathsf{R}_{++}^{M}, \end{array} $$

  where $\tau _{m} \triangleq [\boldsymbol {\tau }]_{m}$. This definition implies that $\hat {\sigma }$ is convex if g is convex. The convexity of g is shown by its second-order condition [30]. By taking the second-order derivatives of g, the Hessian is written as

  $$\nabla^{2} g(\boldsymbol{\tau}) = \frac{6}{M} \text{diag}\left(\left[ \ \tau_{1}^{-4} \ \tau_{2}^{-4} \ \cdots \ \tau_{M}^{-4} \right]\right), $$

  which is a positive definite matrix. Therefore, $\hat {\sigma }(\hat {\textbf {w}})$ is convex.

• By taking second-order derivatives of $\hat {\sigma }$, the Hessian is given by

  $$\left[\nabla^{2} \hat{\sigma}(\hat{\mathbf{w}})\right]_{m,n} = \left\{\begin{array}{ll} \frac{8}{M} \hat{w}_{m}^{2}\hat{\sigma}_{m}^{3} - \frac{2}{M}\hat{\sigma}_{m}^{2}, & m = n \\ \frac{8}{M}\hat{w}_{m} \hat{w}_{n} \hat{\sigma}_{m}^{3}, & |m-n|=M \\ 0, & \text{else} \end{array} \right. $$

  where $\hat {\sigma }_{m} = 1/\left (\hat {w}_{m}^{2} + \hat {w}_{m+M}^{2}\right)$ for 1≤m≤M, and $\hat {\sigma }_{m} = 1/\left (\hat {w}_{m}^{2} + \hat {w}_{m-M}^{2}\right)$ for M+1≤m≤2M. The Hessian matrix $\nabla ^{2} \hat {\sigma }(\hat {\textbf {w}})$ is not positive semidefinite. This property can be shown by considering the following example: $\hat {\textbf {w}} = \frac {1}{\sqrt {2}} \boldsymbol {1}_{2M}$. The resulting minimum eigenvalue of the Hessian is −2/M. Therefore, the function $\hat {\sigma }$ is nonconvex.