Effective joint DOA-DOD estimation for the coexistence of uncorrelated and coherent signals in massive multi-input multi-output array systems

Abstract

This paper deals with the joint direction-of-arrival (DOA) and direction-of-departure (DOD) estimation when the uncorrelated and coherent (i.e., fully correlated) narrowband signals coexist in multiple-input multiple-output (MIMO) array systems. Two new approaches based on weighted subspace fitting and oblique projection for two-dimensional direction estimation, i.e., WSFOPDE and improved WSFOPDE, are proposed. In the WSFOPDE approach, the basic procedure includes three stages. First, the DOA of all signals can be directly acquired by minimizing a reduced-dimensional weighted subspace fitting function. Then, the DOA information of uncorrelated signals are discerned by a classifying indicator; and subsequently, their auto-paired transmit steering vectors with respect to DOD information are derived. Finally, via a new Toeplitz-structured oblique projection, an virtual MIMO array data with only coherent signals remaining is constructed to assist the corresponding auto-paired DOD estimation. In order to promote the accuracy of angle estimation, we also design an improved version. It inherits the above basic procedure and, meanwhile, introduces one-dimensional local DOA spectrum searching to refine the DOA-DOD estimation. Compared with some existing strategies, WSFOPDE and its improved version perform better from the united perspective of computational complexity and estimation accuracy. Numerical simulations verify the advantages and also demonstrate that both can be served as a better alternative to the competitors.

1 Introduction

The multiple-input multiple-output (MIMO) array systems have become a major research issue during the recent decades, especially concentrating on the radar and sonar target detection and localization [1–6]. It has many potential merits over the traditional phased-array radar such as better parameter identifiability, higher accuracy of parameter estimation, and much flexible transmit beampattern design. Generally speaking, there are two different categories according to the antennas’ configuration. One is the so-called statistical MIMO array system [2] with separated transmit and/or receive antennas, which is also known as distributed MIMO array systems. The biggest advantages are, for target detection and parameter estimation, to capture the spatial diversity of the radar cross section (RCS) with noncoherent processing; and for target localization, to provide a resolution far beyond that supported by the radar’s waveform with coherent processing. Oppositely, the other is the co-located MIMO array systems, which can be further classified into two types, i.e., monostatic arrays and bistatic arrays. The former one allows the transmitting and receiving arrays to be closely located, therefore it views the far-field target from the same perspective, e.g., direction of arrival (DOA). The latter one can view the targets from two different perspectives, i.e., DOA and direction of departure (DOD), because the transmitting array and receiving array locate separately. Such type of array systems usually aims at creating a virtual aperture with more degrees of freedom (DOFs) than the real aperture [7–9] to acquire narrower beamwidth, lower sidelobes and higher accuracy of angle estimation. Recently, massive MIMO, by employing large-scale antenna arrays at base station in mobile communication systems, can serve a large number of users, which greatly enhances the spectrum efficiency and energy efficiency in comparison to the traditional MIMO systems [10–12]. Actually, large scale configuration of antennas can generate two important benefits: one is the large number of DOFs, which allows much more flexible and accurate beamforming and null steering; the other is the large array aperture, e.g., if “massive” equips at both transmitting and receiving arrays simultaneously, then the precision of angle estimation with respect to target localization will be dramatically improved due to its high resolution of spatial direction. Based on the above benefits, the whole system is rewarded with the ability of ultra long-range detection and super strengthened identification. However, many potential challenges for enabling massive MIMO are also inevitable in actual radar and communication systems. Among them, how to efficiently utilize such “massive” resource in the aforementioned angle estimation should be considered carefully.

As a typical problem of two-dimensional harmonic retrieval with multiple measurement vectors, joint DOA and DOD estimation in bistatic MIMO array systems has been paid a great attention. Till now, many high-resolution algorithms have been developed, such as ESPRIT-based algorithms [13–16], the parallel factor analysis (PARAFAC)-based algorithms [17, 18], and MUSIC-based algorithms [19, 20]. Through the proof of numerical simulations, the reduced-dimension (RD) MUSIC algorithm shows very close performance to 2D-MUSIC algorithm [19]. However, these algorithms essentially depend on the uncorrelation or low-correlation property of the targets’ reflected signals. In fact, there always exist highly correlated or even coherent signals in practical environment, for example, two targets with a slight difference of Doppler shift or the multipath propagation. For highly correlated signals, one can adopt higher signal-to-noise ratio or larger number of snapshots to distinguish them, which is not a intrinsic problem; but more seriously, the coherence usually invalidates the aforementioned algorithms. Besides, these non-uncorrelated signals also destroy the virtual synthetic array [9]. Therefore, dealing with the rank-deficiency problem is uppermost when the uncorrelated and coherent signals coexist. Forward-backward spatial smoothing (FBSS) technique [21] can be directly utilized to de-correlate the coherence, but it is usually at the cost of array aperture. In [22, 23], a deflation approach is considered with two steps: spectrum searching for uncorrelated signals, then oblique projecting and spatial smoothing for coherent signals; differently, spatial difference technique in [24] takes advantage of the Toeplitz form of auto-variance matrix of uncorrelated signals to eliminate themselves’ contributions.

Although the spatial-differencing-based algorithms can deal with more signals than antennas, there still exist two potential shortcomings. One is that the contributions from a group of coherent signals may act approximately as that of a single-point signal, consequently, the false angle estimation will appear in the scenario of low signal-to-noise ratio and finite number of snapshots. The other is that part of the information of coherent signals will also be subtracted when the differencing operation is utilized to eliminate the uncorrelated signals, which will directly incur a restricted estimation performance for coherent signals. Aiming at the aforementioned problems, literature [25] designs a relative ratio function and a C-matrix to decern the possible false DOAs and achieve coherent DOA estimation, respectively. However, the method cannot be directly applied into the case of multi-dimensional angle estimation. The biggest difficulties are two-fold. First, the uncorrelated and/or partially correlated DOA estimations are usually by means of spectrum searching, the same as [22–24], which will produce huge computational complexity for multi-dimensional case; second, the non-diagonal covariance matrix caused by those uncorrelated or partially correlated signals will directly decrease the effectiveness of their elimination in the coherent DOA estimation stage. Although oblique projecting technique is adopted in [22, 23], it is actually achieved by an alternative projector due to one cannot acquire the array manifold of coherent signals beforehand. Differently, [26] distinguishes the uncorrelated and coherent signals by the moduli of the eigenvalues of a auxiliary matrix; and [27] exploits the sparse signal reconstruction to achieve angle estimation, which can neglect the different types of signals. However, both algorithms have to bear extra computational burden.

In this paper, motivated by simultaneously considering the reduction of computational complexity and the promotion of estimational accuracy, we propose two approaches based on the techniques of weighted subspace fitting and oblique projection, i.e., the WSFOPED and the improved WSFOPED, to achieve joint DOA and DOD estimation for a mixture of uncorrelated and coherent signals. In the WSFOPED approach, the reduced-dimensional weighted subspace fitting technique plays a very important role, the kernel of which is to remodel the partial noise subspace by a series of parameters; consequently, the DOA information of mixed signals can be acquired by rooting a polynomial represented by those estimated parameters. A new oblique projector is designed to isolate the coherent signals from the uncorrelated ones so that a virtual MIMO array observation can be rebuilt. In the DOD estimation, via Lagrange multiplier optimization, the auto-paired transmit array steering vectors of uncorrelated and coherent signals can be estimated with closed-form expressions. Furthermore, considering the inherent shortcoming (i.e., error propagation) in previous approach, the improved WSFOPED strategy is proposed to refine the accuracy of DOA-DOD estimation, in which one-dimensional spectrum searching in local angular domain is adopted. As a result, the proposed approaches have four notable advantages: (1) computational simplicity in DOA estimation of uncorrelated and coherent signals, where only polynomial rooting or local searching is required; (2) computational simplicity in DOD estimation of uncorrelated and coherent signals, where the transmit array steering vectors are estimated in a closed-form expressions; (3) the automatic pairing of DOA and DOD for the same signal; and (4) compared with non-spectrum-searching and spectrum-searching approaches respectively, WSFOPED and its improved version manifest better performance of DOA-DOD estimation. In a word, the proposed approaches can be considered as new alternatives to their competitors.

The rest of this paper is organized as follows. The problem formulation is presented in Section 2. The proposed joint DOA and DOD estimation approaches are introduced detailedly in Section 3 and Section 4. Section 5 gives the systemic discussion of the proposed approaches, and the simulation results to verify their advantages. Finally, we conclude the paper in Section 6.

Notation: (·)^∗, (·)^T, (·)^H, and (·)^† denote the complex conjugate, transpose, Hermitian transpose, and Moore-Penrose inverse, respectively. $\mathbb {E}\{\cdot \}$ represents the statistic expectation operation; diag{·} is the diagonal operator. rank{·} gives the matrix rank. Symbol “ ⊗” denotes Kronecker product, and “ ⊙” stands for Khatri-Rao product (column-wise Kronecker product). I_M is a M×M identity matrix and 0 symbolizes zero matrix. B ^(m) is a submatrix of B formed by its first m rows.

2 Problem formulation

Consider a bistatic massive MIMO array systems with M-element transmitting antenna array and N-element receiving antenna array, both of which adopt the uniform linearly-spaced configuration. The inter-element displacement d of both arrays is set as half the carrier wavelength to avoid the problem of spatial spectrum ambiguity. The DOD and DOA with respect to a far-field source signal are defined according to the normal line of antenna arrays, respectively. We herein concentrate on uniform linear array at both transmitting and receiving ends, however, the main results can be readily extended for other types of massive arrays such as uniform rectangular array (URA) or uniform circular array (UCA). The exact extension to both cases will be left for the future work.

At transmit end, all antennas simultaneously emit M orthogonal coded narrow signals to implement active detection. Based on the point source model, after reflecting by multiple targets, without loss of generality, we assume that it generates K signals in total. Among them, there are K₁ uncorrelated signals with widely separated Doppler frequencies. The DOD-DOA pair of the k-th signal $\phantom {\dot {i}\!}s_{k}(t)=\beta _{k} e^{j2\pi f_{k}t}$ is denoted by {θ_k,ϕ_k}, k=1,2,⋯,K₁, where β_k and f_k represent the complex attenuation coefficient and Doppler frequency, respectively. Besides, there may also exist Q groups of coherent signals, e.g., the l-th version in the q-th group is s_q(t) with DOA-DOD pairs {θ_ql,ϕ_ql} and attenuation coefficient γ_ql, l=1,2,⋯,L_q, q=1,2,⋯,Q, which are either caused by the same Doppler frequency or the multipath propagation. For convenience, define $K_{2}={\sum \nolimits }_{q=1}^{Q} L_{q}$, then we have K=K₁+K₂. Further, we assume that the signals in different categories are uncorrelated, and the s_q(t)’s are also uncorrelated with each other. It is worthy mentioning that the number of non-coherent signals K₁, the number of coherent signals K₂, and the number of coherent groups Q are also assumed to be known, or can be estimated in advance by the existing number detection technique [28], which is based on the smoothed rank profile test to the array covariance matrix.

At the receive end, the output data of the matched filtering can be expressed as [1, 17, 29],

$$\begin{array}{@{}rcl@{}} \mathbf{x}(t) &=& \sum\limits_{k=1}^{K_{1}} \left[\mathbf{b}(\phi_{k}) \otimes \mathbf{a}(\theta_{k})\right] \beta_{k} \mathrm{s}_{k}(t)\\ &&+ \sum\limits_{q=1}^{Q}\sum\limits_{l=1}^{L_{q}} \left[\mathbf{b}(\phi_{ql}) \otimes \mathbf{a}(\theta_{ql})\right] \gamma_{ql} \mathrm{s}_{q}(t) +\mathbf{n}(t)\\ &=& (\mathbf{B}_{u} \odot \mathbf{A}_{u})\mathbf{s}_{u}(t) + (\mathbf{B}_{c} \odot \mathbf{A}_{c})\mathbf{s}_{c}(t) + \mathbf{n}(t) \\ &=& (\mathbf{B} \odot \mathbf{A})\mathbf{s}(t) + \mathbf{n}(t). \end{array} $$

(1)

In the above formula, the k-th columns of the transmit steering matrix A=[A_u,A_c] and the receive steering matrix B=[B_u,B_c] are denoted as $\mathbf {a}(\theta _{k}) = \left [1, e^{j \frac {2\pi d \sin \theta _{k}}{\lambda }}, \cdots, e^{j \frac {2\pi (M-1)d \sin \theta _{k}}{\lambda }}\right ]^{\mathrm {T}}$ and $\mathbf {b}(\phi _{k}) = \left [1, e^{j \frac {2\pi d \sin \phi _{k}}{\lambda }}, \cdots, e^{j \frac {2\pi (N-1)d \sin \phi _{k}}{\lambda }}\right ]^{\mathrm {T}}$, respectively. Especially, we use a_u(θ) and b_u(ϕ) to represent the steering vectors of uncorrelated signals, and use a_c(θ) and b_c(ϕ) to represent the steering vectors of coherent signals. λ is the carrier wavelength of signals and d is the spacing of two adjacent antennas. The uncorrelated signals $\mathbf {s}_{u}(t)={\boldsymbol \beta }_{u} \left [\mathrm {s}_{1}(t), \cdots, \mathrm {s}_{K_{1}}(t)\right ]^{\mathrm {T}}$ with $\phantom {\dot {i}\!}\boldsymbol {\beta }_{u}=\text {diag}\{{\beta }_{1}, \cdots, {\beta }_{K_{1}}\}$. For coherent signals, $\mathbf {s}_{c}(t)=\left [\boldsymbol {\gamma }_{1}^{\mathrm {T}} \mathrm {s}_{K_{1}+1}(t), \cdots, \boldsymbol {\gamma }_{Q}^{\mathrm {T}} \mathrm {s}_{K_{1}+Q}(t)\right ]^{\mathrm {T}}$ with $\boldsymbol {\gamma }_{q}=\left [{\gamma }_{q1}, \cdots, {\gamma }_{q L_{q}}\right ]^{\mathrm {T}}$, so we have s(t)=[s_u(t)^T,s_c(t)^T]^T. n(t) is the additive zero-mean Gaussian noise with covariance $\sigma _{n}^{2}$, and is also uncorrelated with the signals.

Based on (1), we can calculate the covariance matrix of the MIMO array output data by

$$\begin{array}{@{}rcl@{}} \mathbf{R}_{\mathbf{x}} &=& \mathbb{E}\left\{\mathbf{x}(t)\mathbf{x}^{\mathrm{H}}(t)\right\} \\ &=& (\mathbf{B} \odot \mathbf{A})\mathbf{R}_{\mathbf{s}}(\mathbf{B} \odot \mathbf{A})^{\mathrm{H}} + \sigma_{n}^{2} \mathbf{I}_{MN} \\ &=& (\mathbf{B}_{u} \odot \mathbf{A}_{u})\mathbf{R}_{u}(\mathbf{B}_{u} \odot \mathbf{A}_{u})^{\mathrm{H}}\\ &&+ (\mathbf{B}_{c} \odot \mathbf{A}_{c})\mathbf{R}_{c}(\mathbf{B}_{c} \odot \mathbf{A}_{c})^{\mathrm{H}} +\sigma_{n}^{2} \mathbf{I}_{MN} \end{array} $$

(2)

where $\mathbf {R}_{u}=\mathbb {E}\{\mathbf {s}_{u}(t)\mathbf {s}_{u}(t)^{\mathrm {H}}\}$ and $\mathbf {R}_{c}=\mathbb {E}\left \{\mathbf {s}_{c}(t)\mathbf {s}_{c}(t)^{\mathrm {H}}\right \}$ are the correlation matrices of s_u(t) and s_c(t), respectively. R_s=diag{R_u,R_c} is a block diagonal matrix. If the spatial sampling frequency during the snapshot collection is greater than at least twice the largest Doppler shift, then we have $\mathbf {R}_{u}=\text {diag}\left \{\beta _{1}^{2}, \cdots, \beta _{K_{1}}^{2}\right \}$. In addition, based on the aforementioned assumptions, R_c is also a block diagonal matrix with the q-th block being

$$ \mathbf{R}_{c}^{q} = \left[\begin{array}{ccc} \gamma_{q1}^{2} & \cdots & \gamma_{q1}\gamma_{{qL}_{q}}^{\ast} \\ \vdots & \ddots & \vdots \\ \gamma_{{qL}_{q}}\gamma_{q1}^{\ast} & \cdots & \gamma_{{qL}_{q}}^{2} \end{array}\right] $$

(3)

and hence

$$ \mathbf{R}_{c} = \text{diag} \left\{ \mathbf{R}_{c}^{1}, \mathbf{R}_{c}^{2}, \cdots, \mathbf{R}_{c}^{Q} \right\} $$

(4)

Besides, R_x also has a equivalent representation from the perspective of signal subspace and noise subspace, i.e.,

$$ \mathbf{R}_{\mathbf{x}}=\mathbf{U}_{\mathbf{s}} {\boldsymbol{\Sigma}}_{\mathbf{s}} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}} + \mathbf{U}_{\mathbf{0}} {\boldsymbol{\Sigma}}_{\mathbf{0}} \mathbf{U}_{\mathbf{0}}^{\mathrm{H}} $$

(5)

In practice, R_x is usually acquired by the finite T array snapshots. The estimated version and its corresponding eigenvalue decomposition (EVD) can be represented as follows,

$$ \hat{\mathbf{R}}_{\mathbf{x}} = \frac{1}{T}\sum\limits_{t=1}^{T} \mathbf{x}(t)\mathbf{x}^{\mathrm{H}}(t) \overset{EVD}= \hat{\mathbf{U}}_{\mathbf{s}} \hat{\boldsymbol{\Sigma}}_{\mathbf{s}} \hat{\mathbf{U}}_{\mathbf{s}}^{\mathrm{H}} + \hat{\mathbf{U}}_{\mathbf{0}} \hat{\boldsymbol{\Sigma}}_{\mathbf{0}} \hat{\mathbf{U}}_{\mathbf{0}}^{\mathrm{H}} $$

(6)

where the eigenvalues are arranged in decreasing order, and the estimated signal subspace $\hat {\mathbf {U}}_{\mathbf {s}}$ is a MN×(K₁+Q) high matrix consisting of the eigenvectors with respect to the K₁+Q largest eigenvalues, i.e.,

$$ \hat{\boldsymbol{\Sigma}}_{\mathbf{s}}=\text{diag}\left\{\lambda_{1}, \lambda_{2}, \cdots, \lambda_{K_{1}+Q} \right\} $$

(7)

On the other hand, the remnant eigenvectors constitute the noise subspace $\hat {\mathbf {U}}_{\mathbf {0}}$.

For convenience, the derivations thereafter will still adopt non-estimated vectors or matrices. Besides, in the following sections, we first deal with the uncorrelated signals and then subtract their influence by a new oblique projector that can be utilized for assisting the angle estimation of coherent signals.

3 Joint DOA-DOD estimation for mixed signals: WSFOPDE approach

3.1 Weighted subspace fitting for DOA estimation

It has been shown in [30–32] that an asymptotically (for large snapshot number or high signal-to-noise ratio) statistically efficient estimation can be obtained by minimizing the following weighted subspace fitting problem

$$ \mathcal{F}(\boldsymbol{\theta}, \boldsymbol{\phi})=\text{tr} \left\{\mathbf{P}_{\mathbf{B} \odot \mathbf{A}}^{\bot} \mathbf{U}_{\mathbf{s}} \mathbf{W} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}}\right\} $$

(8)

where θ=[θ₁,θ₂,⋯,θ_K]^T and ϕ=[ϕ₁,ϕ₂,⋯,ϕ_K]^T. The diagonal weighted matrix

$$ \mathbf{W}=\left(\boldsymbol{\Sigma}_{\mathbf{s}}-\hat{\sigma}_{n}^{2}\mathbf{I}\right)^{2}\boldsymbol{\Sigma}_{\mathbf{s}}^{-1} $$

(9)

with the estimated noise variance

$$ \hat{\sigma}_{n}^{2}=\frac{1}{MN-K_{1}-Q}\sum\limits_{i=K_{1}+Q+1}^{MN} \lambda_{i} $$

(10)

Besides, PB⊙A⊥ in (8) stands for the orthogonal projector onto the null space of (B⊙A)^H, the expression of which is given by,

$$ \mathbf{P}_{\mathbf{B} \odot \mathbf{A}}^{\bot}=\mathbf{I}- (\mathbf{B} \odot \mathbf{A})\left[\left(\mathbf{B} \odot \mathbf{ A}\right)^{\mathrm{H}}\left(\mathbf{B} \odot \mathbf{A}\right)\right]^{-1} (\mathbf{B} \odot \mathbf{A})^{\mathrm{H}} $$

(11)

Although one can utilize spectrum searching scheme to minimize the function (8), it is of computational inefficiency. We herein adopt a MODE ¹-like algorithm that makes use of polynomial rooting. The difficulty relies on how to parameterize the projector PB⊙A⊥ because B and A are coupling. Therefore, we have to introduce an substitute.

To begin, we first try to parameterize the above projector by two coefficient vectors a=[a₀,a₁,⋯,a_K]^T and b=[b₀,b₁,⋯,b_K]^T. The coefficients construct two polynomials with an identical manner, the specific form of which is given below,

$$ \sum\limits_{i=0}^{K} c_{i} z^{K-i}=c_{0} \prod\limits_{i=1}^{K} \left(z-e^{j \pi \sin \psi_{i}}\right), \; \; c_{0} \neq 0 $$

(12)

where c_i∈{a_i,b_i} and ψ_i∈{θ_i,ϕ_i} correspondingly. If we introduce the following set

$$ \mathbb{L}=\left\{\{c_{i}\} \mid \mathcal{C}(z)=\sum\limits_{i=0}^{K} c_{i} z^{K-i} \neq 0 \; \; \; \text{for} \; \; \; \mid z \mid \neq 1 \right\} $$

(13)

it can be seen that the mapping from $\{\psi _{i}\} \in \mathbb {R}$ to $\{c_{i}\} \in \mathbb {L}$ is one to one providing we eliminate the non-uniqueness implied by the introduction of c₀≠1.

Let $\mathbf {G}_{\mathbf {a}} \in \mathbb {C}^{M \times (M-K)}$ and $\mathbf {G}_{\mathbf {b}} \in \mathbb {C}^{N \times (N-K)}$, for a and b, respectively, be with the following Toeplitz form

$$ \mathbf{G}_{\mathbf{c}}^{\mathrm{H}}= \left[\begin{array}{cccccc} c_{K} & \cdots & c_{1} & c_{0} & \cdots & 0 \\ 0 & \ddots & \ddots & \ddots & \ddots & 0 \\ 0 & \cdots & c_{K} & \cdots & c_{1} & c_{0} \end{array} \right], \; \; \; \mathbf{c} \in \{\mathbf{a}, \mathbf{b}\} $$

(14)

It is observed that rank{G_a}=M−K, rank{G_b}=N−K, and

$$ \mathbf{G}_{\mathbf{b}}^{\mathrm{H}} \mathbf{B}= \mathbf{G}_{\mathbf{a}}^{\mathrm{H}} \mathbf{A} =\mathbf{0} $$

(15)

Based on the above relation, consequently, we can conclude the following theorem that can be utilized to estimate all the DOA information.

Theorem 1

Let the columns of G_b span the null space of B^H, and if defining G=G_b⊗I_M, then span{G}⊂span{U₀}.

Proof

According to (15), on the one hand, the columns of G span a column space that satisfies

$$\mathbf{G}^{\mathrm{H}} (\mathbf{B} \odot \mathbf{A})=\left(\mathbf{G}_{\mathbf{b}}^{\mathrm{H}} \mathbf{B}\right) \odot \mathbf{A}=\mathbf{0}. $$

where we take advantage of the property of Kronecker product. On the other hand, if considering the rank of matrix G, it is shown that

$$ \text{rank}\{ \mathbf{G}\}=\text{rank}\{\mathbf{G}_{\mathbf{b}}\} \times \text{rank}\{\mathbf{I}_{M}\}=(N-K)M $$

(16)

As we know, the rank of noise subspace U₀ is rank{U₀}=MN−K. Obviously, if M≥2,

$$ \text{rank}\{ \mathbf{U}_{0} \} > \text{rank}\{ \mathbf{G}\} $$

(17)

Such result manifests that the column space spanned by G is included in the one spanned by U₀. This completes the proof. □

Therefore, we can construct a projection matrix,

$$ \mathbf{P}_{\mathbf{G}}=\mathbf{G}\left(\mathbf{G}^{\mathrm{H}}\mathbf{G}\right)^{-1}\mathbf{G}^{\mathrm{ H}}=\mathbf{P}_{\mathbf{G}_{\mathbf{b}}} \otimes \mathbf{I}_{M} $$

(18)

where $\mathbf {P}_{\mathbf {G}_{\mathbf {b}}}=\mathbf {G}_{\mathbf {b}}(\mathbf {G}_{\mathbf {b}}^{\mathrm { H}}\mathbf {G}_{\mathbf {b}})^{-1}\mathbf {G}_{\mathbf {b}}^{\mathrm {H}}$ ; then we substitute P_G for PB⊙A⊥ in (8), and correspondingly, a new objective function that needs to be minimized is reformulated as

$$ \mathcal{F}(\mathbf{b})=\text{tr} \left\{(\mathbf{P}_{\mathbf{G}_{\mathbf{b}}} \otimes \mathbf{I}_{M}) \mathbf{U}_{\mathbf{s}} \mathbf{W} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}} \right\} $$

(19)

If comparing function $\mathcal {F}(\boldsymbol {\theta }, \boldsymbol {\phi })$ with function $\mathcal {F}(\mathbf {b})$, we can see that the original two-dimensional optimization problem is transformed into a one-dimensional problem, that is to say, it greatly decreases the complexity.

In order to implement the minimization more conveniently, defining

$$ \bar{\mathbf{U}}=\mathbf{U}_{\mathbf{s}} \mathbf{W} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}}= \left[\begin{array}{ccc} \bar{\mathbf{U}}_{11} & \cdots & \bar{\mathbf{U}}_{1N}\\ \vdots & \ddots & \vdots \\ \bar{\mathbf{U}}_{N1} & \cdots & \bar{\mathbf{U}}_{NN} \end{array}\right] $$

(20)

where $\bar {\mathbf {U}}_{uv}$ is a M×M matrix, u,v=1,⋯,N. Further, we can get

$$ \hat{\mathbf{b}}=\arg \min \text{tr} \left\{ \mathbf{P}_{\mathbf{G}_{\mathbf{b}}} \left[\begin{array}{ccc} \text{tr}(\bar{\mathbf{U}}_{11}) & \cdots & \text{tr}(\bar{\mathbf{U}}_{1N})\\ \vdots & \ddots & \vdots \\ \text{tr}(\bar{\mathbf{U}}_{N1}) & \cdots & \text{tr}(\bar{\mathbf{U}}_{NN}) \end{array}\right] \right\}. $$

(21)

In addition, we also exert constraints on the unknown parameters $\{b_{i}\}_{i=1}^{N}$, i.e., $b_{i}=b_{N-i}^{\ast }$. The detailed discussion with respect to the above constraints and procedures for minimizing the above quadratic function can be found in [30, 33], therefore we skip it for avoiding redundance.

Once we get $\hat {\mathbf {b}}$, the angular phase of the roots of the estimated polynomial in (12) will give the DOA information of all types of signals, i.e., $\left \{\bar {\phi }_{k}\right \}_{k=1}^{K}$.

3.2 Classifying and uncorrelated DOD estimation

Although all DOAs are acquired, unfortunately, we do not know which one group of DOAs belongs to the type of uncorrelated signals. Hence, in the second stage, we introduce an indicator to classify.

As shown in [19], the two-dimensional MUSIC algorithm can be divided into two optimization problems

$$\begin{array}{*{20}l} & \max_{\phi} \ \mathbf{e}^{\mathrm{T}} \mathbf{E}^{-1}({\phi})\mathbf{e} \end{array} $$

(22)

$$\begin{array}{*{20}l} & \min_{\theta} \ \mathbf{a}(\theta)^{\mathrm{H}} \mathbf{E}(\phi) \mathbf{a}(\theta), \; \; \text{s.t.} \; \; \mathbf{e}^{\mathrm{T}}\mathbf{a}(\theta)=1 \end{array} $$

(23)

where

$$ \mathbf{E}(\phi)=[\mathbf{b}(\phi) \otimes \mathbf{I}_{M}]^{\mathrm{H}} \mathbf{U}_{\mathbf{0}}\mathbf{U}_{\mathbf{0}}^{\mathrm{H}} [\mathbf{b}(\phi) \otimes \mathbf{I}_{M}] $$

(24)

and e=[ 1,0,⋯,0]^T.

It is known that the coherent signals result in the rank-deficient in R_x, which is equivalent to a leakage of partial signal subspace into the noise subspace, hence, the orthogonality between them cannot be fulfilled perfectly. Based on that, for each DOA estimation $\bar {\phi }_{k}$, one can calculate the following indicator

$$ \mathcal{E}_{k}= \mathbf{e}^{\mathrm{T}} \mathbf{E}^{-1}\left(\bar{\phi}_{k}\right)\mathbf{e}, \; \; k=1, \cdots, K $$

(25)

and assemble them in set

$$ \mathcal{E}=\left\{\mathcal{E}_{1}, \mathcal{E}_{2}, \cdots, \mathcal{E}_{K} \right\} $$

(26)

Then the uncorrelated signals can be discerned by the DOA with respect to the K₁ largest values in $\mathcal {E}$.

After that, we can further utilize (23) to obtain the estimated auto-paired transmit steering vectors from which the uncorrelated DOD information can be extracted. Constructing a Lagrange cost function

$$ \mathcal{P}(\theta, \eta)=\mathbf{a}(\theta)^{\mathrm{H}} \mathbf{E}(\phi) \mathbf{a}(\theta) + \eta \left[ 1-\mathbf{e}^{\mathrm{T}}\mathbf{a}(\theta) \right] $$

(27)

where η is Lagrange multiplier. If we let the gradient of (27) with respect to a(θ) be equal to zero, we can obtain the estimated uncorrelated steering vectors to this cost function with nonsingular E(ϕ), which can be represented as, for each $\bar {\phi }_{k}$,

$$ \bar{\mathbf{a}}_{u}(\theta_{k}) =\left.\frac{\mathbf{E}({\phi})^{-1}\mathbf{e}}{\mathbf{e}^{\mathrm{ T}}\mathbf{E}(\phi)^{-1}\mathbf{e}} \right|_{\phi=\bar{\phi}_{k}}, \; \; k=1, 2, \cdots, K_{1}. $$

(28)

The estimated auto-paired DOD information can be drawn from the above steering vectors by means of least squares (LS) principle. Defining

$$ \boldsymbol{\alpha}_{k}^{\text{uwp}} \,=\, \frac{1}{\pi} \; {\text{angle}} \left[{\bar{\mathbf{a}}_{u}({\theta}_{k})}\right] \,=\, \left[0, {\cos{\theta}_{k}}, \cdots, (M-1){\cos{\theta}_{k}}\right]^{\mathrm{T}} $$

(29)

Note that the unwrapped phase $\boldsymbol {\alpha }_{k}^{\text {uwp}}(m)$, m=1,2,⋯,M, are given by

$$\begin{array}{*{20}l} \boldsymbol{\alpha}_{k}^{\text{uwp}}(1)&=&\boldsymbol{\alpha}_{k}^{\text{wp}}(1) \end{array} $$

(30)

$$\begin{array}{*{20}l} \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)&=&\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) + {\Delta}_{k}(m) \end{array} $$

(31)

where $\boldsymbol {\alpha }_{k}^{\text {wp}}$ is the wrapped phase and the Δ_k(m) is given by (32).

$$ {\begin{aligned}{\Delta}_{k}(m)=\left\{ \begin{array}{ll} \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) & \; \; \text{if} \; |\boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m)| \leq 1 \\ 2-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)+\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) & \; \; \text{if} \; \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) > 1 \\ \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m)+2 & \; \; \text{if} \; \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) < -1 \end{array}\right. \end{aligned}} $$

(32)

The LS fitting problem is shown as follows

$$ \min_{\mathbf{v}_{k}} \left| \left| \boldsymbol{\Omega}\mathbf{v}_{k}-\boldsymbol{\alpha}_{k}^{\text{uwp}} \right| \right|_{\mathrm{F}}^{2} $$

(33)

where $\mathbf {v}_{k} \in \mathbb {R}^{2 \times 1}$, and

$$\boldsymbol{\Omega}=\left[ \begin{array}{cccc} 0 & 1 & \cdots & M-1\\ 1 & 1 & \cdots & 1 \end{array} \right]{^{\mathrm{T}}} $$

The least square estimation of cosθ_k is given by the first element of $\hat {\mathbf {v}}_{k}=\boldsymbol {\Omega }^{\dagger } \boldsymbol {\alpha }_{k}^{\text {uwp}}$, that is to say,

$$\bar{\theta}_{k} = \arccos \hat{\mathbf{v}}_{k}(1) $$

Till now, the joint DOA and DOD estimation of uncorrelated signals has already been achieved. We will in the next stage take advantage of the coherent DOA information, $\bar {\phi }_{k}$ with k=K₁+1,K₁+2,⋯,K, to acquire the auto-paired coherent DOD information.

3.3 Oblique projecting for coherent DOD estimation

In order to implement DOD estimation of coherent signals, the elimination of the uncorrelated signals’ contribution in R_x is necessary. It can be achieved by the so-called oblique projection (OP) technique.

An oblique projection [34] is a kind of nonorthogonal projection, e.g., $\phantom {\dot {i}\!}\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}$, whose range is spanned by D₁ and null space is spanned by D₂,

$$ \mathbf{P}_{\mathbf{D}_{1}\mathbf{D}_{2}}=\mathbf{D}_{1} \left(\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\mathbf{D}_{2}}^{\bot} \mathbf{D}_{1} \right)^{-1}\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\mathbf{D}_{2}}^{\bot} $$

(34)

so that $\phantom {\dot {i}\!}\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{1}=\mathbf {D}_{1}$ and $\phantom {\dot {i}\!}\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{2}=\mathbf {0}$. In our discussed scenario, if let D₁=B_u⊙A_u and D₂=B_c⊙A_c, we can construct a virtual observation matrix Y with only coherent signals retaining, i.e.,

$$\begin{array}{*{20}l} \mathbf{Y} \triangleq &(\mathbf{I}_{MN}-\mathbf{P}_{\mathbf{D}_{1}\mathbf{D}_{2}})(\mathbf{R}_{\mathbf{x}}- \hat{\sigma}_{n}^{2} \mathbf{I}_{MN})(\mathbf{I}_{MN}-\mathbf{P}_{\mathbf{D}_{1}\mathbf{D}_{2}})^{\mathrm{H}}\\ =& (\mathbf{B}_{c} \odot \mathbf{A}_{c}) \mathbf{R}_{c} (\mathbf{B}_{c} \odot \mathbf{A}_{c})^{\mathrm{H}} \\ =& (\mathbf{B}_{c} \odot \mathbf{A}_{c}) \mathbf{S}_{c} \end{array} $$

(35)

where ${\hat \sigma }_{n}^{2}$ is given by (10).

Hence, the kernel problem is how to design a pragmatic oblique projection. As we know, the calculation of $\mathbf {P}_{\mathbf {D}_{2}}^{\bot }$ in (34) is impractical due to B_c⊙A_c is unknown. Literature [35] suggests that if we substitute R^† for $\mathbf {P}_{\mathbf {D}_{2}}^{\bot }$, where $\mathbf {R}^{\dag }=\mathbf {U}_{\mathbf {s}} \boldsymbol {\Sigma }_{\mathbf {s}}^{-1} \mathbf {U}_{\mathbf {s}}^{\mathrm {H}}$, i.e., $\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}=\mathbf {D}_{1} \left (\mathbf {D}_{1}^{\mathrm {H}} \mathbf {R}^{\dag } \mathbf {D}_{1} \right)^{-1}\mathbf {D}_{1}^{\mathrm {H}} \mathbf {R}^{\dag }$, then it works [22, 23]. However, such approximation usually makes a confusion because the power and information of coherent signals contributing to U_s are subtracted in Y. In [36], the QR factorization to the cross-covariance matrix of two parallel ULA is utilized to construct oblique projector, which can avoid the above confusion, however it is not appropriate for our case. We herein design a new oblique projector based on the pre-estimated angle information of uncorrelated signals.

3.3.1 New oblique projector

The kernel of this new oblique projector is the estimated DOA information of coherent signals from (21), i.e., $\left \{\bar {\phi }_{i}\right \}_{i=K_{1}+1}^{K}$.

Based on them, we can reconstruct a K₂-order polynomial with roots $\left \{e^{j \pi \sin {\bar \phi _{i}}}\right \}_{i=K_{1}+1}^{K}$ and K₂=K−K₁, the coefficients of which, for convenience, are defined by vector

$$ \mathbf{h}=[\!h_{0}, h_{1}, \cdots, h_{K_{2}}]^{\mathrm{T}} $$

(36)

Similar to (14), a Toeplitz matrix $\mathbf {G}_{\mathbf {h}} \in \mathbb {C}^{N \times (N-K_{2})}$ can be utilized to assist in constructing oblique projector because it satisfies $\mathbf {G}_{\mathbf {h}}^{\mathrm {H}}\mathbf {B}_{c}=\mathbf {0}$. If defining $\bar {\mathbf {G}}=\mathbf {G}_{\mathbf {h}} \otimes \mathbf {I}_{M}$, then we have

$$ \bar{\mathbf{G}}^{\mathrm{H}}\mathbf{D}_{2}=\bar{\mathbf{G}}^{\mathrm{H}} (\mathbf{B}_{c} \odot \mathbf{A}_{c})=(\mathbf{G}_{\mathbf{h}}^{\mathrm{H}} \mathbf{B}_{c}) \odot \mathbf{A}_{c}=\mathbf{0}. $$

(37)

That is to say, it can allow us to substitute $\mathbf {P}_{\bar {\mathbf {G}}}$ for $\mathbf {P}_{\mathbf {D}_{2}}^{\bot }$ in (34), so that an alternative choice of oblique projector is generated, i.e.,

$$ \bar{\mathbf{P}}_{\mathbf{D}_{1}\mathbf{D}_{2}}=\mathbf{D}_{1} \left(\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\bar{\mathbf{G}}} \mathbf{D}_{1} \right)^{-1}\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\bar{\mathbf{G}}} $$

(38)

where $\mathbf {P}_{\bar {\mathbf {G}}}=\bar {\mathbf {G}}(\bar {\mathbf {G}}^{\mathrm {H}}\bar {\mathbf {G}})^{-1}\bar {\mathbf {G}}^{\mathrm {H}}$. One can easily examine the properties that $\bar {\mathbf {P}}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{1}=\mathbf {D}_{1}$ and $\bar {\mathbf {P}}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{2}=\mathbf {0}$.

3.3.2 Forward-backward spatial smoothing

Taking (38) into (35), we can get that virtual observation Y. Given that rank{S_c}=rank{R_c} if B_c⊙A_c is a full column rank matrix and R_c is a rank-deficient matrix due to the coherent signals, therefore, we have to adopt two-dimensional spatial smoothing technique. Define a series of selection matrices, n=1,2,⋯,N−Z₁+1, m=1,2,⋯,M−Z₂+1,

$$ \begin{aligned} \mathbf{\Gamma}_{n, m} =& \left[ {\mathbf{0}_{Z_{1} \times (n-1)}} \; \mathbf{I}_{Z_{1}} \ {\mathbf{0}_{Z_{1} \times (N-Z_{1}-n+1)}}\right]\\ &\otimes \left[\mathbf{0}_{Z_{2} \times (m-1)} \; \mathbf{I}_{Z_{2}} \ {\mathbf{0}_{Z_{2} \times (M-Z_{2}-m+1)}}\right] \end{aligned} $$

(39)

where Z₁<N and Z₂<M denote the length of receive and transmit subarrays, respectively.

After stacking Γ_n,mY as the following style

$$\begin{aligned} \left[\mathbf{\Gamma}_{1,1} \mathbf{Y} \ \cdots \ \mathbf{\Gamma}_{1, M-Z_{2}+1} \mathbf{Y} \ \ \mathbf{\Gamma}_{2,1} \mathbf{Y} \ \cdots \ \right. \\ \left.\mathbf{\Gamma}_{2, M-Z_{2}+1} \mathbf{Y} \ \cdots \ \mathbf{\Gamma}_{N-Z_{1}+1, M-Z_{2}+1} \mathbf{Y} \right] \end{aligned} $$

it holds

$$ \bar{\mathbf{Y}} \triangleq \left[\mathbf{B}_{c}^{(Z_{1})} \odot \mathbf{A}_{c}^{(Z_{2})}\right] \bar{\mathbf{S}}_{c}. $$

(40)

Note that $\bar {\mathbf {S}}_{c} \in \mathbb {C}^{K_{2} \times (N-Z_{1}+1)(M-Z_{2}+1)MN}$ is a random phase modulated signal matrix which in turn de-correlated the rank-deficient S_c, so it is a full row rank matrix. In addition, similar to the conventional smoothing technique, the choices of Z₁ and Z₂ depend mainly on the number of coherent signals.

Utilizing the noise subspace of $\bar {\mathbf {Y}}$, i.e., $\bar {\mathbf {U}}_{0}$, we can obtain the estimated auto-paired transmit steering vectors for the coherent signals,

$$ \bar{\mathbf{a}}_{c}({\theta}_{k})=\frac{\bar{\mathbf{E}}(\bar{\phi}_{k})^{-1}\bar{\mathbf{e}}}{\bar{\mathbf{e}}^{\mathrm{T}}\bar{\mathbf{E}}(\bar{\phi}_{k})^{-1}\bar{\mathbf{e}}}, \ \ k=K_{1}+1, \cdots, K $$

(41)

where $\bar {\mathbf {E}}=\left [\bar {\mathbf {b}}({\phi }) \otimes \mathbf {I}_{Z_{2}}\right ]^{\mathrm {H}} {\bar {\mathbf {U}}_{\mathbf {0}}}{\bar {\mathbf {U}}_{\mathbf {0}}}^{\mathrm {H}} \left [\bar {\mathbf {b}}({\phi }) \otimes \mathbf {I}_{Z_{2}}\right ]$ with $\bar {\mathbf {b}}({\phi })=\mathbf {b}^{(Z_{1})}({\phi })$, and $\bar {\mathbf {e}}=\mathbf {e}^{(Z_{1})}$.

We have introduced the complete descriptions of the proposed WSFOPDE approach. For convenience, it is summarized in Table 1.

Table 1 Summary of the proposed WSFOPDE approach

4 The improved WSFOPDE approach

As we can see, the aforementioned approach estimates the DOD information on the premise that the DOA information are successfully obtained, seeing (28) and (41). That is to say, it is a successive manner. The accuracy of DOA estimation has a direct influence on that of DOD estimation. Therefore, in order to avoid the error propagation and make a further effort to improve the accuracy of DOA estimation, we then design an improved version.

Actually, the local spectrum searching can improve the estimation accuracy of uncorrelated DOA and coherent DOA. With the initial uncorrelated DOA estimation $\bar \phi _{i}, k=1, \cdots, K_{1}$, we can further refine them by

$$ {\hat \phi}_{k}=\arg \max_{\phi \in [{\bar \phi}_{k}-{\Delta \phi}, {\bar \phi}_{k}+{\Delta \phi}]} \mathbf{e}^{\mathrm{T}} \mathbf{E}^{-1}(\phi)\mathbf{e}, $$

(42)

where Δϕ is a small positive value, and ${\hat \phi }_{k}$ represent the final DOA estimation of k-th uncorrelated signal. Similarly, with the initial coherent DOA estimation $\bar \phi _{i}, k=K_{1}+1, \cdots, K$, we can also refine them by

$$ \hat{\phi}_{k}=\arg \max_{\phi \in \left[\bar{\phi}_{k}-{\Delta \phi}, \bar{\phi}_{k}+{\Delta \phi}\right]} \bar{\mathbf{e}}^{\mathrm{T}} \bar{\mathbf{E}}^{-1}(\phi)\bar{\mathbf{e}} $$

(43)

where Δϕ is a small positive value, and $\hat {\phi }_{k}$ represent the final DOA estimation of k-th coherent signal. Accordingly, the estimated auto-paired steering vectors for uncorrelated and coherent signals in Table 1, i.e., ${\bar {\mathbf {a}}_{u}({\theta }_{k})}$ and $\bar {\mathbf {a}}_{c}({\theta }_{k})$, are replaced by ${\hat {\mathbf {a}}_{u}({\theta }_{k})}$ and $\hat {\mathbf {a}}_{c}({\theta }_{k})$. Based on that, we can further rebuild an improved WSFOPDE approach that is shown in Table 2.

Table 2 Summary of the improved WSFOPDE approach

5 Results and discussion

5.1 Qualitative discussion

Although the proposed approaches make a aperture loss of transmit array for assisting coherent DOD estimation, seeing the reduced MIMO array in (40), they are still superior to some other approaches such as the 2D spatial smoothing + PARAFAC or 2D spatial smoothing + 2D-MUSIC/RD-MUSIC, because the latter ones sacrifice the array aperture on both receive and transmit ends to compensate the rank deficiency while the WSFOPDE and improved WSFOPDE approaches utilize the whole receive array to achieve DOA estimation.

For the improved WSFOPDE approach, the weighted subspace fitting technique not only provides initial DOA estimation for uncorrelated signals, but also gives a much accuracy DOA estimation for coherent signals. To do so, we avoid using $\bar {\mathbf {Y}}$ to perform joint DOA and DOD estimation of coherent signals. On the other hand, if borrowing the idea as introduced in [23], one also can use maximum likelihood, 2D-MUSIC or RD-MUSIC approaches to directly estimate the uncorrelated signals and then construct Toeplize matrix or oblique projection matrix to estimate coherent signals. Obviously, such scheme will cost much larger computation flops in spectrum searching. But the proposed one preserves the estimation accuracy of uncorrelated signals as done in [19, 20] and also can implement joint DOD and DOA estimation for coherent signals in a lower computational complexity, which is more attractable.

In addition, there are some differences between the proposed approaches and the ones in [13, 14]. First, although the researches in both references have made a detailed discussion on the estimation of DOA and delay in massive MIMO/FD-MIMO systems, they did not pay attention to the scenario of uncorrelated and coherent signals coexisting. Second, although the adopted ESPRIT-type algorithms in [13, 14] are also low-complexity, they were not based on weighted subspace fitting. Third, if such ESPRIT-type algorithm is utilized for mixed signals, it cannot isolate the coherent signals. Therefore, the proposed approaches maintain their own advantages.

5.1.1 Computational complexity

We now make a detailed analysis for the computational complexity of the proposed approaches. It is shown in Table 3. One flop is defined as one-time complex multiplication according to [37]. In WSFOPDE approach, the computational complexity during the minimization of (21) in Step 3 is based on [38]. The Step 5 contains the discerning operation and oblique projector construction. Besides, the improved WSFOPDE approach adds two extra steps to refine the DOA estimation, seeing the Table 2. Among them, Step 6 adds $\mathcal {O} \left \{K_{1}{\bar n}_{1}\left [\left (M^{2}N+M^{2}\right)(MN-K_{1}-Q)+ M^{3}\right ]\right \}$ and Step 9 costs $\mathcal {O} \left \{K_{2}{\bar n}_{2}\left [Z_{2}^{2}(Z_{1}+1)(Z_{1}Z_{2}-K_{2})+Z_{2}^{3}\right ] \right \}$, where ${\bar n}_{1}$ and ${\bar n}_{2}$ are the total searching number in each local angle domain, respectively.

Table 3 Analysis of the computational complexity

For easy to compare, we herein consider the RD-MUSIC + oblique projection + SS + RD-MUSIC algorithm, in which the discerning of uncorrelated signals is easily achieved by the angles with respect to K₁ largest spectrum peak or by calculating (25). The computational complexity is analyzed as follows: the uncorrelated DOA and DOD estimation require $\mathcal {O} \!\left \{\frac {1}{2}TM^{2} N^{2} \,+\, M^{3} N^{3} \,+\,(K_{1}\,+\,K\,+\,{\tilde n_{1}}) \left [\left (M^{2}N\!+M^{2}\right)(MN-{K}_{1}-Q)+M^{3}\right ]\right \}$, where ${\tilde n}_{1}$ is the total searching number in the whole angle domain; the same as the proposed one, oblique projection and SS require $\mathcal {O}\left \{M^{3}(N\,-\,K_{2})\left [N^{2}\,+\,2N(N\,-\,K_{2})\,+\,\left (N-K_{2}\right)^{2}\right ]+3M^{3}N^{3} +2K_{1}M^{2}N^{2} +2K_{1}^{2}MN+K_{1}^{3}\right \}$; the coherent DOA and DOD estimation require $\mathcal {O}\! \left \{ \frac {1}{2}(N-Z_{1}+1)(M-Z_{2}+1)MN {Z}_{1}^{2}Z_{2}^{2}\,+\, Z_{1}^{3}Z_{2}^{3} \,+\, (K_{2}\,+\,{\tilde n_{2}}) \left [Z_{2}^{2}(Z_{1}\,+\,1)(Z_{1}Z_{2}-K_{2})\,+\,Z_{2}^{3}\right ] \right \}$, where ${\tilde n}_{2}$ is the total searching number in the whole angle domain.

If taking some typical values of parameters into account, e.g., M=N=9,T=500, K=6,K₁=2,Q=1,K₂=4, ${\tilde n}_{1}={\tilde n}_{2}=180^{\circ }/0.01^{\circ }$, Z₁=Z₂=6. The local searching range in the improved WSEOPDE approach is set as Δϕ=2^∘ so that ${\bar n}_{1}={\bar n}_{2}=2^{\circ }/0.01^{\circ }$. By some tedious calculations, we can see that the proposed WSFOPDE approach has a computational complexity of $\mathcal {O} \{6.50 \times 10^{6}\}$ and its improved version has a computational complexity of $\mathcal {O} \left \{3.87 \times 10^{7}\right \}$, whereas the compared one is of $\mathcal {O} \left \{1.31 \times 10^{9}\right \}$. That is to say, our proposed approaches are computationally efficient. We also evaluate the average running time of executing one-time algorithm, one can refer to the simulation Example 1.

5.1.2 Parameter identifiability

Given that the parameterization in (14) and the rank of G in (16), it is necessary for the proposed approaches that K+1<N. Different from the theoretical analysis in [22–24] that the number of identifiable signals, including the uncorrelated and coherent signals, is beyond the number of actual antennas, the proposed approaches are of inferiority.

Actually, based on the aforementioned analysis, we can conclude that (1) the proposed approaches make a tradeoff between the array aperture and the computational complexity in comparison to its competitors and (2) the proposed approaches are very suitable for the case of massive arrays. It is the peculiarity of “massive” that allows the proposed ones to manifest a great alleviation in computational burden, i.e., the DOA estimation is achieved by polynomial rooting rather than the global searching [23]. Therefore, the proposed approaches can be viewed as two typical alternatives for providing a relatively higher accuracy of angle estimation but simultaneously consuming a lower computational complexity.

5.2 Quantitative results

In this section, we present some numerical simulations to demonstrate the effectiveness and advantages of the proposed approaches. The antenna spacing in both transmitting and receiving arrays is half the carrier wavelength, i.e., d=λ/2. The average root mean squared error (RMSE) is used to assess the performance of angle estimation, which is defined as

$$\frac{1}{\mathcal{K}} \sum_{k=1}^{\mathcal{K}} \sqrt {E \big [(\hat \theta_{k}- \theta_{k})^{2} + (\hat \phi_{k}- \phi_{k})^{2} \big ]}. $$

where $\mathcal {K}=K_{1}$ for uncorrelated signals and $\mathcal {K}=K_{2}$ for coherent signals. The total number of Monte Carlo simulations is set to be 200. The signal-to-noise ratio (SNR) is defined by (1), i.e.,

$$\text{SNR}=10\log_{10} \frac{\sum_{t} \left\|(\mathbf{B}\odot\mathbf{A})\mathbf{s}(t)\right\|^{2}}{MN \sigma_{n}^{2}} $$

For performance comparison, some typical algorithms such as PARAFAC algorithm [17] and ESPRIT algorithm are calculated after performing spatial smoothing. In addition, we also consider the deflation method [23], which is a typical algorithm of utilizing the MUSIC and oblique projector (it is abbreviated to ‘MUSIC+OP’ in the following content for convenience). In this deflation method, we purposely confine the angle searching range to four degrees for avoiding the huge calculation burden. All the results shown below are based on 200 independent trials, and the search procedure runs with a step size of 0.01^∘.

Example 1

Firstly, we examine the estimation performance with respect to SNR. Considering a scenario that contains two uncorrelated signals with β₁=β₂=1 and (θ,ϕ)∈{(30^∘,35^∘),(40^∘,45^∘)}, one group of four coherent signals with γ₁₁=e^jπ/8, γ₁₂=0.8e^j5π/9, γ₁₃=0.7e^jπ/18, γ₁₄=0.5e^j8π/9 and (θ,ϕ)∈{(−25^∘,−20^∘),(−15^∘,−10^∘),(10^∘,5^∘),(15^∘,25^∘)}. We adopt M=N=9 and the total number of array snapshots T=500. The sub-array length that is used for dealing with the coherent signals is set as Z₁=Z₂=8. From Figs. 1 and 2, we can observe that, the proposed WSFOPDE approach overmatches the other two non-searching algorithms, i.e., FBSS+ESPRIT and FBSS+PARAFAC (it is conducted by periodogram-based algorithm with 1024-point FFT after getting the auto-paired array steering vectors through tensor decomposition); and the proposed improved WSFOPDE approach performs nearly the same as the MUSIC+OP. In addition, the local searching indeed improves the accuracy of angle estimation with at least 5 dB leading than the proposed WSFOPDE approach for both uncorrelated and coherent signals.

Fig. 1

The performance comparison of DOA-DOD estimation between several typical algorithms: uncorrelated signals

Fig. 2

The performance comparison of DOA-DOD estimation between several typical algorithms: coherent signals

We also consider the time efficiency of different algorithms. Table 4 gives the average running time for executing one-time algorithm, where the MATLAB codes are executed in a PC with Intel(R) Core(TM) i5-6400 CPU@2.6GHz and 8 GB RAM. From the statistical results, we can see that, among the non-searching schemes, the WSFOPDE approach manifests a moderate time-consuming; whereas for the searching schemes, the improved WSFOPDE approach shows a great decrease in total computational complexity.

Table 4 The average time of executing one-time algorithm

Therefore, from a joint perspective of computational complexity and angle estimation accuracy, both proposed approaches can serve as better alternatives as compared to the existing competitors.

Example 2

Then, we evaluate the estimation performance versus the number of snapshots. The simulation conditions are similar to those in the previous example, except that the number of snapshots is set from 500 to 3000 and the SNR is fixed at 0 dB. We herein do not consider the FBSS+PARAFAC algorithm because of its unsatisfactory performance. The simulation results, as shown in Figs. 3 and 4, illustrate again that the average RMSE of the improved WSFOPDE approach is very close to the MUSIC+OP algorithm.

Fig. 3

The DOA-DOD estimation performance versus the number of snapshots: uncorrelated signals

Fig. 4

The DOA-DOD estimation performance versus the number of snapshots: coherent signals

Besides, it is shown that the FBSS+ESPRIT has a lower average RMSE for the coherent signals when the number of snapshot is less than 10³, but when comparing the extent of improvement with the increasing number of snapshots, i.e., the slope of the average RMSE curves, the WSFOPDE approach exhibits much better promotion than the FBSS+ESPRIT.

In order to assess the proposed approaches more comprehensively, in the following simulations, we focuss on two factors, i.e., the angular separation and the number of antennas.

Example 3

We also test the estimation performance in terms of the angular separation between the DOA of uncorrelated and coherent signals. Herein, one uncorrelated signal comes from (50^∘,η₁), and one group of two coherent signals come from {(15^∘,20^∘),(30^∘,η₂)} with the attenuation coefficient vector [1,e^jπ/3]^T, where η₁=η₂+Δη, η₂=40^∘, and Δη is varied from 2^∘ to 16^∘ with 2^∘ step. Other related parameters are set as the number of antennas M=N=7, the subarray size Z₁=Z₂=5, the number of snapshots T=500, and the SNR is fixed at 5 dB.

The simulation curves in Figs. 5 and 6 illustrate that the angle distinguishing ability of both approaches are restricted due to the closely spaced uncorrelated signal s₁(t) and coherent signal s₃(t); therefore, they all show unsatisfactory performance with larger average RMSE, especially for the angle estimation of coherent signals. With the angular separation becoming wider and wider, the performance is getting better; and basically, although the improved WSFOPDE approach requires more calculations, it is superior to its predecessor in all level of angular separation.

Fig. 5

The performance variation with angular separation between the DOA of mixed signals: uncorrelated signals

Fig. 6

The performance variation with angular separation between the DOA of mixed signals: coherent signals

Example 4

Besides the angular separation, the behavior of coherent signals will affect the performance of angle estimation. Herein, we mainly focuss on the sub-signals number of one group of coherent signals. In this example, there exist one uncorrelated signals and one group of coherent signals. The case 1 includes 4 sub-signals, and the case 2 includes 2 sub-signals. We restrict the DOA and DOD of coherent signals to the uniform-linear distribution in a fixed angular range [5^∘,35^∘] and [0^∘,30^∘], respectively. The uncorrelated signal is fixed at (θ,ϕ)=(40^∘,45^∘) in both cases. Other related parameters are set as: the number of antennas M=N=12 and T=500. Figures 7 and 8 illustrate the performance variation. As we can see, the increase of sub-signals number in one group of coherent signals will cause RMSE performance deterioration in angle estimation. Such deterioration in the RMSE of coherent signals is greater that the one of uncorrelated signals. In this sense, for ultra-high identifiability, the massive arrays are necessary.

Fig. 7

The RMSE performance variation under different sub-signals number in one group of coherent signals: uncorrelated signals

Fig. 8

The RMSE performance variation under different sub-signals number in one group of coherent signals: coherent signals

Example 5

Finally, we examine the average RMSE performance with different number of M and N. Herein we arrange M=N. In this example, the considered scenario includes one uncorrelated signal with β₁=1 and (θ,ϕ)=(10^∘,45^∘), two group of four coherent signals with γ₁₁=1, γ₁₂=0.5e^jπ/16, γ₂₁=1, γ₂₂=0.75e^jπ/7 and (θ,ϕ)∈{(30^∘,35^∘),(−20^∘,15^∘),(50^∘,−15^∘),(10^∘,−50^∘)}. In addition, the number of snapshots is set as T=500 and the SNR is fixed at 5dB.

It clearly manifests in Figs. 9 and 10 that the accuracy of DOA-DOD estimation is gradually improved with the increase of antenna number for both uncorrelated and coherent signals.

Fig. 9

The RMSE performance with different configuration of M and N: uncorrelated signals

Fig. 10

The RMSE performance with different configuration of M and N: coherent signals

6 Conclusions

In this paper, two computationally efficient approaches based on weighted subspace fitting and oblique projection, called WSFOPED and improved WSFOPED, were proposed for the joint DOA and DOD estimation of a mixture of uncorrelated and coherent narrowband signals in MIMO array systems, where the estimated DOA and DOD information is pair-matched automatically. The whole procedure includes three stages: polynomial rooting, uncorrelated DOA discerning and transmit steering vectors estimating, and virtual MIMO array data constructing via oblique projection. By systematical analysis of the computational complexity and sufficient simulation examples, the effectiveness of the proposed approaches were verified and they can be considered as better alternatives in two-dimensional spectrum estimation.