Skip to main content

Effective joint DOA-DOD estimation for the coexistence of uncorrelated and coherent signals in massive multi-input multi-output array systems


This paper deals with the joint direction-of-arrival (DOA) and direction-of-departure (DOD) estimation when the uncorrelated and coherent (i.e., fully correlated) narrowband signals coexist in multiple-input multiple-output (MIMO) array systems. Two new approaches based on weighted subspace fitting and oblique projection for two-dimensional direction estimation, i.e., WSFOPDE and improved WSFOPDE, are proposed. In the WSFOPDE approach, the basic procedure includes three stages. First, the DOA of all signals can be directly acquired by minimizing a reduced-dimensional weighted subspace fitting function. Then, the DOA information of uncorrelated signals are discerned by a classifying indicator; and subsequently, their auto-paired transmit steering vectors with respect to DOD information are derived. Finally, via a new Toeplitz-structured oblique projection, an virtual MIMO array data with only coherent signals remaining is constructed to assist the corresponding auto-paired DOD estimation. In order to promote the accuracy of angle estimation, we also design an improved version. It inherits the above basic procedure and, meanwhile, introduces one-dimensional local DOA spectrum searching to refine the DOA-DOD estimation. Compared with some existing strategies, WSFOPDE and its improved version perform better from the united perspective of computational complexity and estimation accuracy. Numerical simulations verify the advantages and also demonstrate that both can be served as a better alternative to the competitors.

1 Introduction

The multiple-input multiple-output (MIMO) array systems have become a major research issue during the recent decades, especially concentrating on the radar and sonar target detection and localization [16]. It has many potential merits over the traditional phased-array radar such as better parameter identifiability, higher accuracy of parameter estimation, and much flexible transmit beampattern design. Generally speaking, there are two different categories according to the antennas’ configuration. One is the so-called statistical MIMO array system [2] with separated transmit and/or receive antennas, which is also known as distributed MIMO array systems. The biggest advantages are, for target detection and parameter estimation, to capture the spatial diversity of the radar cross section (RCS) with noncoherent processing; and for target localization, to provide a resolution far beyond that supported by the radar’s waveform with coherent processing. Oppositely, the other is the co-located MIMO array systems, which can be further classified into two types, i.e., monostatic arrays and bistatic arrays. The former one allows the transmitting and receiving arrays to be closely located, therefore it views the far-field target from the same perspective, e.g., direction of arrival (DOA). The latter one can view the targets from two different perspectives, i.e., DOA and direction of departure (DOD), because the transmitting array and receiving array locate separately. Such type of array systems usually aims at creating a virtual aperture with more degrees of freedom (DOFs) than the real aperture [79] to acquire narrower beamwidth, lower sidelobes and higher accuracy of angle estimation. Recently, massive MIMO, by employing large-scale antenna arrays at base station in mobile communication systems, can serve a large number of users, which greatly enhances the spectrum efficiency and energy efficiency in comparison to the traditional MIMO systems [1012]. Actually, large scale configuration of antennas can generate two important benefits: one is the large number of DOFs, which allows much more flexible and accurate beamforming and null steering; the other is the large array aperture, e.g., if “massive” equips at both transmitting and receiving arrays simultaneously, then the precision of angle estimation with respect to target localization will be dramatically improved due to its high resolution of spatial direction. Based on the above benefits, the whole system is rewarded with the ability of ultra long-range detection and super strengthened identification. However, many potential challenges for enabling massive MIMO are also inevitable in actual radar and communication systems. Among them, how to efficiently utilize such “massive” resource in the aforementioned angle estimation should be considered carefully.

As a typical problem of two-dimensional harmonic retrieval with multiple measurement vectors, joint DOA and DOD estimation in bistatic MIMO array systems has been paid a great attention. Till now, many high-resolution algorithms have been developed, such as ESPRIT-based algorithms [1316], the parallel factor analysis (PARAFAC)-based algorithms [17, 18], and MUSIC-based algorithms [19, 20]. Through the proof of numerical simulations, the reduced-dimension (RD) MUSIC algorithm shows very close performance to 2D-MUSIC algorithm [19]. However, these algorithms essentially depend on the uncorrelation or low-correlation property of the targets’ reflected signals. In fact, there always exist highly correlated or even coherent signals in practical environment, for example, two targets with a slight difference of Doppler shift or the multipath propagation. For highly correlated signals, one can adopt higher signal-to-noise ratio or larger number of snapshots to distinguish them, which is not a intrinsic problem; but more seriously, the coherence usually invalidates the aforementioned algorithms. Besides, these non-uncorrelated signals also destroy the virtual synthetic array [9]. Therefore, dealing with the rank-deficiency problem is uppermost when the uncorrelated and coherent signals coexist. Forward-backward spatial smoothing (FBSS) technique [21] can be directly utilized to de-correlate the coherence, but it is usually at the cost of array aperture. In [22, 23], a deflation approach is considered with two steps: spectrum searching for uncorrelated signals, then oblique projecting and spatial smoothing for coherent signals; differently, spatial difference technique in [24] takes advantage of the Toeplitz form of auto-variance matrix of uncorrelated signals to eliminate themselves’ contributions.

Although the spatial-differencing-based algorithms can deal with more signals than antennas, there still exist two potential shortcomings. One is that the contributions from a group of coherent signals may act approximately as that of a single-point signal, consequently, the false angle estimation will appear in the scenario of low signal-to-noise ratio and finite number of snapshots. The other is that part of the information of coherent signals will also be subtracted when the differencing operation is utilized to eliminate the uncorrelated signals, which will directly incur a restricted estimation performance for coherent signals. Aiming at the aforementioned problems, literature [25] designs a relative ratio function and a C-matrix to decern the possible false DOAs and achieve coherent DOA estimation, respectively. However, the method cannot be directly applied into the case of multi-dimensional angle estimation. The biggest difficulties are two-fold. First, the uncorrelated and/or partially correlated DOA estimations are usually by means of spectrum searching, the same as [2224], which will produce huge computational complexity for multi-dimensional case; second, the non-diagonal covariance matrix caused by those uncorrelated or partially correlated signals will directly decrease the effectiveness of their elimination in the coherent DOA estimation stage. Although oblique projecting technique is adopted in [22, 23], it is actually achieved by an alternative projector due to one cannot acquire the array manifold of coherent signals beforehand. Differently, [26] distinguishes the uncorrelated and coherent signals by the moduli of the eigenvalues of a auxiliary matrix; and [27] exploits the sparse signal reconstruction to achieve angle estimation, which can neglect the different types of signals. However, both algorithms have to bear extra computational burden.

In this paper, motivated by simultaneously considering the reduction of computational complexity and the promotion of estimational accuracy, we propose two approaches based on the techniques of weighted subspace fitting and oblique projection, i.e., the WSFOPED and the improved WSFOPED, to achieve joint DOA and DOD estimation for a mixture of uncorrelated and coherent signals. In the WSFOPED approach, the reduced-dimensional weighted subspace fitting technique plays a very important role, the kernel of which is to remodel the partial noise subspace by a series of parameters; consequently, the DOA information of mixed signals can be acquired by rooting a polynomial represented by those estimated parameters. A new oblique projector is designed to isolate the coherent signals from the uncorrelated ones so that a virtual MIMO array observation can be rebuilt. In the DOD estimation, via Lagrange multiplier optimization, the auto-paired transmit array steering vectors of uncorrelated and coherent signals can be estimated with closed-form expressions. Furthermore, considering the inherent shortcoming (i.e., error propagation) in previous approach, the improved WSFOPED strategy is proposed to refine the accuracy of DOA-DOD estimation, in which one-dimensional spectrum searching in local angular domain is adopted. As a result, the proposed approaches have four notable advantages: (1) computational simplicity in DOA estimation of uncorrelated and coherent signals, where only polynomial rooting or local searching is required; (2) computational simplicity in DOD estimation of uncorrelated and coherent signals, where the transmit array steering vectors are estimated in a closed-form expressions; (3) the automatic pairing of DOA and DOD for the same signal; and (4) compared with non-spectrum-searching and spectrum-searching approaches respectively, WSFOPED and its improved version manifest better performance of DOA-DOD estimation. In a word, the proposed approaches can be considered as new alternatives to their competitors.

The rest of this paper is organized as follows. The problem formulation is presented in Section 2. The proposed joint DOA and DOD estimation approaches are introduced detailedly in Section 3 and Section 4. Section 5 gives the systemic discussion of the proposed approaches, and the simulation results to verify their advantages. Finally, we conclude the paper in Section 6.

Notation: (·), (·)T, (·)H, and (·) denote the complex conjugate, transpose, Hermitian transpose, and Moore-Penrose inverse, respectively. \(\mathbb {E}\{\cdot \}\) represents the statistic expectation operation; diag{·} is the diagonal operator. rank{·} gives the matrix rank. Symbol “ ” denotes Kronecker product, and “ ” stands for Khatri-Rao product (column-wise Kronecker product). IM is a M×M identity matrix and 0 symbolizes zero matrix. B (m) is a submatrix of B formed by its first m rows.

2 Problem formulation

Consider a bistatic massive MIMO array systems with M-element transmitting antenna array and N-element receiving antenna array, both of which adopt the uniform linearly-spaced configuration. The inter-element displacement d of both arrays is set as half the carrier wavelength to avoid the problem of spatial spectrum ambiguity. The DOD and DOA with respect to a far-field source signal are defined according to the normal line of antenna arrays, respectively. We herein concentrate on uniform linear array at both transmitting and receiving ends, however, the main results can be readily extended for other types of massive arrays such as uniform rectangular array (URA) or uniform circular array (UCA). The exact extension to both cases will be left for the future work.

At transmit end, all antennas simultaneously emit M orthogonal coded narrow signals to implement active detection. Based on the point source model, after reflecting by multiple targets, without loss of generality, we assume that it generates K signals in total. Among them, there are K1 uncorrelated signals with widely separated Doppler frequencies. The DOD-DOA pair of the k-th signal \(\phantom {\dot {i}\!}s_{k}(t)=\beta _{k} e^{j2\pi f_{k}t}\) is denoted by {θk,ϕk}, k=1,2,,K1, where βk and fk represent the complex attenuation coefficient and Doppler frequency, respectively. Besides, there may also exist Q groups of coherent signals, e.g., the l-th version in the q-th group is sq(t) with DOA-DOD pairs {θql,ϕql} and attenuation coefficient γql, l=1,2,,Lq, q=1,2,,Q, which are either caused by the same Doppler frequency or the multipath propagation. For convenience, define \(K_{2}={\sum \nolimits }_{q=1}^{Q} L_{q}\), then we have K=K1+K2. Further, we assume that the signals in different categories are uncorrelated, and the sq(t)’s are also uncorrelated with each other. It is worthy mentioning that the number of non-coherent signals K1, the number of coherent signals K2, and the number of coherent groups Q are also assumed to be known, or can be estimated in advance by the existing number detection technique [28], which is based on the smoothed rank profile test to the array covariance matrix.

At the receive end, the output data of the matched filtering can be expressed as [1, 17, 29],

$$\begin{array}{@{}rcl@{}} \mathbf{x}(t) &=& \sum\limits_{k=1}^{K_{1}} \left[\mathbf{b}(\phi_{k}) \otimes \mathbf{a}(\theta_{k})\right] \beta_{k} \mathrm{s}_{k}(t)\\ &&+ \sum\limits_{q=1}^{Q}\sum\limits_{l=1}^{L_{q}} \left[\mathbf{b}(\phi_{ql}) \otimes \mathbf{a}(\theta_{ql})\right] \gamma_{ql} \mathrm{s}_{q}(t) +\mathbf{n}(t)\\ &=& (\mathbf{B}_{u} \odot \mathbf{A}_{u})\mathbf{s}_{u}(t) + (\mathbf{B}_{c} \odot \mathbf{A}_{c})\mathbf{s}_{c}(t) + \mathbf{n}(t) \\ &=& (\mathbf{B} \odot \mathbf{A})\mathbf{s}(t) + \mathbf{n}(t). \end{array} $$

In the above formula, the k-th columns of the transmit steering matrix A=[Au,Ac] and the receive steering matrix B=[Bu,Bc] are denoted as \(\mathbf {a}(\theta _{k}) = \left [1, e^{j \frac {2\pi d \sin \theta _{k}}{\lambda }}, \cdots, e^{j \frac {2\pi (M-1)d \sin \theta _{k}}{\lambda }}\right ]^{\mathrm {T}}\) and \(\mathbf {b}(\phi _{k}) = \left [1, e^{j \frac {2\pi d \sin \phi _{k}}{\lambda }}, \cdots, e^{j \frac {2\pi (N-1)d \sin \phi _{k}}{\lambda }}\right ]^{\mathrm {T}}\), respectively. Especially, we use au(θ) and bu(ϕ) to represent the steering vectors of uncorrelated signals, and use ac(θ) and bc(ϕ) to represent the steering vectors of coherent signals. λ is the carrier wavelength of signals and d is the spacing of two adjacent antennas. The uncorrelated signals \(\mathbf {s}_{u}(t)={\boldsymbol \beta }_{u} \left [\mathrm {s}_{1}(t), \cdots, \mathrm {s}_{K_{1}}(t)\right ]^{\mathrm {T}}\) with \(\phantom {\dot {i}\!}\boldsymbol {\beta }_{u}=\text {diag}\{{\beta }_{1}, \cdots, {\beta }_{K_{1}}\}\). For coherent signals, \(\mathbf {s}_{c}(t)=\left [\boldsymbol {\gamma }_{1}^{\mathrm {T}} \mathrm {s}_{K_{1}+1}(t), \cdots, \boldsymbol {\gamma }_{Q}^{\mathrm {T}} \mathrm {s}_{K_{1}+Q}(t)\right ]^{\mathrm {T}}\) with \(\boldsymbol {\gamma }_{q}=\left [{\gamma }_{q1}, \cdots, {\gamma }_{q L_{q}}\right ]^{\mathrm {T}}\), so we have s(t)=[su(t)T,sc(t)T]T. n(t) is the additive zero-mean Gaussian noise with covariance \(\sigma _{n}^{2}\), and is also uncorrelated with the signals.

Based on (1), we can calculate the covariance matrix of the MIMO array output data by

$$\begin{array}{@{}rcl@{}} \mathbf{R}_{\mathbf{x}} &=& \mathbb{E}\left\{\mathbf{x}(t)\mathbf{x}^{\mathrm{H}}(t)\right\} \\ &=& (\mathbf{B} \odot \mathbf{A})\mathbf{R}_{\mathbf{s}}(\mathbf{B} \odot \mathbf{A})^{\mathrm{H}} + \sigma_{n}^{2} \mathbf{I}_{MN} \\ &=& (\mathbf{B}_{u} \odot \mathbf{A}_{u})\mathbf{R}_{u}(\mathbf{B}_{u} \odot \mathbf{A}_{u})^{\mathrm{H}}\\ &&+ (\mathbf{B}_{c} \odot \mathbf{A}_{c})\mathbf{R}_{c}(\mathbf{B}_{c} \odot \mathbf{A}_{c})^{\mathrm{H}} +\sigma_{n}^{2} \mathbf{I}_{MN} \end{array} $$

where \(\mathbf {R}_{u}=\mathbb {E}\{\mathbf {s}_{u}(t)\mathbf {s}_{u}(t)^{\mathrm {H}}\}\) and \(\mathbf {R}_{c}=\mathbb {E}\left \{\mathbf {s}_{c}(t)\mathbf {s}_{c}(t)^{\mathrm {H}}\right \}\) are the correlation matrices of su(t) and sc(t), respectively. Rs=diag{Ru,Rc} is a block diagonal matrix. If the spatial sampling frequency during the snapshot collection is greater than at least twice the largest Doppler shift, then we have \(\mathbf {R}_{u}=\text {diag}\left \{\beta _{1}^{2}, \cdots, \beta _{K_{1}}^{2}\right \}\). In addition, based on the aforementioned assumptions, Rc is also a block diagonal matrix with the q-th block being

$$ \mathbf{R}_{c}^{q} = \left[\begin{array}{ccc} \gamma_{q1}^{2} & \cdots & \gamma_{q1}\gamma_{{qL}_{q}}^{\ast} \\ \vdots & \ddots & \vdots \\ \gamma_{{qL}_{q}}\gamma_{q1}^{\ast} & \cdots & \gamma_{{qL}_{q}}^{2} \end{array}\right] $$

and hence

$$ \mathbf{R}_{c} = \text{diag} \left\{ \mathbf{R}_{c}^{1}, \mathbf{R}_{c}^{2}, \cdots, \mathbf{R}_{c}^{Q} \right\} $$

Besides, Rx also has a equivalent representation from the perspective of signal subspace and noise subspace, i.e.,

$$ \mathbf{R}_{\mathbf{x}}=\mathbf{U}_{\mathbf{s}} {\boldsymbol{\Sigma}}_{\mathbf{s}} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}} + \mathbf{U}_{\mathbf{0}} {\boldsymbol{\Sigma}}_{\mathbf{0}} \mathbf{U}_{\mathbf{0}}^{\mathrm{H}} $$

In practice, Rx is usually acquired by the finite T array snapshots. The estimated version and its corresponding eigenvalue decomposition (EVD) can be represented as follows,

$$ \hat{\mathbf{R}}_{\mathbf{x}} = \frac{1}{T}\sum\limits_{t=1}^{T} \mathbf{x}(t)\mathbf{x}^{\mathrm{H}}(t) \overset{EVD}= \hat{\mathbf{U}}_{\mathbf{s}} \hat{\boldsymbol{\Sigma}}_{\mathbf{s}} \hat{\mathbf{U}}_{\mathbf{s}}^{\mathrm{H}} + \hat{\mathbf{U}}_{\mathbf{0}} \hat{\boldsymbol{\Sigma}}_{\mathbf{0}} \hat{\mathbf{U}}_{\mathbf{0}}^{\mathrm{H}} $$

where the eigenvalues are arranged in decreasing order, and the estimated signal subspace \(\hat {\mathbf {U}}_{\mathbf {s}}\) is a MN×(K1+Q) high matrix consisting of the eigenvectors with respect to the K1+Q largest eigenvalues, i.e.,

$$ \hat{\boldsymbol{\Sigma}}_{\mathbf{s}}=\text{diag}\left\{\lambda_{1}, \lambda_{2}, \cdots, \lambda_{K_{1}+Q} \right\} $$

On the other hand, the remnant eigenvectors constitute the noise subspace \(\hat {\mathbf {U}}_{\mathbf {0}}\).

For convenience, the derivations thereafter will still adopt non-estimated vectors or matrices. Besides, in the following sections, we first deal with the uncorrelated signals and then subtract their influence by a new oblique projector that can be utilized for assisting the angle estimation of coherent signals.

3 Joint DOA-DOD estimation for mixed signals: WSFOPDE approach

3.1 Weighted subspace fitting for DOA estimation

It has been shown in [3032] that an asymptotically (for large snapshot number or high signal-to-noise ratio) statistically efficient estimation can be obtained by minimizing the following weighted subspace fitting problem

$$ \mathcal{F}(\boldsymbol{\theta}, \boldsymbol{\phi})=\text{tr} \left\{\mathbf{P}_{\mathbf{B} \odot \mathbf{A}}^{\bot} \mathbf{U}_{\mathbf{s}} \mathbf{W} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}}\right\} $$

where θ=[θ1,θ2,,θK]T and ϕ=[ϕ1,ϕ2,,ϕK]T. The diagonal weighted matrix

$$ \mathbf{W}=\left(\boldsymbol{\Sigma}_{\mathbf{s}}-\hat{\sigma}_{n}^{2}\mathbf{I}\right)^{2}\boldsymbol{\Sigma}_{\mathbf{s}}^{-1} $$

with the estimated noise variance

$$ \hat{\sigma}_{n}^{2}=\frac{1}{MN-K_{1}-Q}\sum\limits_{i=K_{1}+Q+1}^{MN} \lambda_{i} $$

Besides, PBA in (8) stands for the orthogonal projector onto the null space of (BA)H, the expression of which is given by,

$$ \mathbf{P}_{\mathbf{B} \odot \mathbf{A}}^{\bot}=\mathbf{I}- (\mathbf{B} \odot \mathbf{A})\left[\left(\mathbf{B} \odot \mathbf{ A}\right)^{\mathrm{H}}\left(\mathbf{B} \odot \mathbf{A}\right)\right]^{-1} (\mathbf{B} \odot \mathbf{A})^{\mathrm{H}} $$

Although one can utilize spectrum searching scheme to minimize the function (8), it is of computational inefficiency. We herein adopt a MODE Footnote 1-like algorithm that makes use of polynomial rooting. The difficulty relies on how to parameterize the projector PBA because B and A are coupling. Therefore, we have to introduce an substitute.

To begin, we first try to parameterize the above projector by two coefficient vectors a=[a0,a1,,aK]T and b=[b0,b1,,bK]T. The coefficients construct two polynomials with an identical manner, the specific form of which is given below,

$$ \sum\limits_{i=0}^{K} c_{i} z^{K-i}=c_{0} \prod\limits_{i=1}^{K} \left(z-e^{j \pi \sin \psi_{i}}\right), \; \; c_{0} \neq 0 $$

where ci{ai,bi} and ψi{θi,ϕi} correspondingly. If we introduce the following set

$$ \mathbb{L}=\left\{\{c_{i}\} \mid \mathcal{C}(z)=\sum\limits_{i=0}^{K} c_{i} z^{K-i} \neq 0 \; \; \; \text{for} \; \; \; \mid z \mid \neq 1 \right\} $$

it can be seen that the mapping from \(\{\psi _{i}\} \in \mathbb {R}\) to \(\{c_{i}\} \in \mathbb {L}\) is one to one providing we eliminate the non-uniqueness implied by the introduction of c0≠1.

Let \(\mathbf {G}_{\mathbf {a}} \in \mathbb {C}^{M \times (M-K)}\) and \(\mathbf {G}_{\mathbf {b}} \in \mathbb {C}^{N \times (N-K)}\), for a and b, respectively, be with the following Toeplitz form

$$ \mathbf{G}_{\mathbf{c}}^{\mathrm{H}}= \left[\begin{array}{cccccc} c_{K} & \cdots & c_{1} & c_{0} & \cdots & 0 \\ 0 & \ddots & \ddots & \ddots & \ddots & 0 \\ 0 & \cdots & c_{K} & \cdots & c_{1} & c_{0} \end{array} \right], \; \; \; \mathbf{c} \in \{\mathbf{a}, \mathbf{b}\} $$

It is observed that rank{Ga}=MK, rank{Gb}=NK, and

$$ \mathbf{G}_{\mathbf{b}}^{\mathrm{H}} \mathbf{B}= \mathbf{G}_{\mathbf{a}}^{\mathrm{H}} \mathbf{A} =\mathbf{0} $$

Based on the above relation, consequently, we can conclude the following theorem that can be utilized to estimate all the DOA information.

Theorem 1

Let the columns of Gb span the null space of BH, and if defining G=GbIM, then span{G}span{U0}.


According to (15), on the one hand, the columns of G span a column space that satisfies

$$\mathbf{G}^{\mathrm{H}} (\mathbf{B} \odot \mathbf{A})=\left(\mathbf{G}_{\mathbf{b}}^{\mathrm{H}} \mathbf{B}\right) \odot \mathbf{A}=\mathbf{0}. $$

where we take advantage of the property of Kronecker product. On the other hand, if considering the rank of matrix G, it is shown that

$$ \text{rank}\{ \mathbf{G}\}=\text{rank}\{\mathbf{G}_{\mathbf{b}}\} \times \text{rank}\{\mathbf{I}_{M}\}=(N-K)M $$

As we know, the rank of noise subspace U0 is rank{U0}=MNK. Obviously, if M≥2,

$$ \text{rank}\{ \mathbf{U}_{0} \} > \text{rank}\{ \mathbf{G}\} $$

Such result manifests that the column space spanned by G is included in the one spanned by U0. This completes the proof. □

Therefore, we can construct a projection matrix,

$$ \mathbf{P}_{\mathbf{G}}=\mathbf{G}\left(\mathbf{G}^{\mathrm{H}}\mathbf{G}\right)^{-1}\mathbf{G}^{\mathrm{ H}}=\mathbf{P}_{\mathbf{G}_{\mathbf{b}}} \otimes \mathbf{I}_{M} $$

where \(\mathbf {P}_{\mathbf {G}_{\mathbf {b}}}=\mathbf {G}_{\mathbf {b}}(\mathbf {G}_{\mathbf {b}}^{\mathrm { H}}\mathbf {G}_{\mathbf {b}})^{-1}\mathbf {G}_{\mathbf {b}}^{\mathrm {H}}\) ; then we substitute PG for PBA in (8), and correspondingly, a new objective function that needs to be minimized is reformulated as

$$ \mathcal{F}(\mathbf{b})=\text{tr} \left\{(\mathbf{P}_{\mathbf{G}_{\mathbf{b}}} \otimes \mathbf{I}_{M}) \mathbf{U}_{\mathbf{s}} \mathbf{W} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}} \right\} $$

If comparing function \(\mathcal {F}(\boldsymbol {\theta }, \boldsymbol {\phi })\) with function \(\mathcal {F}(\mathbf {b})\), we can see that the original two-dimensional optimization problem is transformed into a one-dimensional problem, that is to say, it greatly decreases the complexity.

In order to implement the minimization more conveniently, defining

$$ \bar{\mathbf{U}}=\mathbf{U}_{\mathbf{s}} \mathbf{W} \mathbf{U}_{\mathbf{s}}^{\mathrm{H}}= \left[\begin{array}{ccc} \bar{\mathbf{U}}_{11} & \cdots & \bar{\mathbf{U}}_{1N}\\ \vdots & \ddots & \vdots \\ \bar{\mathbf{U}}_{N1} & \cdots & \bar{\mathbf{U}}_{NN} \end{array}\right] $$

where \(\bar {\mathbf {U}}_{uv}\) is a M×M matrix, u,v=1,,N. Further, we can get

$$ \hat{\mathbf{b}}=\arg \min \text{tr} \left\{ \mathbf{P}_{\mathbf{G}_{\mathbf{b}}} \left[\begin{array}{ccc} \text{tr}(\bar{\mathbf{U}}_{11}) & \cdots & \text{tr}(\bar{\mathbf{U}}_{1N})\\ \vdots & \ddots & \vdots \\ \text{tr}(\bar{\mathbf{U}}_{N1}) & \cdots & \text{tr}(\bar{\mathbf{U}}_{NN}) \end{array}\right] \right\}. $$

In addition, we also exert constraints on the unknown parameters \(\{b_{i}\}_{i=1}^{N}\), i.e., \(b_{i}=b_{N-i}^{\ast }\). The detailed discussion with respect to the above constraints and procedures for minimizing the above quadratic function can be found in [30, 33], therefore we skip it for avoiding redundance.

Once we get \(\hat {\mathbf {b}}\), the angular phase of the roots of the estimated polynomial in (12) will give the DOA information of all types of signals, i.e., \(\left \{\bar {\phi }_{k}\right \}_{k=1}^{K}\).

3.2 Classifying and uncorrelated DOD estimation

Although all DOAs are acquired, unfortunately, we do not know which one group of DOAs belongs to the type of uncorrelated signals. Hence, in the second stage, we introduce an indicator to classify.

As shown in [19], the two-dimensional MUSIC algorithm can be divided into two optimization problems

$$\begin{array}{*{20}l} & \max_{\phi} \ \mathbf{e}^{\mathrm{T}} \mathbf{E}^{-1}({\phi})\mathbf{e} \end{array} $$
$$\begin{array}{*{20}l} & \min_{\theta} \ \mathbf{a}(\theta)^{\mathrm{H}} \mathbf{E}(\phi) \mathbf{a}(\theta), \; \; \text{s.t.} \; \; \mathbf{e}^{\mathrm{T}}\mathbf{a}(\theta)=1 \end{array} $$


$$ \mathbf{E}(\phi)=[\mathbf{b}(\phi) \otimes \mathbf{I}_{M}]^{\mathrm{H}} \mathbf{U}_{\mathbf{0}}\mathbf{U}_{\mathbf{0}}^{\mathrm{H}} [\mathbf{b}(\phi) \otimes \mathbf{I}_{M}] $$

and e=[ 1,0,,0]T.

It is known that the coherent signals result in the rank-deficient in Rx, which is equivalent to a leakage of partial signal subspace into the noise subspace, hence, the orthogonality between them cannot be fulfilled perfectly. Based on that, for each DOA estimation \(\bar {\phi }_{k}\), one can calculate the following indicator

$$ \mathcal{E}_{k}= \mathbf{e}^{\mathrm{T}} \mathbf{E}^{-1}\left(\bar{\phi}_{k}\right)\mathbf{e}, \; \; k=1, \cdots, K $$

and assemble them in set

$$ \mathcal{E}=\left\{\mathcal{E}_{1}, \mathcal{E}_{2}, \cdots, \mathcal{E}_{K} \right\} $$

Then the uncorrelated signals can be discerned by the DOA with respect to the K1 largest values in \(\mathcal {E}\).

After that, we can further utilize (23) to obtain the estimated auto-paired transmit steering vectors from which the uncorrelated DOD information can be extracted. Constructing a Lagrange cost function

$$ \mathcal{P}(\theta, \eta)=\mathbf{a}(\theta)^{\mathrm{H}} \mathbf{E}(\phi) \mathbf{a}(\theta) + \eta \left[ 1-\mathbf{e}^{\mathrm{T}}\mathbf{a}(\theta) \right] $$

where η is Lagrange multiplier. If we let the gradient of (27) with respect to a(θ) be equal to zero, we can obtain the estimated uncorrelated steering vectors to this cost function with nonsingular E(ϕ), which can be represented as, for each \(\bar {\phi }_{k}\),

$$ \bar{\mathbf{a}}_{u}(\theta_{k}) =\left.\frac{\mathbf{E}({\phi})^{-1}\mathbf{e}}{\mathbf{e}^{\mathrm{ T}}\mathbf{E}(\phi)^{-1}\mathbf{e}} \right|_{\phi=\bar{\phi}_{k}}, \; \; k=1, 2, \cdots, K_{1}. $$

The estimated auto-paired DOD information can be drawn from the above steering vectors by means of least squares (LS) principle. Defining

$$ \boldsymbol{\alpha}_{k}^{\text{uwp}} \,=\, \frac{1}{\pi} \; {\text{angle}} \left[{\bar{\mathbf{a}}_{u}({\theta}_{k})}\right] \,=\, \left[0, {\cos{\theta}_{k}}, \cdots, (M-1){\cos{\theta}_{k}}\right]^{\mathrm{T}} $$

Note that the unwrapped phase \(\boldsymbol {\alpha }_{k}^{\text {uwp}}(m)\), m=1,2,,M, are given by

$$\begin{array}{*{20}l} \boldsymbol{\alpha}_{k}^{\text{uwp}}(1)&=&\boldsymbol{\alpha}_{k}^{\text{wp}}(1) \end{array} $$
$$\begin{array}{*{20}l} \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)&=&\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) + {\Delta}_{k}(m) \end{array} $$

where \(\boldsymbol {\alpha }_{k}^{\text {wp}}\) is the wrapped phase and the Δk(m) is given by (32).

$$ {\begin{aligned}{\Delta}_{k}(m)=\left\{ \begin{array}{ll} \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) & \; \; \text{if} \; |\boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m)| \leq 1 \\ 2-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)+\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) & \; \; \text{if} \; \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) > 1 \\ \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m)+2 & \; \; \text{if} \; \boldsymbol{\alpha}_{k}^{\text{uwp}}(m+1)-\boldsymbol{\alpha}_{k}^{\text{uwp}}(m) < -1 \end{array}\right. \end{aligned}} $$

The LS fitting problem is shown as follows

$$ \min_{\mathbf{v}_{k}} \left| \left| \boldsymbol{\Omega}\mathbf{v}_{k}-\boldsymbol{\alpha}_{k}^{\text{uwp}} \right| \right|_{\mathrm{F}}^{2} $$

where \(\mathbf {v}_{k} \in \mathbb {R}^{2 \times 1}\), and

$$\boldsymbol{\Omega}=\left[ \begin{array}{cccc} 0 & 1 & \cdots & M-1\\ 1 & 1 & \cdots & 1 \end{array} \right]{^{\mathrm{T}}} $$

The least square estimation of cosθk is given by the first element of \(\hat {\mathbf {v}}_{k}=\boldsymbol {\Omega }^{\dagger } \boldsymbol {\alpha }_{k}^{\text {uwp}}\), that is to say,

$$\bar{\theta}_{k} = \arccos \hat{\mathbf{v}}_{k}(1) $$

Till now, the joint DOA and DOD estimation of uncorrelated signals has already been achieved. We will in the next stage take advantage of the coherent DOA information, \(\bar {\phi }_{k}\) with k=K1+1,K1+2,,K, to acquire the auto-paired coherent DOD information.

3.3 Oblique projecting for coherent DOD estimation

In order to implement DOD estimation of coherent signals, the elimination of the uncorrelated signals’ contribution in Rx is necessary. It can be achieved by the so-called oblique projection (OP) technique.

An oblique projection [34] is a kind of nonorthogonal projection, e.g., \(\phantom {\dot {i}\!}\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}\), whose range is spanned by D1 and null space is spanned by D2,

$$ \mathbf{P}_{\mathbf{D}_{1}\mathbf{D}_{2}}=\mathbf{D}_{1} \left(\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\mathbf{D}_{2}}^{\bot} \mathbf{D}_{1} \right)^{-1}\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\mathbf{D}_{2}}^{\bot} $$

so that \(\phantom {\dot {i}\!}\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{1}=\mathbf {D}_{1}\) and \(\phantom {\dot {i}\!}\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{2}=\mathbf {0}\). In our discussed scenario, if let D1=BuAu and D2=BcAc, we can construct a virtual observation matrix Y with only coherent signals retaining, i.e.,

$$\begin{array}{*{20}l} \mathbf{Y} \triangleq &(\mathbf{I}_{MN}-\mathbf{P}_{\mathbf{D}_{1}\mathbf{D}_{2}})(\mathbf{R}_{\mathbf{x}}- \hat{\sigma}_{n}^{2} \mathbf{I}_{MN})(\mathbf{I}_{MN}-\mathbf{P}_{\mathbf{D}_{1}\mathbf{D}_{2}})^{\mathrm{H}}\\ =& (\mathbf{B}_{c} \odot \mathbf{A}_{c}) \mathbf{R}_{c} (\mathbf{B}_{c} \odot \mathbf{A}_{c})^{\mathrm{H}} \\ =& (\mathbf{B}_{c} \odot \mathbf{A}_{c}) \mathbf{S}_{c} \end{array} $$

where \({\hat \sigma }_{n}^{2}\) is given by (10).

Hence, the kernel problem is how to design a pragmatic oblique projection. As we know, the calculation of \(\mathbf {P}_{\mathbf {D}_{2}}^{\bot }\) in (34) is impractical due to BcAc is unknown. Literature [35] suggests that if we substitute R for \(\mathbf {P}_{\mathbf {D}_{2}}^{\bot }\), where \(\mathbf {R}^{\dag }=\mathbf {U}_{\mathbf {s}} \boldsymbol {\Sigma }_{\mathbf {s}}^{-1} \mathbf {U}_{\mathbf {s}}^{\mathrm {H}}\), i.e., \(\mathbf {P}_{\mathbf {D}_{1}\mathbf {D}_{2}}=\mathbf {D}_{1} \left (\mathbf {D}_{1}^{\mathrm {H}} \mathbf {R}^{\dag } \mathbf {D}_{1} \right)^{-1}\mathbf {D}_{1}^{\mathrm {H}} \mathbf {R}^{\dag }\), then it works [22, 23]. However, such approximation usually makes a confusion because the power and information of coherent signals contributing to Us are subtracted in Y. In [36], the QR factorization to the cross-covariance matrix of two parallel ULA is utilized to construct oblique projector, which can avoid the above confusion, however it is not appropriate for our case. We herein design a new oblique projector based on the pre-estimated angle information of uncorrelated signals.

3.3.1 New oblique projector

The kernel of this new oblique projector is the estimated DOA information of coherent signals from (21), i.e., \(\left \{\bar {\phi }_{i}\right \}_{i=K_{1}+1}^{K}\).

Based on them, we can reconstruct a K2-order polynomial with roots \(\left \{e^{j \pi \sin {\bar \phi _{i}}}\right \}_{i=K_{1}+1}^{K}\) and K2=KK1, the coefficients of which, for convenience, are defined by vector

$$ \mathbf{h}=[\!h_{0}, h_{1}, \cdots, h_{K_{2}}]^{\mathrm{T}} $$

Similar to (14), a Toeplitz matrix \(\mathbf {G}_{\mathbf {h}} \in \mathbb {C}^{N \times (N-K_{2})}\) can be utilized to assist in constructing oblique projector because it satisfies \(\mathbf {G}_{\mathbf {h}}^{\mathrm {H}}\mathbf {B}_{c}=\mathbf {0}\). If defining \(\bar {\mathbf {G}}=\mathbf {G}_{\mathbf {h}} \otimes \mathbf {I}_{M} \), then we have

$$ \bar{\mathbf{G}}^{\mathrm{H}}\mathbf{D}_{2}=\bar{\mathbf{G}}^{\mathrm{H}} (\mathbf{B}_{c} \odot \mathbf{A}_{c})=(\mathbf{G}_{\mathbf{h}}^{\mathrm{H}} \mathbf{B}_{c}) \odot \mathbf{A}_{c}=\mathbf{0}. $$

That is to say, it can allow us to substitute \(\mathbf {P}_{\bar {\mathbf {G}}}\) for \(\mathbf {P}_{\mathbf {D}_{2}}^{\bot }\) in (34), so that an alternative choice of oblique projector is generated, i.e.,

$$ \bar{\mathbf{P}}_{\mathbf{D}_{1}\mathbf{D}_{2}}=\mathbf{D}_{1} \left(\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\bar{\mathbf{G}}} \mathbf{D}_{1} \right)^{-1}\mathbf{D}_{1}^{\mathrm{H}} \mathbf{P}_{\bar{\mathbf{G}}} $$

where \(\mathbf {P}_{\bar {\mathbf {G}}}=\bar {\mathbf {G}}(\bar {\mathbf {G}}^{\mathrm {H}}\bar {\mathbf {G}})^{-1}\bar {\mathbf {G}}^{\mathrm {H}}\). One can easily examine the properties that \(\bar {\mathbf {P}}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{1}=\mathbf {D}_{1}\) and \(\bar {\mathbf {P}}_{\mathbf {D}_{1}\mathbf {D}_{2}}\mathbf {D}_{2}=\mathbf {0}\).

3.3.2 Forward-backward spatial smoothing

Taking (38) into (35), we can get that virtual observation Y. Given that rank{Sc}=rank{Rc} if BcAc is a full column rank matrix and Rc is a rank-deficient matrix due to the coherent signals, therefore, we have to adopt two-dimensional spatial smoothing technique. Define a series of selection matrices, n=1,2,,NZ1+1, m=1,2,,MZ2+1,

$$ \begin{aligned} \mathbf{\Gamma}_{n, m} =& \left[ {\mathbf{0}_{Z_{1} \times (n-1)}} \; \mathbf{I}_{Z_{1}} \ {\mathbf{0}_{Z_{1} \times (N-Z_{1}-n+1)}}\right]\\ &\otimes \left[\mathbf{0}_{Z_{2} \times (m-1)} \; \mathbf{I}_{Z_{2}} \ {\mathbf{0}_{Z_{2} \times (M-Z_{2}-m+1)}}\right] \end{aligned} $$

where Z1<N and Z2<M denote the length of receive and transmit subarrays, respectively.

After stacking Γn,mY as the following style

$$\begin{aligned} \left[\mathbf{\Gamma}_{1,1} \mathbf{Y} \ \cdots \ \mathbf{\Gamma}_{1, M-Z_{2}+1} \mathbf{Y} \ \ \mathbf{\Gamma}_{2,1} \mathbf{Y} \ \cdots \ \right. \\ \left.\mathbf{\Gamma}_{2, M-Z_{2}+1} \mathbf{Y} \ \cdots \ \mathbf{\Gamma}_{N-Z_{1}+1, M-Z_{2}+1} \mathbf{Y} \right] \end{aligned} $$

it holds

$$ \bar{\mathbf{Y}} \triangleq \left[\mathbf{B}_{c}^{(Z_{1})} \odot \mathbf{A}_{c}^{(Z_{2})}\right] \bar{\mathbf{S}}_{c}. $$

Note that \(\bar {\mathbf {S}}_{c} \in \mathbb {C}^{K_{2} \times (N-Z_{1}+1)(M-Z_{2}+1)MN}\) is a random phase modulated signal matrix which in turn de-correlated the rank-deficient Sc, so it is a full row rank matrix. In addition, similar to the conventional smoothing technique, the choices of Z1 and Z2 depend mainly on the number of coherent signals.

Utilizing the noise subspace of \(\bar {\mathbf {Y}}\), i.e., \(\bar {\mathbf {U}}_{0}\), we can obtain the estimated auto-paired transmit steering vectors for the coherent signals,

$$ \bar{\mathbf{a}}_{c}({\theta}_{k})=\frac{\bar{\mathbf{E}}(\bar{\phi}_{k})^{-1}\bar{\mathbf{e}}}{\bar{\mathbf{e}}^{\mathrm{T}}\bar{\mathbf{E}}(\bar{\phi}_{k})^{-1}\bar{\mathbf{e}}}, \ \ k=K_{1}+1, \cdots, K $$

where \(\bar {\mathbf {E}}=\left [\bar {\mathbf {b}}({\phi }) \otimes \mathbf {I}_{Z_{2}}\right ]^{\mathrm {H}} {\bar {\mathbf {U}}_{\mathbf {0}}}{\bar {\mathbf {U}}_{\mathbf {0}}}^{\mathrm {H}} \left [\bar {\mathbf {b}}({\phi }) \otimes \mathbf {I}_{Z_{2}}\right ]\) with \(\bar {\mathbf {b}}({\phi })=\mathbf {b}^{(Z_{1})}({\phi })\), and \(\bar {\mathbf {e}}=\mathbf {e}^{(Z_{1})}\).

We have introduced the complete descriptions of the proposed WSFOPDE approach. For convenience, it is summarized in Table 1.

Table 1 Summary of the proposed WSFOPDE approach

4 The improved WSFOPDE approach

As we can see, the aforementioned approach estimates the DOD information on the premise that the DOA information are successfully obtained, seeing (28) and (41). That is to say, it is a successive manner. The accuracy of DOA estimation has a direct influence on that of DOD estimation. Therefore, in order to avoid the error propagation and make a further effort to improve the accuracy of DOA estimation, we then design an improved version.

Actually, the local spectrum searching can improve the estimation accuracy of uncorrelated DOA and coherent DOA. With the initial uncorrelated DOA estimation \(\bar \phi _{i}, k=1, \cdots, K_{1}\), we can further refine them by

$$ {\hat \phi}_{k}=\arg \max_{\phi \in [{\bar \phi}_{k}-{\Delta \phi}, {\bar \phi}_{k}+{\Delta \phi}]} \mathbf{e}^{\mathrm{T}} \mathbf{E}^{-1}(\phi)\mathbf{e}, $$

where Δϕ is a small positive value, and \({\hat \phi }_{k}\) represent the final DOA estimation of k-th uncorrelated signal. Similarly, with the initial coherent DOA estimation \(\bar \phi _{i}, k=K_{1}+1, \cdots, K\), we can also refine them by

$$ \hat{\phi}_{k}=\arg \max_{\phi \in \left[\bar{\phi}_{k}-{\Delta \phi}, \bar{\phi}_{k}+{\Delta \phi}\right]} \bar{\mathbf{e}}^{\mathrm{T}} \bar{\mathbf{E}}^{-1}(\phi)\bar{\mathbf{e}} $$

where Δϕ is a small positive value, and \(\hat {\phi }_{k}\) represent the final DOA estimation of k-th coherent signal. Accordingly, the estimated auto-paired steering vectors for uncorrelated and coherent signals in Table 1, i.e., \({\bar {\mathbf {a}}_{u}({\theta }_{k})}\) and \(\bar {\mathbf {a}}_{c}({\theta }_{k})\), are replaced by \({\hat {\mathbf {a}}_{u}({\theta }_{k})}\) and \(\hat {\mathbf {a}}_{c}({\theta }_{k})\). Based on that, we can further rebuild an improved WSFOPDE approach that is shown in Table 2.

Table 2 Summary of the improved WSFOPDE approach

5 Results and discussion

5.1 Qualitative discussion

Although the proposed approaches make a aperture loss of transmit array for assisting coherent DOD estimation, seeing the reduced MIMO array in (40), they are still superior to some other approaches such as the 2D spatial smoothing + PARAFAC or 2D spatial smoothing + 2D-MUSIC/RD-MUSIC, because the latter ones sacrifice the array aperture on both receive and transmit ends to compensate the rank deficiency while the WSFOPDE and improved WSFOPDE approaches utilize the whole receive array to achieve DOA estimation.

For the improved WSFOPDE approach, the weighted subspace fitting technique not only provides initial DOA estimation for uncorrelated signals, but also gives a much accuracy DOA estimation for coherent signals. To do so, we avoid using \(\bar {\mathbf {Y}}\) to perform joint DOA and DOD estimation of coherent signals. On the other hand, if borrowing the idea as introduced in [23], one also can use maximum likelihood, 2D-MUSIC or RD-MUSIC approaches to directly estimate the uncorrelated signals and then construct Toeplize matrix or oblique projection matrix to estimate coherent signals. Obviously, such scheme will cost much larger computation flops in spectrum searching. But the proposed one preserves the estimation accuracy of uncorrelated signals as done in [19, 20] and also can implement joint DOD and DOA estimation for coherent signals in a lower computational complexity, which is more attractable.

In addition, there are some differences between the proposed approaches and the ones in [13, 14]. First, although the researches in both references have made a detailed discussion on the estimation of DOA and delay in massive MIMO/FD-MIMO systems, they did not pay attention to the scenario of uncorrelated and coherent signals coexisting. Second, although the adopted ESPRIT-type algorithms in [13, 14] are also low-complexity, they were not based on weighted subspace fitting. Third, if such ESPRIT-type algorithm is utilized for mixed signals, it cannot isolate the coherent signals. Therefore, the proposed approaches maintain their own advantages.

5.1.1 Computational complexity

We now make a detailed analysis for the computational complexity of the proposed approaches. It is shown in Table 3. One flop is defined as one-time complex multiplication according to [37]. In WSFOPDE approach, the computational complexity during the minimization of (21) in Step 3 is based on [38]. The Step 5 contains the discerning operation and oblique projector construction. Besides, the improved WSFOPDE approach adds two extra steps to refine the DOA estimation, seeing the Table 2. Among them, Step 6 adds \(\mathcal {O} \left \{K_{1}{\bar n}_{1}\left [\left (M^{2}N+M^{2}\right)(MN-K_{1}-Q)+ M^{3}\right ]\right \}\) and Step 9 costs \(\mathcal {O} \left \{K_{2}{\bar n}_{2}\left [Z_{2}^{2}(Z_{1}+1)(Z_{1}Z_{2}-K_{2})+Z_{2}^{3}\right ] \right \}\), where \({\bar n}_{1}\) and \({\bar n}_{2}\) are the total searching number in each local angle domain, respectively.

Table 3 Analysis of the computational complexity

For easy to compare, we herein consider the RD-MUSIC + oblique projection + SS + RD-MUSIC algorithm, in which the discerning of uncorrelated signals is easily achieved by the angles with respect to K1 largest spectrum peak or by calculating (25). The computational complexity is analyzed as follows: the uncorrelated DOA and DOD estimation require \(\mathcal {O} \!\left \{\frac {1}{2}TM^{2} N^{2} \,+\, M^{3} N^{3} \,+\,(K_{1}\,+\,K\,+\,{\tilde n_{1}}) \left [\left (M^{2}N\!+M^{2}\right)(MN-{K}_{1}-Q)+M^{3}\right ]\right \}\), where \({\tilde n}_{1}\) is the total searching number in the whole angle domain; the same as the proposed one, oblique projection and SS require \(\mathcal {O}\left \{M^{3}(N\,-\,K_{2})\left [N^{2}\,+\,2N(N\,-\,K_{2})\,+\,\left (N-K_{2}\right)^{2}\right ]+3M^{3}N^{3} +2K_{1}M^{2}N^{2} +2K_{1}^{2}MN+K_{1}^{3}\right \}\); the coherent DOA and DOD estimation require \(\mathcal {O}\! \left \{ \frac {1}{2}(N-Z_{1}+1)(M-Z_{2}+1)MN {Z}_{1}^{2}Z_{2}^{2}\,+\, Z_{1}^{3}Z_{2}^{3} \,+\, (K_{2}\,+\,{\tilde n_{2}}) \left [Z_{2}^{2}(Z_{1}\,+\,1)(Z_{1}Z_{2}-K_{2})\,+\,Z_{2}^{3}\right ] \right \}\), where \({\tilde n}_{2}\) is the total searching number in the whole angle domain.

If taking some typical values of parameters into account, e.g., M=N=9,T=500, K=6,K1=2,Q=1,K2=4, \({\tilde n}_{1}={\tilde n}_{2}=180^{\circ }/0.01^{\circ }\), Z1=Z2=6. The local searching range in the improved WSEOPDE approach is set as Δϕ=2 so that \({\bar n}_{1}={\bar n}_{2}=2^{\circ }/0.01^{\circ }\). By some tedious calculations, we can see that the proposed WSFOPDE approach has a computational complexity of \(\mathcal {O} \{6.50 \times 10^{6}\}\) and its improved version has a computational complexity of \(\mathcal {O} \left \{3.87 \times 10^{7}\right \}\), whereas the compared one is of \(\mathcal {O} \left \{1.31 \times 10^{9}\right \}\). That is to say, our proposed approaches are computationally efficient. We also evaluate the average running time of executing one-time algorithm, one can refer to the simulation Example 1.

5.1.2 Parameter identifiability

Given that the parameterization in (14) and the rank of G in (16), it is necessary for the proposed approaches that K+1<N. Different from the theoretical analysis in [2224] that the number of identifiable signals, including the uncorrelated and coherent signals, is beyond the number of actual antennas, the proposed approaches are of inferiority.

Actually, based on the aforementioned analysis, we can conclude that (1) the proposed approaches make a tradeoff between the array aperture and the computational complexity in comparison to its competitors and (2) the proposed approaches are very suitable for the case of massive arrays. It is the peculiarity of “massive” that allows the proposed ones to manifest a great alleviation in computational burden, i.e., the DOA estimation is achieved by polynomial rooting rather than the global searching [23]. Therefore, the proposed approaches can be viewed as two typical alternatives for providing a relatively higher accuracy of angle estimation but simultaneously consuming a lower computational complexity.

5.2 Quantitative results

In this section, we present some numerical simulations to demonstrate the effectiveness and advantages of the proposed approaches. The antenna spacing in both transmitting and receiving arrays is half the carrier wavelength, i.e., d=λ/2. The average root mean squared error (RMSE) is used to assess the performance of angle estimation, which is defined as

$$\frac{1}{\mathcal{K}} \sum_{k=1}^{\mathcal{K}} \sqrt {E \big [(\hat \theta_{k}- \theta_{k})^{2} + (\hat \phi_{k}- \phi_{k})^{2} \big ]}. $$

where \(\mathcal {K}=K_{1}\) for uncorrelated signals and \(\mathcal {K}=K_{2}\) for coherent signals. The total number of Monte Carlo simulations is set to be 200. The signal-to-noise ratio (SNR) is defined by (1), i.e.,

$$\text{SNR}=10\log_{10} \frac{\sum_{t} \left\|(\mathbf{B}\odot\mathbf{A})\mathbf{s}(t)\right\|^{2}}{MN \sigma_{n}^{2}} $$

For performance comparison, some typical algorithms such as PARAFAC algorithm [17] and ESPRIT algorithm are calculated after performing spatial smoothing. In addition, we also consider the deflation method [23], which is a typical algorithm of utilizing the MUSIC and oblique projector (it is abbreviated to ‘MUSIC+OP’ in the following content for convenience). In this deflation method, we purposely confine the angle searching range to four degrees for avoiding the huge calculation burden. All the results shown below are based on 200 independent trials, and the search procedure runs with a step size of 0.01.

Example 1

Firstly, we examine the estimation performance with respect to SNR. Considering a scenario that contains two uncorrelated signals with β1=β2=1 and (θ,ϕ){(30,35),(40,45)}, one group of four coherent signals with γ11=ejπ/8, γ12=0.8ej5π/9, γ13=0.7ejπ/18, γ14=0.5ej8π/9 and (θ,ϕ){(−25,−20),(−15,−10),(10,5),(15,25)}. We adopt M=N=9 and the total number of array snapshots T=500. The sub-array length that is used for dealing with the coherent signals is set as Z1=Z2=8. From Figs. 1 and 2, we can observe that, the proposed WSFOPDE approach overmatches the other two non-searching algorithms, i.e., FBSS+ESPRIT and FBSS+PARAFAC (it is conducted by periodogram-based algorithm with 1024-point FFT after getting the auto-paired array steering vectors through tensor decomposition); and the proposed improved WSFOPDE approach performs nearly the same as the MUSIC+OP. In addition, the local searching indeed improves the accuracy of angle estimation with at least 5 dB leading than the proposed WSFOPDE approach for both uncorrelated and coherent signals.

Fig. 1
figure 1

The performance comparison of DOA-DOD estimation between several typical algorithms: uncorrelated signals

Fig. 2
figure 2

The performance comparison of DOA-DOD estimation between several typical algorithms: coherent signals

We also consider the time efficiency of different algorithms. Table 4 gives the average running time for executing one-time algorithm, where the MATLAB codes are executed in a PC with Intel(R) Core(TM) i5-6400 CPU@2.6GHz and 8 GB RAM. From the statistical results, we can see that, among the non-searching schemes, the WSFOPDE approach manifests a moderate time-consuming; whereas for the searching schemes, the improved WSFOPDE approach shows a great decrease in total computational complexity.

Table 4 The average time of executing one-time algorithm

Therefore, from a joint perspective of computational complexity and angle estimation accuracy, both proposed approaches can serve as better alternatives as compared to the existing competitors.

Example 2

Then, we evaluate the estimation performance versus the number of snapshots. The simulation conditions are similar to those in the previous example, except that the number of snapshots is set from 500 to 3000 and the SNR is fixed at 0 dB. We herein do not consider the FBSS+PARAFAC algorithm because of its unsatisfactory performance. The simulation results, as shown in Figs. 3 and 4, illustrate again that the average RMSE of the improved WSFOPDE approach is very close to the MUSIC+OP algorithm.

Fig. 3
figure 3

The DOA-DOD estimation performance versus the number of snapshots: uncorrelated signals

Fig. 4
figure 4

The DOA-DOD estimation performance versus the number of snapshots: coherent signals

Besides, it is shown that the FBSS+ESPRIT has a lower average RMSE for the coherent signals when the number of snapshot is less than 103, but when comparing the extent of improvement with the increasing number of snapshots, i.e., the slope of the average RMSE curves, the WSFOPDE approach exhibits much better promotion than the FBSS+ESPRIT.

In order to assess the proposed approaches more comprehensively, in the following simulations, we focuss on two factors, i.e., the angular separation and the number of antennas.

Example 3

We also test the estimation performance in terms of the angular separation between the DOA of uncorrelated and coherent signals. Herein, one uncorrelated signal comes from (50,η1), and one group of two coherent signals come from {(15,20),(30,η2)} with the attenuation coefficient vector [1,ejπ/3]T, where η1=η2+Δη, η2=40, and Δη is varied from 2 to 16 with 2 step. Other related parameters are set as the number of antennas M=N=7, the subarray size Z1=Z2=5, the number of snapshots T=500, and the SNR is fixed at 5 dB.

The simulation curves in Figs. 5 and 6 illustrate that the angle distinguishing ability of both approaches are restricted due to the closely spaced uncorrelated signal s1(t) and coherent signal s3(t); therefore, they all show unsatisfactory performance with larger average RMSE, especially for the angle estimation of coherent signals. With the angular separation becoming wider and wider, the performance is getting better; and basically, although the improved WSFOPDE approach requires more calculations, it is superior to its predecessor in all level of angular separation.

Fig. 5
figure 5

The performance variation with angular separation between the DOA of mixed signals: uncorrelated signals

Fig. 6
figure 6

The performance variation with angular separation between the DOA of mixed signals: coherent signals

Example 4

Besides the angular separation, the behavior of coherent signals will affect the performance of angle estimation. Herein, we mainly focuss on the sub-signals number of one group of coherent signals. In this example, there exist one uncorrelated signals and one group of coherent signals. The case 1 includes 4 sub-signals, and the case 2 includes 2 sub-signals. We restrict the DOA and DOD of coherent signals to the uniform-linear distribution in a fixed angular range [5,35] and [0,30], respectively. The uncorrelated signal is fixed at (θ,ϕ)=(40,45) in both cases. Other related parameters are set as: the number of antennas M=N=12 and T=500. Figures 7 and 8 illustrate the performance variation. As we can see, the increase of sub-signals number in one group of coherent signals will cause RMSE performance deterioration in angle estimation. Such deterioration in the RMSE of coherent signals is greater that the one of uncorrelated signals. In this sense, for ultra-high identifiability, the massive arrays are necessary.

Fig. 7
figure 7

The RMSE performance variation under different sub-signals number in one group of coherent signals: uncorrelated signals

Fig. 8
figure 8

The RMSE performance variation under different sub-signals number in one group of coherent signals: coherent signals

Example 5

Finally, we examine the average RMSE performance with different number of M and N. Herein we arrange M=N. In this example, the considered scenario includes one uncorrelated signal with β1=1 and (θ,ϕ)=(10,45), two group of four coherent signals with γ11=1, γ12=0.5ejπ/16, γ21=1, γ22=0.75ejπ/7 and (θ,ϕ){(30,35),(−20,15),(50,−15),(10,−50)}. In addition, the number of snapshots is set as T=500 and the SNR is fixed at 5dB.

It clearly manifests in Figs. 9 and 10 that the accuracy of DOA-DOD estimation is gradually improved with the increase of antenna number for both uncorrelated and coherent signals.

Fig. 9
figure 9

The RMSE performance with different configuration of M and N: uncorrelated signals

Fig. 10
figure 10

The RMSE performance with different configuration of M and N: coherent signals

6 Conclusions

In this paper, two computationally efficient approaches based on weighted subspace fitting and oblique projection, called WSFOPED and improved WSFOPED, were proposed for the joint DOA and DOD estimation of a mixture of uncorrelated and coherent narrowband signals in MIMO array systems, where the estimated DOA and DOD information is pair-matched automatically. The whole procedure includes three stages: polynomial rooting, uncorrelated DOA discerning and transmit steering vectors estimating, and virtual MIMO array data constructing via oblique projection. By systematical analysis of the computational complexity and sufficient simulation examples, the effectiveness of the proposed approaches were verified and they can be considered as better alternatives in two-dimensional spectrum estimation.


  1. The method of direction estimation (MODE) is first proposed by P. Stoica and K. C. Sharman in [33]







Degree of freedom


Eigenvalue decomposition


Forward-backward spatial smoothing


Least squares


Multiple-input multiple-output


Oblique projection


Radar cross section


Root mean squared error


Signal-to-noise ratio


Weighted subspace fitting and oblique projection based approaches for two-dimensional direction estimation


  1. J. Li, P. Stoica, Mimo radar with colocated antennas. IEEE Signal Process. Mag.24(5), 106–114 (2007).

    Article  Google Scholar 

  2. A. M. Haimovich, R. Blum, L. Cimini, Mimo radar with widely separated antennas. IEEE Signal Process. Mag.25(1), 116–129 (2008).

    Article  Google Scholar 

  3. D. Ciuonzo, G. Romano, R. Solimene, Performance analysis of time-reversal music. IEEE Trans. Signal Process.63(10), 2650–2662 (2015).

    Article  MathSciNet  Google Scholar 

  4. F. K. Gruber, E. A. Marengo, A. J. Devaney, Time-reversal imaging with multiple signal classificationconsiderring multiple scattering between the targets. J. Acoust. Soc. Am.115(6), 3042–3047 (2004).

    Article  Google Scholar 

  5. D. Ciuonzo, P. S. Rossi, Noncolocated time-reversal music: High-SNR distribution of null spectrum. IEEE Signal Process. Lett.24(4), 397–401 (2017).

    Article  Google Scholar 

  6. D. Ciuonzo, On time-reversal imaging by statistical testing. IEEE Signal Process. Lett.24(7), 1024–1028 (2017).

    Article  Google Scholar 

  7. J. Li, P Stoica, L. Z. Xu, W. Roberts, On parameter identifiability of MIMO radar. IEEE Signal Process. Lett.14(12), 968–971 (2007).

    Article  Google Scholar 

  8. B. B. Yao, W. J. Wang, W. Han, Q. Y. Yin, Distributed angle estimation by multiple frequencies synthetic array in wireless sensor localization system. IEEE Trans. Wireless Commun.13(2), 876–887 (2014).

    Article  Google Scholar 

  9. B. B. Yao, W. J. Wang, Q. Y. Yin, Dod and doa estimation in bistatic non-uniform multiple-input multiple-output radar systems. IEEE Commun. Lett.16(11), 1796–1799 (2012).

    Article  Google Scholar 

  10. L. Lu, G. Y. Li, A. L. Swindlehurst, A. Ashikhmin, R. Zhang, An overview of massive mimo: benefits and challenges. IEEE J. Sel. Topics Signal Process.8(5), 742–758 (2014).

    Article  Google Scholar 

  11. W. L. Zhang, F. F. Gao, Blind frequency synchronization for multiuser ofdm uplink with large number of receive antennas. IEEE Trans. Signal Process.64(9), 2255–2268 (2016).

    Article  MathSciNet  Google Scholar 

  12. W. L. Zhang, F. F. Gao, H. Minn, H. M. Wang, Scattered pilot-based frequency synchronization for multiuser OFDM systems with large number of receive antennas. IEEE Trans. Commun.65(4), 1733–1745 (2017).

    Article  Google Scholar 

  13. R. Shafin, L. J. Liu, J. Z. Zhang, Y. -C. Wu, Doa estimation and capacity analysis for 3D millimeter wave massive MIMO/FD-MIMO OFDM systems. IEEE Trans. Wireless Commun.15(10), 6963–6978 (2016).

    Article  Google Scholar 

  14. R. Shafin, L. J. Liu, Y. Li, A. D. Wang, J. Z. Zhang, Angle and delay estimation for 3D massive mimo systems based on parametric channel modeling. IEEE Trans. Wireless Commun.16(8), 5370–5383 (2017).

    Article  Google Scholar 

  15. Y. H. Cao, Joint estimation of angle and Doppler frequency for bistatic MIMO radar. Electron. Lett.46(2), 170–172 (2010).

    Article  Google Scholar 

  16. G. M. Zheng, B. X. Chen, M. L. Yang, Unitary esprit algorithm for bistatic MIMO radar. Electron. Lett.48(3), 179–181 (2012).

    Article  Google Scholar 

  17. D. Nion, N. D. Sidiropoulos, Tensor algebra and multidimensional harmonic retrieval in signal processing for MIMO radar. IEEE Trans. Signal Process.58(11), 5693–5705 (2010).

    Article  MathSciNet  Google Scholar 

  18. Y. B. Cheng, R. S. Yu, H. Gu, W. M. Su, Multi-SVD based subspace estimation to improve angle estimation accuracy in bistatic MIMO radar. Signal Process.93(7), 2003–2009 (2013).

    Article  Google Scholar 

  19. X. F. Zhang, L. Y. Xu, L. L. Xu, D. Z. Xu, Direction of departure (DOD) and direction of arrival (DOA) estimation in mimo radar with reduced-dimension music. IEEE Commun. Lett.14(12), 1161–1163 (2010).

    Article  Google Scholar 

  20. J. F. Li, X. F. Zhang, Improved joint DOD and DOA estimation for MIMO array with velocity receive sensors. IEEE Signal Process. Lett.18(12), 717–720 (2011).

    Article  Google Scholar 

  21. S. U. Pillai, B. H. Kwon, Forward/backward spatial smoothing techniques for coherent signa identification. IEEE Trans. Acoust. Speech Singal Process.7(1), 8–15 (1898).

    MATH  Google Scholar 

  22. X. Xu, Z. F. Ye, C. Q. Chang, Method of direction-of-arrival estimation for uncorrelated, partially correlated and coherent sources. IET Microw. Antennas Propag.1(4), 949–954 (2007).

    Article  Google Scholar 

  23. X. Xu, Z. F. Ye, Y. F. Zhang, C. Q. Chang, A deflation approach to direction of arrival estimation for symmetric uniform linear array. IEEE Antennas Wirel. Propaga. Lett.5(1), 486–489 (2006).

    Article  Google Scholar 

  24. F. L. Liu, J. K. Wang, C. Y. Sun, R. Y. Du, Spatial differencing method for DOA estimation under the coexistence of both uncorrelated and coherent signals. IEEE Trans. Antennas Propaga.60(4), 2052–2062 (2012).

    Article  MathSciNet  Google Scholar 

  25. Y. F. Zhang, Z. F. Ye, Efficient method of doa estimation for uncorrelated and coherent signals. IEEE Antenna Wirel. Propag. Lett.7:, 799–802 (2008).

    Article  Google Scholar 

  26. W. Si, P. Zhao, Z. Qu, Two-dimensional doa and polarization estimation for a mixture of uncorrelated and coherent sources with sparsely-distributed vector sensor array. Sensors. 16(1), 789 (2016).

    Article  Google Scholar 

  27. F. Wang, X. W. Cui, M. Q. Lu, Z. M. Feng, Decoupled 2D direction-of-arrival estimation based on sparse signal reconstruction. EURASIP J. Adv. Signal Process.7:, 1–16 (2015).

    Google Scholar 

  28. T. J. Shan, A. Paulraj, T. Kailath, On smoothed rank profile tests in eigenstructure methods for directions-of-arrival estimation. IEEE Trans. Acoust. Speech Signal Process.35(10), 1377–1385 (1987).

    Article  Google Scholar 

  29. J. He, M. N. S. Swamy, M. O. Ahmad, Joint DOD and DOA estimation for MIMO array with velocity receive sensors. IEEE Signal Process. Lett.18(7), 399–402 (2011).

    Article  Google Scholar 

  30. R. Kumaresan, A. K. Shaw, Superresolution by structured matrix approximation. IEEE Trans. Antennas Propag.36(1), 34–44 (1998).

    Article  Google Scholar 

  31. H. L. Van Trees, Optimum array processing part IV of detection, estimation, and modulation theory (John Wiley & Sons Press, New York, 2002).

    Google Scholar 

  32. B. B. Yao, W. L. Zhang, Q. S. Wu, Weighted subspace fitting for two-dimension DOA estimation in massive MIMO systems. IEEE Access. 5(1), 14020–14027 (2017).

    Article  Google Scholar 

  33. P. Stoica, K. C. Sharman, Novel eigenanalysis method for direction estimation. Proc. Inst. Elect. Eng., pt. F.13(1), 19–26 (1990).

    MathSciNet  Google Scholar 

  34. R. T. Behrens, L. L. Scharf, Signal processing applications of oblique projection operators. IEEE Trans. Signal Process.42(6), 1413–1424 (1994).

    Article  Google Scholar 

  35. M. L. McCloud, L. L. Scharf, A new subspace identification algorithm for high resolution DOA estimation. IEEE Trans. Antennas Propag.50(10), 1382–1390 (2002).

    Article  MathSciNet  Google Scholar 

  36. H. Tao, J. M. Xin, J. S. Wang, N. N. Zheng, A. Sano, Two-dimensional direction estimation for a mixture of noncoherent and coherent signals. IEEE Trans. Signal Process.63(2), 318–333 (2015).

    Article  MathSciNet  Google Scholar 

  37. G. H. Golub, C. F. Van Loan, Matrix Computation 3rd Ed (John Hopkins University Press, Baltimore, 1996).

    Google Scholar 

  38. J. Li, P. Stoica, Z. S. Liu, Comparative study of IQML and MODE direction-of-arrival estimators. IEEE Trans. Signal Process.46(1), 149–160 (1998).

    Article  Google Scholar 

Download references


The authors would like to thank the editor and anonymous reviewers for their constructive comments which helped to improve the quality of this paper.


This work has been supported by the National Key R&D Program of China (Grant No. 2018YFC0808706), National Natural Science Foundation of China (Grant No. 61601058), the the Youth Talent Promotion Plan of Shaanxi Association for Science and Technology (Grant No. 20170508), and the Fundamental Research Funds for the Central Universities (Grant No. 310832173701).

Availability of data and materials

We declare that materials described in the manuscript, including all relevant raw data, will be freely available to any scientist wishing to use them for non-commercial purposes, without breaching participant confidentiality.

Author information

Authors and Affiliations



BY conceived the work, designed the experiments, and wrote the paper. ZD performed the experiments and wrote the paper. WL analyzed the simulation results. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Bobin Yao.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yao, B., Dong, Z. & Liu, W. Effective joint DOA-DOD estimation for the coexistence of uncorrelated and coherent signals in massive multi-input multi-output array systems. EURASIP J. Adv. Signal Process. 2018, 64 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: