### 3.1 Beam-by-beam matched filter (BBB-MF)

We first discard the DMI and IUI, then (9) can be rewritten as:

$$\begin{aligned} H_{n}^{k} & \approx \sum\limits_{i = 1}^{{N_{t} }} {A_{i}^{e} {\mathbf{a}}^{T} (\theta_{i}^{e} ){\mathbf{w}}_{1} s_{1}^{k} (n)e^{{ - j2\pi n\Delta f\tau_{i}^{e} }} e^{{j2\pi f_{d,i} kT_{o} }} } \\ & \quad { + }\sum\limits_{b = 2}^{{N_{w} }} {\sum\limits_{i = 1}^{{N_{t} }} {A_{i}^{e} {\mathbf{a}}^{T} (\theta_{i}^{e} ){\mathbf{w}}_{b} s_{b}^{k} (n)e^{{ - j2\pi n\Delta f\tau_{i}^{e} }} e^{{j2\pi f_{d,i} kT_{o} }} } } + n_{n}^{k} \\ \end{aligned}$$

(10)

Since \({\mathbf{w}}_{1} s_{1}^{k} (n)\) is the signal transmitted to the first beam sector, where the radar receiver is located, the direct-path signal associated with \({\mathbf{w}}_{1} s_{1}^{k} (n)\) usually has large SINR. Therefore, we assume that \(s_{1}^{k} (n)\) can be decoded by the radar receiver. With \(s_{1}^{k} (n)\), the matched filter of the first beam sector is developed as:

$$\chi_{1} (\tau ,f_{d} ) = \sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2 - 1} {H_{n}^{k} } \left( {s_{1}^{k} (n)} \right)^{*} e^{j2\pi n\Delta f\tau } e^{{ - j2\pi f_{d} kT_{o} }} }$$

(11)

where \(\left( \cdot \right)^{*}\) means taking the complex conjugate of the entity. \(K\) is the number of OFDM blocks within the coherent processing interval (CPI). It is ready to see that (11) can be implemented with the 2D Fourier transform of \(H_{n}^{k}\) compensated by \(\left( {s_{1}^{k} (n)} \right)^{*}\). By properly choosing the numbers of subcarriers and OFDM blocks for processing, fast Fourier transform (FFT) can be used to increase the computational efficiency.

It is noted that the SSM proposed in [40] first removes data symbols from the channel estimate and then performs the 2D FFT. The SSM is expressed as:

$$\chi_{1}^{{{\text{strip}}}} (\tau ,f_{d} ) = \sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2} {\frac{{H_{n}^{k} }}{{s_{1}^{k} (n)}}} e^{j2\pi n\Delta f\tau } e^{{ - j2\pi f_{d} kT_{o} }} }$$

(12)

It is seen from (10) and (12) that the data symbols cannot be removed completely in the MU-MIMO-OFDM passive radar. Moreover, the SSM can amplify the noise, as can be seen in the later simulations.

In the following, we take a deep insight into (11), which is rewritten as:

$$\begin{gathered} \chi_{1} (\tau ,f_{d} ) \hfill \\ = \sum\limits_{i = 1}^{{N_{t} }} {A_{i}^{e} {\mathbf{a}}^{T} (\theta_{i}^{e} ){\mathbf{w}}_{1} \sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2 - 1} {\left| {s_{1}^{k} (n)} \right|^{2} e^{{ - j2\pi n\Delta f(\tau_{i}^{e} - \tau )}} e^{{j2\pi (f_{d,i} - f_{d} )kT_{o} }} } } } \hfill \\ { + }\sum\limits_{b = 2}^{{N_{w} }} {\sum\limits_{i = 1}^{{N_{t} }} {A_{i}^{e} {\mathbf{a}}^{T} (\theta_{i}^{e} ){\mathbf{w}}_{b} \sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2 - 1} {\left( {s_{1}^{k} (n)} \right)^{*} s_{b}^{k} (n)e^{{ - j2\pi n\Delta f(\tau_{i}^{e} - \tau )}} e^{{j2\pi (f_{d,i} - f_{d} )kT_{o} }} } } } } + n_{g} \hfill \\ \end{gathered}$$

(13)

It is reasonable to assume that the data symbols transmitted are random. That is, \(s_{b}^{k} (n)\) is statistically independent for different subcarrier *n*, OFDM block *k*, and beam sector *b*. For a specific modulation, such as the quadrature phase shift keying (QPSK), 64-quadrature amplitude modulation (64-QAM), and 256-QAM, we have

$$\sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2 - 1} {\left( {s_{1}^{k} (n)} \right)^{*} s_{b}^{k} (n)} \ll } \sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2 - 1} {\left| {s_{1}^{k} (n)} \right|^{2} } }$$

(14)

Then, the second term in (13) is significantly smaller than the first term and can be discarded. That is, the influence of the target-reflected signals corresponding to the transmitted signals to the other beam sectors can be ignored. For a specific target located in the delay and Doppler cell \((\tau_{l}^{e} ,f_{d,l} )\), the matched filter result at this cell can be represented as:

$$\chi_{1} (\tau_{l}^{e} ,f_{d,l} ){ = }A_{l}^{e} {\mathbf{a}}^{T} (\theta_{l}^{e} ){\mathbf{w}}_{1} \sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2 - 1} {\left| {s_{1}^{k} (n)} \right|^{2} } } + n_{1} (\tau_{l}^{e} ,f_{d,l} )$$

(15)

where \(\sum\limits_{k = 1}^{K} {\sum\limits_{n = - N/2}^{N/2 - 1} {\left| {s_{1}^{k} (n)} \right|^{2} } }\) corresponds to the integration gain, the magnitude of which is proportional to the product of the signal bandwidth and length. For 5G signal with typical bandwidth of 50–100 MHz, this integration gain can be very high. \({\mathbf{a}}^{T} (\theta_{l}^{e} ){\mathbf{w}}_{1}\) corresponds to the gain of beamforming. Specifically, for an M-element antenna array, the SNR increase because of the beamforming can be expressed as \(20\log \left( {\left| {{\mathbf{a}}^{T} (\theta_{l}^{e} ){\mathbf{w}}_{1} } \right|/\sqrt M } \right)\). When the target locates within the first beam sector, \(20\log \left( {\left| {{\mathbf{a}}^{T} (\theta_{l}^{e} ){\mathbf{w}}_{1} } \right|/\sqrt M } \right)\) has a large gain. In this case, the target energy can increase further. \(n_{1} (\tau_{l}^{e} ,f_{d,j} )\) includes the noise term, the DMI and IUI residual, the sidelobes from the other targets and the target interference from the other beams (i.e., the second term in (13)). When the target after processing has sufficient SINR, it will be detected on the 2-d map of the matched filter. However, it is noted that the beamforming gain \(20\log \left( {\left| {{\mathbf{a}}^{T} (\theta_{l}^{e} ){\mathbf{w}}_{1} } \right|/\sqrt M } \right)\) will be smaller for targets locating in the other beam sectors than the first beam sector. What was worse, \(20\log \left( {\left| {{\mathbf{a}}^{T} (\theta_{l}^{e} ){\mathbf{w}}_{1} } \right|/\sqrt M } \right)\) may be even smaller than zero. In this case, the SINR will decrease. It is concluded that the matched filter of the first beam sector can have better target detection performance for the targets within the first beam sector, but degrades for the targets locating in the other beam sectors.

For the targets locating in the other beam sectors than the first beam sector, the data symbols transmitted in their beam sectors can be used to develop the matched filters. This results in the BBB-MF. That is, we use \(s_{b}^{k} (n)\) to develop the matched filter for the *b*th beam sector, for \(b = 1,2,...,N_{w}\), and each matched filter is designed similarly with (11). As the matched filter of the first beam sector, each matched filter can have good detection performance for targets in its corresponding beam sector. Performing the matched filtering beam by beam is similar to beam scanning in traditional radars. The diagram of the BBB-MF is shown in Fig. 2.

It is noted that obtaining the data symbols in the other beam sectors than the first beam sectors is not easy, but not impossible. In cooperative passive radars [41] or joint communication and radar systems, all the data symbols transmitted by the BS can be known. In a passive radar network, multiple radar receivers can be exploited and each beam sector can be deployed with one radar receiver. The radar receivers can communicate the data symbols they obtain. In GoB-based beamforming, the beam sectors are defined globally and not changed. Therefore, the deployment of radar receivers in each beam sector is possible.

### 3.2 DMI and IUI cancellation

In this section, we discuss the cancellation of DMI and IUI. It is known in passive radar that the DMI has a big influence on target detection, because the DMI is far stronger than the target-reflected signals. In MU-MIMO-OFDM passive radar, apart from the DMI, the IUI also imposes a significant impact on target detection, since the IUI, i.e., the clutter-reflected signals corresponding to the transmitted signal to the other beam sectors, can also be significantly larger than the target echoes. In the following, we propose a joint cancellation method for suppressing the DMI and IUI.

Since the target-reflected signals are far weaker than the DMI and IUI, the target-reflected signal terms in (9) can be discarded when dealing with the DMI and IUI cancellation. Furthermore, the beamforming gain \({\mathbf{a}}^{T} (\theta_{i} ){\mathbf{w}}_{b}\) can be included in the amplitude. Then, (9) can be rewritten as:

$$H_{n}^{k} \approx \sum\limits_{{b{ = 1}}}^{{N_{w} }} {\sum\limits_{i = 1}^{{N_{c} }} {B_{b}^{i} s_{b}^{k} (n)e^{{ - j2\pi n\Delta f\tau_{i} }} } } + n_{n}^{k}$$

(16)

where \(B_{b}^{i} = A_{i} {\mathbf{a}}^{T} (\theta_{i} ){\mathbf{w}}_{b} ,b = 1,2,...,N_{w}\). Encapsulate \(H_{n}^{k}\) into a vector as:

$${\mathbf{H}} = \left[ {H_{ - N/2}^{1} ,H_{ - N/2}^{2} ,...,H_{ - N/2}^{K} ,H_{ - N/2 + 1}^{1} ,H_{ - N/2 + 1}^{2} ,...,H_{N/2 - 1}^{K} } \right]^{T}$$

(17)

Quantify the delay \(\tau\) into a number of grids, with \(\Delta \tau\) denoting the delay spacing between two adjacent grids, and formulate the measurement matrix in the following:

$${\mathbf{U}} = \left[ {{\mathbf{U}}_{1} ,{\mathbf{U}}_{2} ,...,{\mathbf{U}}_{{N_{w} }} } \right]^{T}$$

(18)

where

$${\mathbf{U}}_{b} = [{\mathbf{u}}_{b}^{1} ,{\mathbf{u}}_{b}^{2} ,...{\mathbf{u}}_{b}^{P} ]$$

(19)

$${\mathbf{u}}_{b}^{p} = \left[ {\begin{array}{*{20}c} {{\mathbf{s}}_{b} ( - N/2)e^{{ - j2\pi \Delta f( - N/2)\tau_{p} }} } \\ {{\mathbf{s}}_{b} ( - N/2 + 1)e^{{ - j2\pi \Delta f( - N/2 + 1)\tau_{p} }} } \\ \vdots \\ {{\mathbf{s}}_{b} (N/2 - 1)e^{{ - j2\pi \Delta f(N/2 - 1)\tau_{p} }} } \\ \end{array} } \right]$$

(20)

$${\mathbf{s}}_{b} (n) = \left[ {s_{b}^{1} (n),s_{b}^{2} (n),...,s_{b}^{K} (n)} \right]^{T}$$

(21)

\(\tau_{p} { = }p\Delta \tau ,p = 1,2,...,P\), with \(P\Delta \tau\) denoting the maximum delay of the DMI or IUI to be cancelled. Combining (16) to (21), we have:

$${\mathbf{H}} = {\mathbf{UB}}{ + }{\mathbf{n}}$$

(22)

where \({\mathbf{B}}{ = [}B_{1} {,}B_{2} {,}...{,]}^{T}\) represents the amplitude of the potential DMI and IUI. \({\mathbf{n}}\) is the noise. Then, the amplitude of the DMI and IUI can be estimated using the least square technique, i.e.,

$${\hat{\mathbf{B}}}{ = }\arg \mathop {\min }\limits_{{\mathbf{B}}} \left\| {{\mathbf{H}}{ - }{\mathbf{UB}}} \right\|_{2}^{2}$$

(23)

where \(\left\| \cdot \right\|_{2}\) is the *l*_{2} norm of the entity. The amplitude vector \({\mathbf{B}}\) is estimated by solving (23) as:

$${\hat{\mathbf{B}}} \approx \left( {{\mathbf{U}}^{H} {\mathbf{U}}} \right)^{ - 1} {\mathbf{U}}^{H} {\mathbf{H}}$$

(24)

where \(\left( \cdot \right)^{H}\) means taking the conjugate transpose of the entity. It is noted that, when the delay grid spacing \(\Delta \tau\) is small, the matrix \({\mathbf{U}}^{H} {\mathbf{U}}\) may be singular, then the matrix inversion in (24) will lead to numerical problem. The diagonal loading technique can be exploited to overcome this problem. Then, (24) can be further represented as:

$${\hat{\mathbf{B}}} \approx \left( {{\mathbf{U}}^{H} {\mathbf{U}}{ + }\lambda {\mathbf{I}}} \right)^{ - 1} {\mathbf{U}}^{H} {\mathbf{H}}$$

(25)

where \(\lambda\) is a small positive constant. The value of \(\lambda\) can be selected by inspecting the eigenvalues of the matrix \({\mathbf{U}}^{H} {\mathbf{U}}\). Traditionally, it is chosen higher than the nonsignificant eigenvalues, but lower than the smallest significant eigenvalue of the matrix \({\mathbf{U}}^{H} {\mathbf{U}}\). Diagonal loading is equivalent to imposing a constraint on the *l*_{2} norm of the amplitude vector \({\mathbf{B}}\) when using (23) to estimate \({\mathbf{B}}\), i.e.,

$${\hat{\mathbf{B}}}{ = }\arg \mathop {\min }\limits_{{\mathbf{B}}} \left\{ {\left\| {{\mathbf{H}}{ - }{\mathbf{UB}}} \right\|_{2}^{2} + \lambda \left\| {\mathbf{B}} \right\|_{2}^{2} } \right\}$$

(26)

Solving (26) can get the same result with (25). The DMI and IUI can be cancelled as:

$${\mathbf{H}}_{{{\text{remain}}}} = {\mathbf{H}} - {\mathbf{U\hat{B}}}$$

(27)

where \({\mathbf{H}}_{{{\text{remain}}}}\) is the remaining signal after DMI and IUI cancellation.

The proposed DMI and IUI cancellation method is similar to the extensive cancellation method (ECA) in passive radar [40], except that the proposed method operates with the channel estimate \(H_{n}^{k}\), but the ECA performs in the original digital data domain. The MU-MIMO-OFDM signal usually has great bandwidth, thus the sampling frequency is large, leading to huge data size in the original digital data domain. Cancellation in the original digital data domain has huge computational complexity and requires large storage capability. However, the data size of the channel estimate \(H_{n}^{k}\) is significantly smaller; therefore, operating with \(H_{n}^{k}\) increases the computational efficiency greatly. Similar to the parallel implementation version of ECA, i.e., the batch-ECA [42], where the received signal is divided into several segments and the cancellation is performed segment by segment, the proposed method can also be performed segment by segment by properly arranging the channel estimate \(H_{n}^{k}\). This can further increase the computational efficiency.