Transmit and receive hybrid beamforming design for OFDM dual-function radar-communication systems

Abstract

This article investigates the design of hybrid digital and analog transmit/receive beamformers for orthogonal frequency division multiplexing dual-function radar-communication system. Specifically, we model the hybrid beamforming design by simultaneously optimizing the spectrum efficiency of communication and approximate output signal-to-interference noise ratio of radar sensing, through a flexible performance tradeoff. Due to the coupling relationship among the beamformers in the hybrid structure and the independence of signals in different frequency bands, the resultant optimization problem is NP-hard. Hence, a consensus alternating direction method of multipliers approach is designed to tackle the difficulty. The transmit hybrid beamformer is obtained by introducing auxiliary variables and exploiting the block continuous upper bound minimization method. Then, the receive beamformer can be optimized via generalized eigenvalue decomposition. Numerical simulations are provided to demonstrate the effectiveness of the proposed scheme.

1 Introduction

Radar and communications represent fundamentally important use of the electromagnetic (EM) spectrum. Due to the growing demand for higher radar sensing capability and communication data rates, greater bandwidth is needed for radar and communications, as well as many other services. Undoubtedly, this has led to the congestion of EM spectrum, which is known as precious resource. As an example, the spectrum allocated to world interoperability for microwave access (WiMAX) and long-term evolution (LTE) (2500–2690 MHz band) is adjacent to the air traffic control radar (2700–2900 MHz) [1]. Moreover, the millimeter-wave band used by automotive radar and high-resolution imaging radar [2] has also been shared with 5G network [3]. Therefore, DFRC systems (also named as RadCom in the literature), which share hardware platforms and frequency bands, have become the main objectives guiding active sensing and wireless technologies [4].

In the earlier work [5], it is argued that DFRC systems can allow both environmental sensing and vehicle to everything (V2X) communications to be performed more efficiently. A unique waveform design concept has been presented in [6] to allow for simultaneously performing both digital beamforming radar and multiple-input multiple-output (MIMO) communication operations. To increase the degree of freedom, the MIMO-DFRC system that embeds the amplitude-shift keying signals into the radar emission is discussed in [7]. In addition, by modulating the radar emission pulse into the communication signal, the radar-embedded communication system with a low probability of interception has been proposed in [8] and [9]. To realize dual-function beampattern, in [10], the dual-function waveform design is modeled as a trade-off problem. These DFRC systems are investigated by adopting fully digital beamformers, which could result in unfavorable hardware cost and power consumption. Therefore, hybrid beamforming architectures are considered in [11], to develop DFRC systems with OFDM modulation. Apart from traditional waveforms, some new waveforms are investigated for both sensing and communications to improve the overall performance of DFRC systems [12] and [13]. The orthogonal time frequency space (OTFS) waveform multiplexes information symbols in the delay-Doppler domain [12] and [14]. From the perspective of communications, the OTFS has the ability to deal with large Doppler shift and obtain both time and frequency diversities in doubly selective channels. Moreover, it can achieve similar sensing performance with OFDM as shown in [13] and [14].

Typically, the emission energy should be focused towards the target angle for radar sensing. Nevertheless, the method of minimizing the difference between the transmitted waveform and the predetermined waveform is not suitable for the case of uncertain beampatterns. Consequently, it is essential to consider the different performance indicators in the DFRC system. For radar systems, various performance parameters are used in radar beamforming design, such as mutual information maximization [15,16,17] and radar ambiguity function minimization [18]. Since the performance of target parameter estimation relies greatly on the radar SINR [19], the radar transmission waveform is often designed via SINR maximization [20,21,22].

Note that, the above-mentioned studies focus on the radar receive filter of the DFRC system in the narrowband case, and the DFRC system was developed by adopting fully digital beamforming. So far, the design problem for wideband DFRC system has not been well investigated. Further, it is necessary to investigate radar receive filters with hybrid digital and analog beamforming architecture. This motivates us to carry out the study on hybrid beamforming transceivers for OFDM-DFRC systems in this paper. The proposed method models the hybrid beamforming of the DFRC system as a tradeoff problem, which is solved by updating the hybrid beamforming transmitter, hybrid beamforming radar receiver, and user receiver in an alternating manner.

The main contributions of this work can be summarized as follows:

• We propose to jointly design the hybrid beamforming transmitter, hybrid beamforming radar receiver, and user receiver in an OFDM-DFRC system. Specifically, the hybrid beamforming design is formulated as a tradeoff problem composed of spectral efficiency (SE) of communication and the approximate output SINR for radar sensing.

• We derive an alternating optimization method to tackle the nonconvex weighted problem. More specifically, the communication and radar sensing performance metrics are normalized to deal with their different ranges. The hybrid radar receive filter is updated after the hybrid beamforming transmitter and user receiver optimization. To optimize the hybrid beamforming transmitter and user receiver, the block successive upper-bound minimization method (BSUM) [23] is employed. The resulting problem can be solved in a distributed manner based on the consensus alternating direction method of multipliers (CADMM) [24].

• We convert the design problem of analog precoding into a phase-only problem and solve it with the variable step size gradient projection (VSGP) algorithm [23]. In order to validate the proposed design, various numerical results are provided.

The remainder of this paper is organized as follows. Section 2 introduces the system model. In Sect. 3, the performance indicator and optimization problems for the DFRC system are presented. The alternating optimization method based on CADMM is proposed in Sect. 4. Section 5 presents various numerical simulations and Sect. 6 concludes the paper.

Notations: Unless otherwise specified, vectors are denoted by bold lowercase letters ${\textbf{a}}$ and matrices are represented by bold uppercase letters $\textbf{A}$, $(\cdot )^T$, $(\cdot )^H$ and $(\cdot )^*$ are stand for the transpose, Hermitian transpose and complex conjugate of the matrices, respectively. The sets of n-dimensional complex-valued vectors and $N\times N$ complex-valued matrices are denoted by $\mathbb {C}^{n}$ and $\mathbb {C}^{N\times N}$. $\nabla$ and $\angle$ denotes the gradient and angle operator, respectively. $\textbf{I}_N$ denotes the $N\times N$ identity matrix and $\mathbb {E}\{\cdot \}$ denotes the statistical expectation. $\Vert \cdot \Vert _2$ denotes the $l_2$ norm and $\vert \cdot \vert$ represent determinant or absolute value depending on context.

2 Background

Consider a wideband OFDM-DFRC system that sends communication symbols to U users equipped with $N_{cr}$ antennas, while performing radar detection. A hybrid digital and analog beamforming architecture is adopted at the transmitter, where the number of radio frequency (RF) chains $N_{{\text{RF}}_t}$ is much less than the number of transmitter antennas $N_t$. Further, let $N_{rr}$ and $N_{\text{RF}_r}$ be the numbers of receiver antennas and RF chains, respectively. The data symbols are modulated and spread across K subcarriers, the symbol width is $\Delta t$ and the subcarrier frequency step is $\Delta f$. Moreover, we assume $\Delta f =1/\Delta t$ to ensure the orthogonality of subcarriers, and each antenna array is a uniform linear array (ULA).

2.1 Transmit model

Let $\textbf{s}_k\in \mathbb {C}^U$ be the vector composed of data symbols at the k-th subcarrier, i.e., each user transmits a communication symbol on each subcarrier, and further assume that $\mathbb {E}\{\textbf{s}_k\textbf{s}_k^H\}=\textbf{I}_U$, $k=1,...,K$. In the hybrid beamforming system, the data symbols are first processed by the low-dimensional digital beamformer $\textbf{F}_{\text{BB}_k}=[\textbf{f}_{\text{BB}_{k,1}},...,\textbf{f}_{\text{BB}_{k,U}}]\in \mathbb {C}^{N_{\text{RF}_t}\times U}$. Then, after $N_{\text{RF}_t}$ K-point inverse fast Fourier transform (IFFT) and adding a cyclic prefix of length D, the data symbols are converted from the frequency domain to the time domain. Finally, an analog beamformer $\textbf{F}_{\text{RF}}\in \mathbb {C}^{N_t\times N_{\text{RF}_t}}$ is used to generate the final transmission signal. Therefore, the final transmitted signal at subcarrier k can be written as

$$\begin{aligned} \textbf{x}_{k} = \textbf{F}_{{\text{RF}}}\textbf{F}_{\text{BB}_{k }}\textbf{s}_{k}. \end{aligned}$$

(1)

To investigate the performance upper bound of the method, we consider a hybrid beamformer with a fully connected structure. Specifically, each RF chain is connected to all the antenna elements via a set of phase shifters, i.e., $|\textbf{F}_{\text{RF}}(i,j) |= 1$, $\forall i, j$. Furthermore, we assume the power of transmitter for each user on each subcarrier satisfies the following condition:

$$\begin{aligned} \Vert \textbf{F}_{{\text{RF}}}\textbf{f}_{{\text{BB}_{k,u}}}\Vert _2^2=\frac{\mathcal {P}_k}{U}. \end{aligned}$$

(2)

2.2 Communication model

2.2.1 Receive signal model

Assume that the user uses a fully digital beamformer, and for the u-th user on the k-th subcarrier, the signal arriving through the channel $\textbf{H}_{k,u}\in \mathbb {C}^{N_{cr}\times N_t}$ is mixed with additive white Gaussian noise (AWGN). After removing the cyclic prefix and $N_{cr}$ K-point FFT, the frequency domain signal is processed by the digital combiner $\textbf{u}_{k,u}$ to obtain the communication symbol of the user u on the subcarrier k as

$$\begin{aligned} \hat{y}_{k, u}&= \textbf{u}_{k, u}^{H}\textbf{H}_{k, u} \textbf{F}_{\text{RF}} \textbf{F}_{\text{BB}_k} \textbf{s}_{k}+\textbf{u}_{k, u}^{H}\hat{\textbf{n}}_{k, u} \nonumber \\&=\underbrace{\textbf{u}_{k, u}^{H} \textbf{H}_{k, u} \textbf{F}_{\text{RF}} \textbf{f}_{\text{BB}_{k, u}} s_{k, u}}_{\text{ desired } \text{ signal } } +\underbrace{\sum _{i \ne u}^{U} \textbf{u}_{k, u}^{H} \textbf{H}_{k, u} \textbf{F}_{\text{RF}} \textbf{f}_{\text{BB}_{k, i}} s_{k, i}}_{\text {multi-user interference }(\text{MUI})}+\textbf{u}_{k, u}^{H} \hat{\textbf{n}}_{k, u}, \end{aligned}$$

(3)

where $\hat{\textbf{n}}_{k, u}$ denotes the AWGN of the u-th user at the k-th subcarrier with zero mean and covariance matrix $\sigma _{\hat{n}}^2\textbf{I}_{N_{cr}}$.

2.2.2 Channel model

Following the previous works [25,26,27,28], in this paper we consider a geometric wideband channel model with $N_{cl}$ clusters and each cluster contributes with $N_{ray}$ rays between the transmitter and each user. The delay d wideband mmWave channel matrix for u-th user on k-th subcarrier can be modeled as

$$\begin{aligned} \textbf{H}_{k,u}[d] = \sum _{i=1}^{N_{cl}}\sum _{l=1}^{N_{ray}} \alpha _{i,l}\textbf{a}_{r}(\theta _{i,l},k)\textbf{a}_t^H(\phi _{i,l},k)p(dT_s-\tau _i), \end{aligned}$$

(4)

where $\alpha _{i,l}\sim \mathcal{C}\mathcal{N}(0,\frac{N_tN_{cr}}{N_{cl}N_{ray}})$ denotes the complex path gain of l-th ray in the i-th cluster, and $p(dT_s-\tau _i)$ is the pulse-shaping function for $T_s$-spaced signaling at time delay $\tau _{i}$. Further, $\textbf{a}_{r}(\theta _{i,l},k)$ and $\textbf{a}_t(\phi _{i,l},k)$ are the receive and transmit array response vectors at k-th subcarrier, where $\theta _{i,l}$ and $\phi _{i,l}$ represent the angles of arrival and departure (AoAs/AoDs), respectively. For a ULA with N antennas, the steering vector at k-th subcarrier is expressed as:

$$\begin{aligned} \textbf{a}(\theta ,k) = \frac{1}{\sqrt{N}}\left[ 1,e^{j\pi \frac{f_k}{f_{\text{cent}}}\text{sin}\theta },...,e^{j\pi \frac{f_k}{f_{\text{cent}}}(N-1)\text{sin}\theta }\right] ^T, \end{aligned}$$

(5)

where $f_{\text{cent}}$ is the center frequency, and $f_k = f_{\text{cent}}+(k-\frac{K+1}{2})\Delta f$. Given the delay d channel matrix (4), the channel of u-th user at k-th subcarrier can be expressed as

$$\begin{aligned} \textbf{H}_{k,u} = \sum _{d=1}^D \textbf{H}_{k,u}[d]e^{-j\frac{2\pi k}{K}d}. \end{aligned}$$

(6)

2.3 Radar model

Denoting the steering matrix from the transmitter to the target (or interference) to the radar receiver as $\textbf{A}(\varphi ,k)=\textbf{a}_r(\varphi ,k)\textbf{a}^T_t(\varphi ,k)$, $k=1,...,K$. Therefore, the radar received signal at the k-th subcarrier can be modeled as

$$\begin{aligned} \tilde{\textbf{y}}_{r,k}&=\underbrace{\xi _g\textbf{A}(\varphi _g,k)\textbf{F}_{\text {RF}}\textbf{F}_{\text{BB}_{k}}\textbf{s}_{k}}_{\text{target}}+\underbrace{\sum _{j=1}^J\xi _j\textbf{A}(\varphi _j,k)\textbf{F}_{\text {RF}}\textbf{F}_{\text {BB}_{k}}\textbf{s}_{k}}_{\text {interference}}+\bar{\textbf{n}}_{k}, \end{aligned}$$

(7)

where $\xi _g$ and $\xi _j$, respectively, denote the reflection coefficients of the target and the j-th interference source, and they are assumed to be independent zero mean random variables with variances $\mathbb {E}\{\vert \xi _g \vert ^2 \}=\sigma _g^2$ and $\mathbb {E}\{\vert \xi _j \vert ^2 \}=\sigma _j^2$, respectively. $\varphi _g$ and $\varphi _j$, respectively represent the angles of the target and the j-th interference source, and $\bar{\textbf{n}}_{k}$ denotes the AWGN of k-th subcarrier with zero mean and covariance matrix $\sigma _{\bar{n}}^2\textbf{I}_{N_{rr}}$.

Do the reverse operations of transmitted signal to received signal, the received signal of the k-th subcarrier is first processed by an analog beam combiner $\textbf{W}_{\text{RF}}\in \mathbb {C}^{N_{rr}\times N_{\text{RF}_r}}$ implemented by analog phase shifters, i.e., $|\textbf{W}_{\text{RF}}(i,j) |= 1$, $\forall i, j$. After removing the cyclic prefix and $N_{rr}$ K-point fast Fourier transformation (FFT), the signal is converted to the frequency domain. At each subcarrier k, a digital combiner $\textbf{w}_{\text{BB}_k}\in \mathbb {C}^{N_{\text{RF}_r}}$ is applied to estimate the radar target as

$$\begin{aligned} \bar{y}_{r,k}&=\underbrace{\sum _{u=1}^U\textbf{w}_{\text{BB}_k}^H\textbf{W}^H_{\text{RF}}\xi _g\textbf{A}(\varphi _g,k)\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}s_{k,u}}_{\text{target}} \nonumber \\&\quad +\underbrace{\sum _{u=1}^U\textbf{w}_{\text{BB}_k}^H\textbf{W}^H_{\text{RF}}\sum _{j=1}^J\xi _j\textbf{A}(\varphi _j,k)\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}s_{k,u}}_{\text {interference}} +\frac{1}{U}\sum _{u=1}^U\textbf{w}_{\text{BB}_k}^H\textbf{W}^H_{\text{RF}}\bar{\textbf{n}}_{k}. \end{aligned}$$

(8)

2.4 Performance indicators

In the trade-off problem, it is difficult to achieve the purpose of trade-off by setting the trade-off coefficient in the problem for two indicators with different ranges. Hence, it is better to select performance indicators with similar variation ranges for the problem to achieve fair performance trade-off. In this paper, we design a hybrid beamforming transceiver and user receiver to achieve the trade-off between radar and communication performance. For communication, we consider SE as a performance parameter of the communication function, which is defined as

$$\begin{aligned} \text{SE} = \frac{1}{K}\sum \limits _{k=1}^K\sum \limits _{u=1}^U\text{SE}_{k,u}, \end{aligned}$$

(9)

where $\text{SE}_{k,u}$ is the spectrum efficiency of the u-th user on the k-th subcarrier as

$$\begin{aligned} \text{SE}_{k,u} = \text{log}_2\left( 1+\frac{|\textbf{u}^H_{k,u}\textbf{H}_{k,u}\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}|^2 }{\sum \limits _{i\ne u}^U|\textbf{u}^H_{k,u}\textbf{H}_{k,u}\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,i}}|^2+\sigma _{\hat{n}}^2\textbf{u}^H_{k,u}\textbf{u}_{k,u}}\right) . \end{aligned}$$

(10)

For radar sensing, the SINR can be used to measure the effect of transmitter and radar filter on estimating the target and suppressing interference. For the ease of design, we use the following formula to define the approximate SINR:

$$\begin{aligned} \text{SINR} = \frac{1}{KU}\sum \limits _{k=1}^K\sum _{u=1}^U\text{SINR}_{k,u}, \end{aligned}$$

(11)

where the $\text{SINR}_{k,u}$ is expressed as

$$\begin{aligned} \text{SINR}_{k,u} = \frac{\mathbb {E}\{|\textbf{v}^H_{_k}\xi _g\textbf{A}(\varphi _g,k)\textbf{p}_{k,u} |^2\} }{\mathbb {E}\left\{ |\textbf{v}^H_k\sum \limits _{j=1}^J\xi _j\textbf{A}(\varphi _j,k)\textbf{p}_{k,u} |^2\right\} +\frac{\sigma ^2_{\bar{n}}}{U}\Vert \textbf{v}_k\Vert _2^2}. \end{aligned}$$

(12)

where $\textbf{p}_{k,u}=\textbf{F}_{\text{RF}} \textbf{f}_{\text{BB}_{k, u}} s_{k,u}$ and $\textbf{v}_{_k}=\textbf{W}_{\text{RF}}\textbf{w}_{\text{BB}_k}$. Let ${\varvec{\Theta }}_{g,k}$ and ${\varvec{\Theta }}_{in,k}$ be defined, respectively, as

$$\begin{aligned} {\varvec{\Theta }}_{g,k}= & {} \sigma ^2_g\textbf{A}^H(\varphi _g,k)\textbf{v}_k\textbf{v}_k^H\textbf{A}(\varphi _g,k), \end{aligned}$$

(13)

$$\begin{aligned} {\varvec{\Theta }}_{in,k}= & {} \sum \limits _{j=1}^J\sigma ^2_j\textbf{A}^H(\varphi _j,k)\textbf{v}_k\textbf{v}_k^H\textbf{A}(\varphi _j,k)+\frac{\sigma _{\bar{n}}^2}{\mathcal {P}_k}\Vert \textbf{v}_k \Vert _2^2\textbf{I}_{N_t}. \end{aligned}$$

(14)

Then, the SINR in (12) can be re-expressed as

$$\begin{aligned} \text{SINR}_{k,u}=\frac{\textbf{f}^H_{\text{BB}_{k,u}}\textbf{F}^H_{\text{RF}}{\varvec{\Theta }}_{g,k}\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}}{\textbf{f}^H_{\text{BB}_{k,u}}\textbf{F}^H_{\text{RF}}{\varvec{\Theta }}_{in,k}\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}}. \end{aligned}$$

(15)

It is worth noting that the optimization of SINR needs the knowledge of angles as well as the variances of reflection coefficients of the target and interferences. These parameters can be obtained from a cognitive paradigm [29,30,31] and in this work they are assumed to be known for simplicity.

2.5 Normalization

In this paper, the objective function is established as the weighted summation of the communication SE and radar SINR, which is defined as

$$\begin{aligned} \eta \text{SINR}+(1-\eta )\text{SE}, \end{aligned}$$

(16)

where $0\le \eta \le 1$ is the weight parameter, which is used to indicate the relative importance of the two objective functions.

The weighting method indicates the relative importance of the two functions in the integrated system by setting the weight factor. However, the effectiveness of the above method is based on the condition that the two objective functions have the same range of variation. In other words, for two objective functions with inconsistent variation ranges, it is difficult to find suitable weights to compensate for the difference in amplitude and indicate relative importance. According to [32, 33], the way to deal with this defect is normalizing the objective function, which can be mathematically expressed as

$$\begin{aligned} \frac{\eta }{KU}\sum \limits _{k=1}^K\sum \limits _{u=1}^U\frac{\text{SINR}_{k,u}-\mathrm {SINR_{min}}}{\mathrm {SINR_{max}}-\mathrm {SINR_{min}}} +\frac{(1-\eta )}{K}\sum \limits _{k=1}^K\sum \limits _{u=1}^U\frac{\text{SE}_{k,u}-\mathrm {SE_{min}}}{\mathrm {SE_{max}}-\mathrm {SE_{min}}}, \end{aligned}$$

(17)

where the maximum and minimum values of SINR and SE of each user on each subcarrier, i.e., $\mathrm {SINR_{max}}$, $\mathrm {SE_{min}}$, $\mathrm {SINR_{min}}$ and $\mathrm {SE_{max}}$, are calculated by setting the tradeoff parameter to 0 and 1, respectively.

With the normalization, the problem of hybrid beamforming design for OFDM-DFRC is modeled as

$$\begin{aligned} \min \limits _{\begin{array}{c} \textbf{F}_{\text{RF}},\textbf{f}_{\text{BB}_{k,u}}\\ \textbf{u}_{k,u},\textbf{W}_{\text{RF}},\textbf{w}_{\text{BB}_k} \end{array}}&\sum \limits _{k=1}^K\sum \limits _{u=1}^U\beta _r\text{SINR}_{k,u}+\beta _c\text{SE}_{k,u}\\ \mathrm {s.t.}\,\,&\Vert \textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}\Vert ^2_2=\frac{\mathcal {P}_k}{U}, \forall k,u\\&\vert \textbf{F}_{\text {RF}}(i,j) \vert =1,\forall i,j,\\&\vert \textbf{W}_{\text {RF}}(i,j) \vert =1,\forall i,j, \end{aligned}$$

(18)

where $\beta _r=-\eta /KU(\mathrm {SINR_{max}}-\mathrm {SINR_{min}})$, $\beta _c=-(1-\eta )/K(\mathrm {SE_{max}}-\mathrm {SE_{min}})$. Obviously, since the non-convex objective function and the coupling of the digital and analog precoder in the constraint terms, the problem (18) is difficult to solve. To effectively tackle the problem, a CADMM algorithm is developed.

3 Methods

In this paper, the hybrid beamforming transmitter, hybrid beamforming radar receiver, and user receiver are jointly optimized to minimize the objective function in problem (18). Due to the objective function containing multiple variables, this joint optimization problem is complicated. This motivates us to apply alternate optimization strategies [24, 34, 35], to decouple and simplify this multi-variable joint optimization problem. Specifically, we first alternately optimize the hybrid beamforming transmitters $\textbf{F}_{\text{RF}}$ and $\textbf{F}_{\text{BB}_{k}}$ and user receivers $\textbf{u}_{k,u}$ by assuming fixed hybrid beamforming radar receiver. Next, we design the hybrid beamforming radar receivers $\textbf{W}_{\text{RF}}$ and $\textbf{w}_{\text{BB}_{k}}$ for given transmitter and user receiver.

3.1 Hybrid transmitter and user receiver design

In this section, we update $\textbf{F}_{\text{RF}}$, $\textbf{F}_{\text{BB}_{k}}$ and $\textbf{u}_{k,u}$, based on the given hybrid beamforming radar receiver $\textbf{W}_{\text{RF}}$ and $\textbf{w}_{\text{BB}_{k}}$, which removes the constraint $\vert \textbf{W}_{\text {RF}}(i,j) \vert =1$. Furthermore, we introduce auxiliary variables $\textbf{f}_{k,u}=\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}},\forall k,u$ to decouple the analog precoder and digital precoder, and then add it as coupling constraints (19b) to the problem (18):

$$\begin{aligned}&\min \limits _{\begin{array}{c} \textbf{F}_{\text{RF}},\textbf{f}_{\text{BB}_{k,u}} \\ \textbf{u}_{k,u},\textbf{f}_{k,u} \end{array}}&\sum \limits _{k=1}^K\sum \limits _{u=1}^U\beta _r\text{SINR}_{k,u}+\beta _c\text{SE}_{k,u}, \end{aligned}$$

(19a)

$$\begin{aligned}&\mathrm {s.t.}&\textbf{f}_{k,u}= \textbf{F}_{\text{RF}}\textbf{f}_{\text {BB}_{k,u}}, \forall k,u, \end{aligned}$$

(19b)

$$\begin{aligned}{} & {} \Vert \textbf{f}_{k,u}\Vert ^2_2=\frac{\mathcal {P}_k}{U}, \forall k,u, \end{aligned}$$

(19c)

$$\begin{aligned}{} & {} \vert \textbf{F}_{\text {RF}}(i,j) \vert =1,\forall i,j, \end{aligned}$$

(19d)

where $\text{SINR}_{k,u}$ and $\text{SE}_{k,u}$ can be expressed as

$$\begin{aligned}&\text{SINR}_{k,u}=\frac{\textbf{f}^H_{k,u}{\varvec{\Theta }}_{g,k}\textbf{f}_{k,u}}{\textbf{f}^H_{k,u}{\varvec{\Theta }}_{in,k}\textbf{f}_{k,u}}, \end{aligned}$$

(20)

$$\begin{aligned}&\text{SE}_{k,u} = \text{log}_2\left( 1+\frac{\textbf{f}^H_{k,u}\overline{\textbf{H}}_{k,u}\textbf{f}_{k,u}}{\sum \limits _{i\ne u}^U\textbf{f}^H_{k,i}\overline{\textbf{H}}_{k,u}\textbf{f}_{k,i}+\sigma ^2_{\hat{n}}\textbf{u}^H_{k,u}\textbf{u}_{k,u}}\right) , \end{aligned}$$

(21)

with $\overline{\textbf{H}}_{k,u}$ being defined as

$$\begin{aligned} \overline{\textbf{H}}_{k,u}=\textbf{H}_{k,u}^H\textbf{u}_{k,u}\textbf{u}_{k,u}^H\textbf{H}_{k,u}. \end{aligned}$$

(22)

To deal with the resulting problem, the CADMM algorithm [24] is employed. Specifically, the augmented Lagrangian function of (19) is given in (23) shown as follows:

$$\begin{aligned} \mathcal {L}(\textbf{f}_{k,u},\textbf{u}_{k,u},\textbf{f}_{\text{BB}_{k,u}},\textbf{F}_{\text{RF}}, \textbf{d}_{k,u})=\sum \limits _{k=1}^K\sum \limits _{u=1}^U\mathcal {L}_{k,u}(\textbf{f}_{k,u}, \textbf{u}_{k,u}, \textbf{f}_{\text{BB}_{k,u}},\textbf{F}_{\text{RF}},\textbf{d}_{k,u}), \end{aligned}$$

(23)

and

$$\begin{aligned} \mathcal {L}_{k,u}(\textbf{f}_{k,u}, \textbf{u}_{k,u}, \textbf{f}_{\text{BB}_{k,u}},\textbf{F}_{\text{RF}},\textbf{d}_{k,u}) =F(\textbf{f}_{k,u})+\frac{\rho }{2}\Vert \textbf{f}_{k,u}-\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}+\textbf{d}_{k,u} \Vert _2^2, \end{aligned}$$

(24)

where $F(\textbf{f}_{k,u})=\beta _r\text{SINR}_{k,u}+\beta _c\text{SE}_{k,u}$, $\textbf{d}_{k,u}\in \mathbb {C}^{N_t}$ and $\rho >0$ are the dual variables and corresponding penalty parameters, respectively. Under the CADMM framework, we first update the original variable to minimize the Lagrangian function, and then update the dual variable and penalty parameters [36]. More precisely, at the $(m+1)$-th iteration, the update produces of the corresponding CADMM are given below:

$$\begin{aligned} \textbf{f}^{(m+1)}_{k,u}= & {} \mathop {\text{argmin}}_{\textbf{f}_{k,u}\in \mathcal {D}_{\textbf{f}_{k,u}}}\mathcal {L}_{k,u}(\textbf{f}_{k,u}, \textbf{u}^{(m)}_{k,u},\textbf{f}^{(m)}_{\text{BB}_{k,u}},\textbf{F}^{(m)}_{\text{RF}}, \textbf{d}^{(m)}_{k,u}), \end{aligned}$$

(25a)

$$\begin{aligned} \textbf{u}^{(m+1)}_{k,u}= & {} \mathop {\text{argmin}}_{\textbf{u}_{k,u}}\mathcal {L}_{k,u}(\textbf{f}^{(m+1)}_{k,u},\textbf{u}_{k,u},\textbf{f}^{(m+1)}_{\text{BB}_{k,u}},\textbf{F}^{(m)}_{\text{RF}},\textbf{d}^{(m)}_{k,u}), \end{aligned}$$

(25b)

$$\begin{aligned} \textbf{f}^{(m+1)}_{\text{BB}_{k,u}}= & {} \mathop {\text{argmin}}_{\textbf{f}_{\text{BB}_{k,u}}}\mathcal {L}_{k,u}(\textbf{f}^{(m+1)}_{k,u},\textbf{u}^{(m)}_{k,u}, \textbf{f}_{\text{BB}_{k,u}},\textbf{F}^{(m)}_{\text{RF}},\textbf{d}^{(m)}_{k,u}), \end{aligned}$$

(25c)

$$\begin{aligned} \textbf{F}^{(m+1)}_{\text{RF}}= & {} \mathop {\text{argmin}}_{\textbf{F}_{\text{RF}}\in \mathcal {D}_{\textbf{F}_{\text{RF}}}}\mathcal {L}(\textbf{f}^{(m+1)}_{k,u},\textbf{u}^{(m+1)}_{k,u}, \textbf{f}^{(m+1)}_{\text{BB}_{k,u}},\textbf{F}_{\text{RF}},\textbf{d}^{(m)}_{k,u}), \end{aligned}$$

(25d)

$$\begin{aligned} \textbf{d}^{(m+1)}_{k,u}= & {} \textbf{d}^{(m)}_{k,u}+\textbf{f}^{(m+1)}_{k,u}-\textbf{F}^{(m+1)}_{\text{RF}}\textbf{f}^{(m+1)}_{\text{BB}_{k,u}}, \end{aligned}$$

(25e)

where the sets $\mathcal {D}_{\textbf{f}_{k,u}}$ and $\mathcal {D}_{\textbf{F}_{\text{RF}}}$ are, respectively, defined as

$$\begin{aligned} \mathcal {D}_{\textbf{f}_{k,u}}= & {} \left\{ \textbf{f}_{k,u}\mid \Vert \textbf{f}_{k,u}\Vert ^2_2=\frac{\mathcal {P}_k}{U}\right\} ,\\ \mathcal {D}_{\textbf{F}_{\text{RF}}}= & {} \{ \textbf{F}_{\text{RF}} \mid \vert \textbf{F}_{\text {RF}}(i,j) \vert =1 \}. \end{aligned}$$

The above equations are repeated until some termination conditions are satisfied. For example, when the maximum number of iterations $N_{max}$ is reached. Next, the solutions to the sub-problems are presented.

3.1.1 Update of $\{\textbf{f}_{k,u}\}$

Given $\textbf{u}^{(m)}_{k,u}$, $\textbf{f}^{(m)}_{\text{BB}_{k,u}}$, $\textbf{F}^{(m)}_{\text{RF}}$ and $\textbf{d}^{(m)}_{k,u}$, the sub-problem with respect to $\textbf{f}_{k,u}$ is given by

$$\begin{aligned} \min \limits _{\textbf{f}_{k,u}}\,\,&F(\textbf{f}_{k,u})+\frac{\rho }{2}\Vert \textbf{f}_{k,u}-\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}+\textbf{d}_{k,u} \Vert _2^2 \\ \mathrm {s.t.}\,\,&\Vert \textbf{f}_{k,u}\Vert ^2_2=\frac{\mathcal {P}_k}{U} \end{aligned}$$

(26)

which is difficult to solve due to the non-convex function $F(\textbf{f}_{k,u})$. In this work, the problem is tackled by minimizing a upper bound function of $F(\textbf{f}_{k,u})$, inspired by the BSUM method [23]. More precisely, the quadratic upper bound function of $F(\textbf{f}_{k,u})$ at the current point $\textbf{f}_{k,u}^{(m)}$ can be written as

$$\begin{aligned} f(\textbf{f}_{k,u})&=F(\textbf{f}^{(m)}_{k,u})+2Re\{\nabla ^HF(\textbf{f}_{k,u}^{(m)})(\textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u})\}\\&\quad +\frac{1}{2}\begin{pmatrix}\textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u}\\ \textbf{f}^*_{k,u}-\textbf{f}^{*(m)}_{k,u}\end{pmatrix}^H{\varvec{\Psi }}_{k,u}^{(m)}\begin{pmatrix}\textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u}\\ \textbf{f}^*_{k,u}-\textbf{f}^{*(m)}_{k,u}\end{pmatrix}, \end{aligned}$$

(27)

where $\nabla F(\textbf{f}_{k,u})$ is the partial derivative of $\textbf{f}^*_{k,u}$ and ${\varvec{\Psi }}_{k,u}$ is a positive semi-definite matrix which should be selected such that ${\varvec{\Psi }}_{k,u}-\mathcal {H}_{k,u}$ is also positive semi-definite. Here, $\mathcal {H}_{k,u}$ denotes the Hessian matrix of $F(\textbf{f}_{k,u})$ at the current point $\textbf{f}_{k,u}^{(m)}$. $\nabla F(\textbf{f}_{k,u})$ and $\mathcal {H}_{k,u}$ are deviated in Appendix A. In order to make sure that ${\varvec{\Psi }}_{k,u}-\mathcal {H}_{k,u}$ is positive semi-definite, we can choose ${\varvec{\Psi }}_{k,u}=\lambda _{\text{max}}\textbf{I}$, and $\lambda _{\text{max}}$ denotes the largest eigenvalue of $\mathcal {H}_{k,u}$. Furthermore, (27) can be converted into the following form which only contains $\textbf{f}_{k,u}$:

$$\begin{aligned} f(\textbf{f}_{k,u})&=F(\textbf{f}^{(m)}_{k,u})+2Re\{\nabla ^HF(\textbf{f}_{k,u}^{(m)})(\textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u})\}\\&\quad +\frac{\lambda ^{(m)}_{\text{max}}}{2}\Vert \textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u} \Vert _2^2+ \frac{\lambda ^{(m)}_{\text{max}}}{2}\Vert \textbf{f}^*_{k,u}-\textbf{f}^{*(m)}_{k,u} \Vert _2^2\\&=F(\textbf{f}^{(m)}_{k,u})+2Re\{\nabla ^HF(\textbf{f}_{k,u}^{(m)})(\textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u})\} +\lambda ^{(m)}_{\text{max}}\Vert \textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u} \Vert _2^2. \end{aligned}$$

(28)

Replacing $F(\textbf{f}_{k,u})$ in (26) by (28) yields the optimization problem as

$$\begin{aligned}{} & {} \min \limits _{\textbf{f}_{k,u}}\,\,\, 2Re\{\nabla ^HF(\textbf{f}_{k,u}^{(m)})(\textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u})\} +\lambda ^{(m)}_{\text{max}}\Vert \textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u} \Vert _2^2\nonumber \\{} & {} \qquad +\frac{\rho }{2}\Vert \textbf{f}_{k,u}-\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}+\textbf{d}_{k,u} \Vert _2^2, \end{aligned}$$

(29a)

$$\begin{aligned}{} & {} \mathrm {s.t.}\,\,\,\Vert \textbf{f}_{k,u}\Vert ^2_2=\frac{\mathcal {P}_k}{U}. \end{aligned}$$

(29b)

which can be further transformed into the following optimization problem (details can be found in Appendix B):

$$\begin{aligned} \min \limits _{\textbf{f}_{k,u}}&\frac{\rho }{2}\left\| \textbf{f}_{k,u}-\left( \textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}-\textbf{d}_{k,u}-\frac{2\nabla F(\textbf{f}_{k,u})}{\rho }\right) \right\| _2^2 +\lambda ^{(m)}_{\text{max}}\Vert \textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u}, \Vert _2^2\\ \mathrm {s.t.}&\Vert \textbf{f}_{k,u}\Vert ^2_2=\frac{\mathcal {P}_k}{U}. \end{aligned}$$

(30)

It is seen that the above problem have a closed-form optimal solution, which can be obtained by utilizing Karush–Kuhn–Tucker (KKT) conditions, as

$$\begin{aligned} \textbf{f}_{k,u}^{(m+1)}=\sqrt{\frac{\mathcal {P}_k}{U}}\frac{\tilde{\textbf{f}}_{k,u}^{(m+1)}}{\Vert \tilde{\textbf{f}}_{k,u}^{(m+1)} \Vert _2}, \end{aligned}$$

(31)

where $\tilde{\textbf{f}}_{k,u}^{(m+1)}$ is defined as

$$\begin{aligned} \tilde{\textbf{f}}_{k,u}^{(m+1)} = \frac{\frac{\rho }{2}\left( \textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}-\textbf{d}_{k,u}-\frac{2\nabla F(\textbf{f}_{k,u})}{\rho }\right) +\lambda ^{(m)}_{\text{max}}\textbf{f}_{k,u}^{(m)}}{\frac{\rho }{2}+\lambda ^{(m)}_{\text{max}}}. \end{aligned}$$

(32)

3.1.2 Update of $\{\textbf{u}_{k,u}\}$

The subproblem (25b) is to updating $\textbf{u}_{k,u}$ to maximize the SE. In fact, the problem of maximizing SE is the same as the problem of optimizing the SINR of the user’s received signal. Based on this observation, we can obtain $\textbf{u}_{k,u}$ by solving the following problem

$$\begin{aligned}&\min \limits _{\textbf{u}_{k,u}}&\frac{\textbf{u}^H_{k,u}\varvec{\Xi }_{k,u}\textbf{u}_{k,u}}{\textbf{u}^H_{k,u}\overline{\varvec{\Xi }}_{k,u}\textbf{u}_{k,u}}, \end{aligned}$$

(33)

where $\varvec{\Xi }_{k,u}$ and $\overline{\varvec{\Xi }}_{k,u}$ are defined as

$$\begin{aligned}\varvec{\Xi }_{k,u}& = \textbf{H}_{k,u}\textbf{F}_{\text{RF}}\textbf{f}_{\text {BB}_{k,u}}\textbf{f}^H_{\text {BB}_{k,u}}\textbf{F}^H_{\text{RF}}\textbf{H}_{k,u}^H, \\ \overline{\varvec{\Xi }}_{k,u} &= \sum \limits _{i\ne u}^U\textbf{H}_{k,u}\textbf{F}_{\text{RF}}\textbf{f}_{\text {BB}_{k,i}}\textbf{f}^H_{\text {BB}_{k,i}}\textbf{F}^H_{\text{RF}}\textbf{H}_{k,u}^H+\sigma ^2_{\hat{n}}\textbf{I}_{N_{cr}}. \end{aligned}$$

(34)

Obviously, problem (33) is a unconstrained maximization problem, whose optimal solution is the maximum generalized eigenvalue of $(\varvec{\Xi }_{k,u},\overline{\varvec{\Xi }}_{k,u})$ or the principal eigenvector of $\overline{\varvec{\Xi }}^{-1}_{k,u}\varvec{\Xi }_{k,u}$.

3.1.3 Update of $\{\textbf{f}_{\text{BB}_{k,u}}\}$

When the analog precoder $\textbf{F}_{\text{RF}}$, auxiliary variables $\textbf{f}_{k,u}$ and user receiver $\textbf{u}_{k,u}$ are all fixed, the minimization of (25c) with respect to $\textbf{f}_{\text{BB}_{k,u}}$ becomes

$$\begin{aligned}&\min \limits _{\textbf{f}_{\text{BB}_{k,u}}}&\Vert \textbf{f}_{k,u}-\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}+\textbf{d}_{k,u} \Vert _2^2, \end{aligned}$$

(35)

whose closed-form solution is

$$\begin{aligned} \textbf{f}^{(m+1)}_{\text{BB}_{k,u}}=(\textbf{F}^H_{\text{RF}}\textbf{F}_{\text{RF}})^{-1}\textbf{F}^H_{\text{RF}}(\textbf{d}_{k,u}+\textbf{f}_{k,u}). \end{aligned}$$

(36)

3.1.4 Update of $\textbf{F}_{\text{RF}}$

The analog precoder $\textbf{F}_{\text{RF}}$ appears only in the second term of the objective (24) with other fixed blocks. Therefore, the sub-problem with respect to $\textbf{F}_{\text{RF}}$ is given by

$$\begin{aligned} \min \limits _{\textbf{F}_{\text{RF}}}\,\,&\sum \limits _{k=1}^K\sum \limits _{u=1}^U\Vert \textbf{f}_{k,u}-\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}+\textbf{d}_{k,u}\Vert _2^2,\\ \mathrm {s.t.}\,\,&\vert \textbf{F}_{\text{RF}}(i,j)\vert =1,\forall i,j. \end{aligned}$$

(37)

Since the contribution of each row element of $\textbf{F}_{\text{RF}}$ to the objective function is independent, we can transform the problem into the following form:

$$\begin{aligned} \min \limits _{\textbf{F}_{\text{RF}}}\,\,&\sum \limits _{i=1}^{N_t}\sum \limits _{k=1}^K\sum \limits _{u=1}^U\Vert \textbf{b}_{k,u}(i)-\textbf{F}_{\text{RF}}(i,:)\textbf{f}_{\text{BB}_{k,u}}\Vert _2^2,\\ \mathrm {s.t.}\,\,&\vert \textbf{F}_{\text{RF}}(i,j)\vert =1,\forall i,j, \end{aligned}$$

(38)

where $\textbf{b}_{k,u}=\textbf{f}_{k,u}+\textbf{d}_{k,u}$. The above problem can be divided into $N_t$ parallel sub-problems, the i-th sub-problem is expressed as

$$\begin{aligned} \min \limits _{\textbf{F}^T_{\text{RF}}(i,:)}\,\,&\sum \limits _{k=1}^K\sum \limits _{u=1}^U\Vert \textbf{b}_{k,u}(i)-\textbf{f}^T_{\text{BB}_{k,u}}\textbf{F}^T_{\text{RF}}(i,:)\Vert _2^2,\\ \mathrm {s.t.}\,\,&\vert \textbf{F}^T_{\text{RF}}(i,j)\vert =1,\forall i,j. \end{aligned}$$

(39)

Problem (39) is similar to the constant modulus waveform design problem, which can be solved by a phase-only beamforming algorithm. First, we rewrite the objective function into the following form:

$$\begin{aligned}&\sum \limits _{k=1}^K\sum \limits _{u=1}^U\Vert \textbf{b}_{k,u}(i)-\textbf{f}^T_{\text{BB}_{k,u}}\textbf{F}^T_{\text{RF}}(i,:)\Vert _2^2\\&\quad =\sum \limits _{k=1}^K\sum \limits _{u=1}^U(\textbf{b}_{k,u}(i)-\textbf{f}^T_{\text{BB}_{k,u}}\textbf{F}^T_{\text{RF}}(i,:))^H(\textbf{b}_{k,u}(i)-\textbf{f}^T_{\text{BB}_{k,u}}\textbf{F}^T_{\text{RF}}(i,:))\\&\quad =\sum \limits _{k=1}^K\sum \limits _{u=1}^U\textbf{b}^H_{k,u}(i)\textbf{b}_{k,u}(i)+\textbf{F}^*_{\text{RF}}(i,:)\sum \limits _{k=1}^K\sum \limits _{u=1}^U\textbf{f}^*_{\text{BB}_{k,u}}\textbf{f}^T_{\text{BB}_{k,u}}\textbf{F}^T_{\text{RF}}(i,:)\\&\quad -\sum \limits _{k=1}^K\sum \limits _{u=1}^U\textbf{b}^T_{k,u}(i)\textbf{f}^T_{\text{BB}_{k,u}}\textbf{F}^T_{\text{RF}}(i,:)-\textbf{F}^*_{\text{RF}}(i,:)\sum \limits _{k=1}^K\sum \limits _{u=1}^U\textbf{f}^*_{\text{BB}_{k,u}}\textbf{b}_{k,u}(i)\\&\quad =\textbf{f}_{\text{RF}}^H\textbf{P}\textbf{f}_{\text{RF}}-\textbf{q}^H\textbf{f}_{\text{RF}}-\textbf{f}_{\text{RF}}^H\textbf{q}+r, \end{aligned}$$

where $\textbf{f}_{\text{RF}}=\textbf{F}^T_{\text{RF}}(i,:)$, $\textbf{P}=\mathop {\sum }\limits _{k=1}^{K}\mathop {\sum }\limits _{u=1}^{U}\textbf{f}^{*}_{\text{BB}_{k,u}}\textbf{f}^T_{\text{BB}_{k,u}}$, $\textbf{q}=\mathop {\sum }\limits _{k=1}^{K}\mathop {\sum }\limits _{u=1}^{U}\textbf{f}^{*}_{\text{BB}_{k,u}}\textbf{b}_{k,u}(i)$ and $r=\sum \limits _{k=1}^K\sum \limits _{u=1}^U\textbf{b}^H_{k,u}(i)\textbf{b}_{k,u}(i)$. Furthermore, problem (39) can be recast as

$$\begin{aligned}&\min \limits _{\textbf{f}_{\text{RF}}}&\Vert \textbf{P}^{\frac{-1}{2}}\textbf{q}-\textbf{P}^{\frac{1}{2}}\textbf{f}_{\text{RF}}\Vert _2^2, \end{aligned}$$

(40a)

$$\begin{aligned}&\mathrm {s.t.}&\vert \textbf{f}_{\text{RF}}(n)\vert =1,\quad n=1,...,N_{\text{RF}_t}. \end{aligned}$$

(40b)

Obviously, (40) is a constant modulus least squares problem, and its optimal solution can be obtained by the VSGP algorithm [37], the successive closed forms algorithm [38], or the fixed step size gradient projection algorithm [39]. In this article, we use the VSGP algorithm. According to the above steps, the proposed CADMM algorithm for designing the hybrid transmitter and user receiver is summarized in Algorithm 1.

3.2 Hybrid radar receiver design

In this section, we update the radar hybrid receivers $\textbf{W}_{\text{RF}}$ and $\textbf{w}_{\text{BB}_k}$ for fixed hybrid transmitters. In this case, the sub-problem of $\textbf{W}_{\text{RF}}$ and $\textbf{w}_{\text{BB}_k}$ can be formulated as a non-weighted problem, i.e.

$$\begin{aligned}&\max \limits _{\textbf{W}_{\text{RF}},\textbf{w}_{\text{BB}_k}}&\sum \limits _{k=1}^K\text{SINR}_k, \end{aligned}$$

(41a)

$$\begin{aligned}&\mathrm {s.t.}&\vert \textbf{W}_{\text{RF}}(i,j)\vert =1,\forall i,j. \end{aligned}$$

(41b)

Due to the objective function of problem (41) is a nonconvex function, this implies that it is impossible to directly optimize $\textbf{W}_{\text{RF}}$ and $\textbf{w}_{\text{BB}_k}$. Therefore, we first use generalized eigenvalue decomposition to obtain the optimal digital receiver $\textbf{v}_k$ for each subcarrier, and then update $\textbf{W}_{\text{RF}}$ and $\textbf{w}_{\text{BB}_k}$ with the least square method. For the definition $\textbf{v}_{k}=\textbf{W}_{\text{RF}}\textbf{w}_{\text{BB}_k}$, $\forall k$, problem (41) can be formulated as

$$\begin{aligned} \max \limits _{\textbf{v}_{k}}\sum \limits _{k=1}^K\frac{\textbf{v}^H_{k}{\varvec{\Upsilon }}_{g,k}\textbf{v}_{k}}{\textbf{v}^H_{k}{\varvec{\Upsilon }}_{in,k}\textbf{v}_{k}}, \end{aligned}$$

(42)

Where ${\varvec{\Upsilon }}_{g,k}$ and ${\varvec{\Upsilon }}_{in,k}$ are defined as follows

$$\begin{aligned} {\varvec{\Upsilon }}_{g,k} =&\sigma ^2_g\textbf{A}(\varphi _g,k)\textbf{F}_{\text{RF}}\textbf{F}_{\text{BB}_k}\textbf{F}_{\text{BB}_k}^H\textbf{F}_{\text{RF}}^H\textbf{A}^H(\varphi _g,k), \end{aligned}$$

(43)

$$\begin{aligned} {\varvec{\Upsilon }}_{in,k}=&\sum \limits _{j=1}^J\sigma ^2_j\textbf{A}(\varphi _j,k)\textbf{F}_{\text{RF}}\textbf{F}_{\text{BB}_k}\textbf{F}_{\text{BB}_k}^H\textbf{F}_{\text{RF}}^H\textbf{A}^H(\varphi _j,k)+\sigma ^2_{\bar{n}}\textbf{I}_{N_{rr}}. \end{aligned}$$

(44)

Obviously, we can easily obtain the optimal solution of the problem (42) by the maximum generalized eigenvalue of $({\varvec{\Upsilon }}_{g,k},{\varvec{\Upsilon }}_{in,k})$ or the principal eigenvector of ${\varvec{\Upsilon }}^{-1}_{in,k}{\varvec{\Upsilon }}_{g,k}$.

Then, we update $\textbf{W}_{\text{RF}}$ and $\textbf{w}_{\text{BB}_k}$ by solving the following problem

$$\begin{aligned}&\min \limits _{\textbf{W}_{\text{RF}},\textbf{w}_{\text{BB}_k}}&\sum \limits _{k=1}^K\Vert \textbf{v}_k-\textbf{W}_{\text{RF}}\textbf{w}_{\text{BB}_k} \Vert _2^2, \end{aligned}$$

(45a)

$$\begin{aligned}&\mathrm {s.t.}&\vert \textbf{W}_{\text{RF}}(i,j)\vert =1,\forall i,j. \end{aligned}$$

(45b)

For the above problem, we can design the analog combiner $\textbf{W}_{\text{RF}}$ and the digital combiner $\textbf{w}_{\text{BB}_k}$ with an alternate optimization algorithm.

3.2.1 Update of $\{\textbf{w}_{\text{BB}_{k}}\}$

We first consider the design of the digital combiner $\textbf{w}_{\text{BB}_k}$ based on the fixed analog combiner $\textbf{W}_{\text{RF}}$. Therefore, problem (45) can be rewritten as an unconstrained problem, i.e.

$$\begin{aligned}&\min \limits _{\textbf{W}_{\text{RF}},\textbf{w}_{\text{BB}_k}}&\sum \limits _{k=1}^K\Vert \textbf{v}_k-\textbf{W}_{\text{RF}}\textbf{w}_{\text{BB}_k} \Vert _2^2, \end{aligned}$$

(46)

whose optimal solution is

$$\begin{aligned} \textbf{w}_{\text{BB}_k}=(\textbf{W}^H_{\text{RF}}\textbf{W}_{\text{RF}})^{-1}\textbf{W}_{\text{RF}}\textbf{v}_k. \end{aligned}$$

(47)

3.2.2 Update of $\textbf{W}_{\text{RF}}$

Similar to the analog beamformer $\textbf{F}_{\text{RF}}$, with given other beamformers, the sub-problem to $\textbf{W}_{\text{RF}}$ can be recast as a parallel sub-problem. Specifically, for the i-th row of $\textbf{W}_{\text{RF}}$, the sub-problem can be expressed as

$$\begin{aligned}&\min \limits _{\textbf{W}_{\text{RF}}(i,:)}&\sum \limits _{k=1}^K\Vert \textbf{v}_k(i)-\textbf{w}^T_{\text{BB}_k}\textbf{W}^T_{\text{RF}}(i,:) \Vert _2^2, \end{aligned}$$

(48a)

$$\begin{aligned}&\mathrm {s.t.}&\vert \textbf{W}_{\text{RF}}(i,j)\vert =1,\forall i,j. \end{aligned}$$

(48b)

Further, this problem can be rewritten as

$$\begin{aligned}&\min \limits _{\textbf{w}_{\text{RF}}}&\Vert \overline{\textbf{P}}^{\frac{-1}{2}}\overline{\textbf{q}}-\overline{\textbf{P}}^{\frac{1}{2}}\textbf{w}_{\text{RF}}\Vert _2^2, \end{aligned}$$

(49a)

$$\begin{aligned}&\mathrm {s.t.}&\vert \textbf{w}_{\text{RF}}(n)\vert =1,\quad n=1,...,N_{\text{RF}_r}, \end{aligned}$$

(49b)

where $\overline{\textbf{q}}=\sum \limits _{k=1}^K\textbf{w}_{\text{BB}_k}^*\textbf{v}_{k}(i)$ and $\overline{\textbf{P}}=\sum \limits _{k=1}^K\textbf{w}_{\text{BB}_k}^*\textbf{w}_{\text{BB}_k}^T$. Problem (49) can also be solved by the VSGP algorithm.

Finally, the proposed alternating optimization algorithm for the hybrid radar receiver is summarized in Algorithm 2. In fact, the SINR of the radar received signal is affected by the hybrid transmitter and hybrid receiver, which implies that we need to alternately optimize the hybrid transmitter and hybrid receiver. Therefore, the final algorithm for the five variables is summarized in Algorithm 3.

4 Performance analysis

4.1 Convergence analysis

Let $g^k$ be the sum of SE and SINR values obtained by Algorithm 3 at the k-th iteration. Since the update of each block in the method with other variable fixed, the objective in (19) monotonically non-increasing after each iteration, it also means that the sum of SINR and SE is monotonically non-decreasing. i.e.

$$\begin{aligned} g^k \ge g^{k-1} . \end{aligned}$$

(50)

Then, we can conclude the proposed algorithm will converge to at least a local optimum since the value of the objective in (19) is bounded. This will be verified by simulations in the next section.

4.2 Complexity analysis

The main computational complexities of consensus-ADMM method for hybrid transmitter are contributed by solving subproblems (30), (35) and (37). More precisely, solving (32) needs updating $\textbf{u}_{k,u}$ and $\textbf{f}_{k,u}$. The update of $\textbf{u}_{k,u}$ need $\mathcal {O}\left( N_{cr}^{3}\right)$, and the update of $\textbf{f}_{k,u}$ according to (32) needs to compute $\Delta F(\textbf{f}_{k,u})$ and ${\varvec{\Psi }}_{k,u}$. Their complexities are $\mathcal {O}\left( N_t^{2}\right)$ and $\mathcal {O}\left( N_t^{2}\right)$, so the complexities of $\textbf{f}_{k,u}$ is $\mathcal {O}\left( N_t^{2}\right)$. Solving (35) needs a complexity of $\mathcal {O}\left( N_t^{3}\right)$. Solving (37) with VSGP method needs a complexity of $\mathcal {O}\left( I_1 N_{\text{t}} N_{\text{RF}}^2\right)$, where $I_1$ s the number of iterations required in VSGP method. To summarize, the overall complexity of the consensus-ADMM method is $\mathcal {O}\left( I_0(N_{cr}^3+N_t^3 + I_1N_{\text{t}} N_{\text{RF}}^2)\right)$, where $I_0$ is the number of consensus-ADMM iterations. Now we discuss the complexity of the hybrid receiver design method, which consists of three steps. The update of $\textbf{v}_{k}$ need $\mathcal {O}\left( N_{rr}^{3}\right)$, the update of $\textbf{w}_{\text{BB}_k}$ need $\mathcal {O}(N_{rr}^{3})$ and the update of $\textbf{W}_{\text{RF}}$ with aid of VSGP method takes a complexity of $\mathcal {O}(I_2 N_{\text{rr}} N_{\text{RF}_r}^2)$, $I_2$ s the number of iterations required in VSGP method. Therefore, the overall complexity of the hybrid receiver design method is $\mathcal {O}(I_3(N_{rr}^3+I_2N_{\text{rr}} N_{\text{RF}_r}^2))$, where $I_3$ is the number of outer iterations. As a result, the overall complexity of the proposed algorithm is given by $\mathcal {O}(I_4(I_0(N_{cr}^3+N_t^3 + I_1N_{\text{t}} N_{\text{RF}}^2)+I_3(N_{rr}^3+I_2N_{\text{rr}} N_{\text{RF}_r}^2)))$, where $I_4$ is the number of outer iterations.

5 Results and discussion

In this section, we will provide numerical simulations to verify the performance of the proposed algorithms. The simulation parameters are set as follows unless specified otherwise. We consider that the DFRC system has $N_t=64$ transmit antennas and $N_{\text{RF}_t}=16$ RF chains. This transmitter serves $U=4$ users, and each user is equipped with $N_{cr}=4$ receive antennas. We assume that the receive antennas of the radar hybrid combiner is $N_{rr}=32$ and $N_{\text{RF}_r}=4$ RF chains. The center frequency of the DRFC system is set to $f_{\text{cent}}=10$ GHz, and the frequency step of OFDM is $\Delta f=20$ MHz. The number of subcarriers in the OFDM system is set to $K=128$, and $U=4$ data streams are transmitted on each subcarrier. Furthermore, we consider that per subcarrier channel with 5 clusters and 10 scatters per cluster in which the angels of arrival (departure) are generated according to Laplacian distribution with random mean cluster angels $\bar{\theta }_i\in [0,2\pi )$ ($\bar{\phi }_i\in [0,2\pi )$) and angular spreads of 10 degrees within each cluster. The number of cyclic prefix lengths is $D=16$. The delay $\tau _i$ of each cluster is uniformly distributed in $[0, DT_s]$, with $T_s=1$. We assume that the power per user of the transmitter on each subcarrier is the same, and the power on each subcarrier $P_k=1$. Further, we assume that the radar target is located at $\varphi _g=20^{\circ }$ and the power is $\sigma _g^2=20$ dB. There are 2 interferences in the detection area, their directions are $\varphi _1=-50^{\circ }$ and $\varphi _2=40^{\circ }$, and the power of each interference is $\sigma _j^2=30$ dB. The noise variance of the user and radar receiver are assumed to be $\sigma _{\hat{n}}^2=0$ dB and $\sigma _{\bar{n}}^2=0$ dB, respectively. For the proposed method, the initial penalty parameter and convergence tolerance are set to be $\rho _{k,u}=0.3,\forall k,u$, $\epsilon _{\text{CADMM}}=10^{-4}$ and $\epsilon _{\text{ADMM}}=10^{-3}$, respectively. In addition, We set the convergence tolerance of VSGP to $\epsilon _{\text{RF}}=10^{-5}$.

Fig. 1

The convergence behaviors of the consensus-ADMM when $N_t=64$, $N_{\text{RF}}=16$ $U=4$, and $\eta =0.5$

Fig. 2

The communication spectral efficiency and radar signal-to-noise ratio versus different penalty coefficients

Fig. 3

The normalized spectral efficiency and radar SINR versus tradeoff parameter $\eta$ with the communication $\text{SNR}=0$ dB

Fig. 4

The convergence behaviors of the CADMM when$N_t=64$, $N_{\mathrm {RF_t}}=16$, $N_{{cr}}=4$, $U=4$ and $N_{{rr}}=32$, $N_{\mathrm {RF_r}}=4$

Fig. 5

The SINR of radar versus the for different numbers of radar RF chains with $U=4$, $N_{{t}}=32$, $N_{\mathrm {RF_r}}=8$, $N_{rr}=32$, and $\eta =0.5$

Fig. 6

The transmit beampattern versus tradeoff parameter $\eta$ with the $\text{SNR}=0$ dB

Fig. 7

The beampattern behaviors for different number of interference

Fig. 8

The averaged spectral efficiency and radar SINR versus the number of users with $N_t=64$, $N_{\mathrm {RF_t}}=8$, $N_{{cr}}=4$, $N_{{rr}}=32$, $N_{\mathrm {RF_r}}=4$, $\eta =0.5$ and $\text{SNR}=0$ dB

Fig. 9

The spectral efficiency and radar SINR versus the number of RF chains with $N_t=64$, $U=4$, $N_{{rr}}=32$, $N_{\mathrm {RF_r}}=4$, $\eta =0.5$ and $\text{SNR}=0$ dB

Figure 1 reveals the sum of SE and SINR values versus the iteration number. The result shows that the objective value obtained by the consensus-ADMM decrease with the iteration number increasing. This verifies the effectiveness of the proposed consensus-ADMM. In addition, Fig. 1 also displays two direct criteria concerning communication and radar functions (i.e., SE and radar SINR) versus the iteration number. The result reveals that both communication and radar performance becomes better as the iteration number increasing. This illustrates the proposed method works well for the OFDM-DFRC system. This illustrates the proposed method works well for the OFDM-DFRC system.

Figure 2 shows the communication spectral efficiency and radar signal-to-noise ratio versus different penalty coefficients. From the figure, we can see that the system performance obtained under different penalty coefficients is same. Hence, when the value of the penalty coefficient is reasonable, the penalty coefficient has no effect on the algorithm.

Figure 3 analyzes the impact of the trade-off parameter $\eta$ on the performance of the DFRC system. For the hybrid transmit beamformer, hybrid receive beamformer, and fully digital receiver, we, respectively, calculated the normalized radar receive SINR and communication SE. As shown in the figure, the curve of communication spectrum efficiency decreases as $\eta$ increases, and the curve of radar receive SINR increases as the trade-off coefficient increases. Further, we can choose the appropriate trade-off parameter according to the intersection of the two lines.

Figure 4 compares the convergence performance of the algorithm based on the phase-only method and the algorithm using the block coordinate descent (BCD) method by considering $\eta =0.5$. The result reveals that compared with the BCD method, the phase-only method has a significant convergence performance advantage, and the target performance achieved by the phase-only method is close to the performance achieved by the BCD method. In this method, we normalize the performance index, and the penalty coefficient $\rho$ is constant. In the iterative process, the increase of the weight ratio of the penalty term leads to the change of $\textbf{F}_{\text{RF}}$ and $\textbf{F}_{\text{BB}}$, while SE is sensitive to the change of $\textbf{F}_{\text{RF}}$ and $\textbf{F}_{\text{BB}}$, resulting in the continuous rise of the SE curve.

In Fig. 5, we aim to show the spectral efficiency of the DFRC system versus signal-to-noise ratios ($SNR=\mathcal {P}_k/\sigma ^2_{\hat{n}}$) for different trade-off parameters $\eta$. For comparison purposes, the fully digital beamformer is also provided. It can be observed in Fig. 3 that, the spectral efficiency decreases as the trade-off parameter $\eta$ increases from 0 to 1.

Figure 6 demonstrates the synthesized transmit beam patterns for different trade-off parameters $\eta$. From the results, we note that as the tradeoff parameter $\eta$ increases, the achieved beam pattern has a higher energy concentration on the target angle. Combining the analysis of the influence of the above-mentioned trade-off parameters $\eta$ on the spectrum efficiency, we should select an appropriate trade-off parameter to achieve the balance of communication and radar performance.

Figure 7 describes the transmit beam pattern under different number of interference. Further, we assume that the radar target is located at $\varphi _{g}=20^{\circ }$ and the power is $\sigma _{g}^{2}=20 \,\text{dB}$. There are 3 interferences in the detection area, their directions are $\varphi _{1}=-50^{\circ }$, $\varphi _{2}= 40^{\circ }$ and $\varphi _{3}=-40^{\circ }$, each interference is $\sigma _{j}^{2}=30 \,\text{dB}$. From the figure, we can conclude that the power of the transmit beam pattern at the target angle is not sensitive to the amount of interference items, because the performance indicator in the radar is SINR, which causes the transmit power to concentrate on the target azimuth.

In Figure 8, we describe the effect of the number of users on the performance of the fully digital beamformer and proposed method when $N_t=64$, $N_{\mathrm {RF_t}}=8$, $N_{{cr}}=4$, $N_{{rr}}=32$, $N_{\mathrm {RF_r}}=4$, $\eta =0.5$ and $\text{SNR}=0$ dB. In this work, we focus on designing a hybrid beamformer that approaches the performance of a fully digital precoder while reducing the number of RF chains. It can be considered that the performance obtained by the fully digital precoder is the upper bound of the performance that hybrid beamforming can achieve. Specifically, we introduce the average spectrum efficiency (i.e., SE/U) to replace the spectrum efficiency per subcarrier. It can be observed in Fig. 8 that, as the U increases, the average spectral efficiency of the DFRC system for users becomes worse. This is because the increase in the number of users leads to a stronger MUI received by each user. Furthermore, we noticed that increasing U will cause deterioration of radar SINR.

Next, we display the impact of the transmitter RF chains $N_{\text {RF}_t}$ on radar SINR and communication spectrum efficiency. It can be seen in Fig. 9 that, increasing the number of RF chains $N_{\text {RF}_t}$ can improve the communication spectrum efficiency and radar SINR. In addition, it can be observed that as the number of RF chains increases, the performance gap between the proposed method and the fully digital method gradually decreases.

6 Conclusion

We considered the wideband OFDM-DFRC system with a hybrid beamforming transceiver and the user receiving filter in this work. For the joint optimization problem of multiple beamformers, a trade-off optimization problem consisting of communication spectrum efficiency and radar signal SINR is modeled. To solve the non-convex optimization problem, a CADMM method based on block successive upper-bound minimization method is proposed to update the hybrid beamforming transmitter and user receiver. Furthermore, an approach based on the phase-only algorithm is proposed to optimize the hybrid analog precoder. Numerical results show that the proposed method guarantees radar and communication performance. In addition, our simulation results show that the performance of the two systems can be well balanced by setting appropriate trade-off parameters. In addition, the results also show that as the number of RF chains increases, the performance gap between the fully connected hybrid beamformer and the all-digital beamformer gradually decreases.

Appendix

1.1 Calculation of $\nabla F(\textbf{f}_{k,u})$ and $\mathcal {H}_{k,u}$

Note: $F(\textbf{f}_{k,u})=F_0(\textbf{f}_{k,u})+F_1(\textbf{f}_{k,u})$, with $F_0(\textbf{f}_{k,u})$ and $F_1(\textbf{f}_{k,u})$ are, respectively, defined as

$$\begin{aligned} F_0(\textbf{f}_{k,u})= & {} \beta _r\text{SINR}_{k,u} \end{aligned}$$

(51)

$$\begin{aligned} F_1(\textbf{f}_{k,u})= & {} \beta _c\text{SE}_{k,u} \end{aligned}$$

(52)

Furthermore, $\nabla F(\textbf{f}_{k,u})$ can be calculated by

$$\begin{aligned} \nabla F(\textbf{f}_{k,u})=\nabla F_0(\textbf{f}_{k,u})+\nabla F_1(\textbf{f}_{k,u}) \end{aligned}$$

(53)

where $\nabla F_0(\textbf{f}_{k,u})$ and $\nabla F_1(\textbf{f}_{k,u})$ can be computed by

$$\begin{aligned}&\nabla F_0(\textbf{f}_{k,u})=\frac{\alpha ({\varvec{\Theta }}_{g,k}-F_0(\textbf{f}_{k,u}){\varvec{\Theta }}_{in,k})\textbf{f}_{k,u}}{\textbf{f}^H_{k,u}{\varvec{\Theta }}_{in,k}\textbf{f}_{k,u}} \end{aligned}$$

(54)

$$\begin{aligned}&\nabla F_1(\textbf{f}_{k,u})=\frac{\beta \overline{\textbf{H}}_{k,u}\textbf{f}_{k,u}}{\left( \sum \limits _{i=1}^U\textbf{f}^H_{k,i}\overline{\textbf{H}}_{k,u}\textbf{f}_{k,i}+\sigma ^2_{\hat{n}}\textbf{u}^H_{k,u}\textbf{u}_{k,u}\right) \text{ln}2} \end{aligned}$$

(55)

Further, the complex Hessian matrix can be defined as

$$\begin{aligned} \mathcal {H}&= \left( \begin{array}{ll} \mathcal {H}_{\textbf{f}\textbf{f}} &{} \mathcal {H}_{\textbf{f}^*\textbf{f}} \\ \mathcal {H}_{\textbf{f}\textbf{f}^*} &{} \mathcal {H}_{\textbf{f}^*\textbf{f}^*} \end{array}\right) \end{aligned}$$

(56)

which is, however, complicated to directly calculate the complex Hessian matrix because the Hessian matrix contains four second-order complex derivatives. In this work, the complex Hessian matrix is transformed from the real Hessian matrix since the relationship between the second-order terms in the complex and real forms as shown below.

$$\begin{aligned} \mathcal {H}&= \frac{1}{4} \textbf{J}\overline{\mathcal {H}}\textbf{J}^{H} \end{aligned}$$

(57)

where $\overline{\mathcal {H}}$ is the real Hessian matrix, and $\textbf{J}$ is defined as

$$\begin{aligned} \textbf{J}&= \left( \begin{array}{cc} \textbf{I}_{N_t} &{} j \textbf{I}_{N_t} \\ \textbf{I}_{N_t} &{} -j \textbf{I}_{N_t} \end{array}\right) \end{aligned}$$

(58)

In addition (57) shows that the eigenvalues of the real Hessian matrix are twice the size of the eigenvalues of the complex Hessian matrix [40]. This means that the maximum eigenvalue of the complex Hessian matrix can be obtained by calculating the maximum eigenvalue of the real Hessian matrix.

To calculate the real Hessian matrix $\overline{\mathcal {H}}_{k,u}$, we first transform $F(\textbf{f}_{k,u})$ into a real-value form as $F(\textbf{f}_{k,u,r})=F_0(\textbf{f}_{k,u,r})+F_1(\textbf{f}_{k,u,r})$, with $F_0(\textbf{f}_{k,u,r})$ and $F_1(\textbf{f}_{k,u,r})$ are, respectively, defined as

$$\begin{aligned}&F_0(\textbf{f}_{k,u,r})= \alpha \frac{\textbf{f}^T_{k,u,r}{\varvec{\Theta }}_{g,k,r}\textbf{f}_{k,u,r}}{\textbf{f}^T_{k,u,r}{\varvec{\Theta }}_{in,k,r}\textbf{f}_{k,u,r}} \end{aligned}$$

(59)

$$\begin{aligned}&F_1(\textbf{f}_{k,u,r})= \beta \text{log}_2\left( 1+\frac{\textbf{f}^T_{k,u,r}\overline{\textbf{H}}_{k,u,r}\textbf{f}_{k,u,r}}{\sum \limits _{i\ne u}^U\textbf{f}^H_{k,i,r}\overline{\textbf{H}}_{k,u,r}\textbf{f}_{k,i,r}+\sigma ^2_{\hat{n}}\textbf{u}^H_{k,u,r}\textbf{u}_{k,u,r}}\right) \end{aligned}$$

(60)

where $\textbf{f}_{k,i,r}$, $\overline{\textbf{H}}_{k,u,r}$, $\textbf{u}_{k,u,r}$, ${\varvec{\Theta }}_{g,k,r}$ and ${\varvec{\Theta }}_{in,k,r}$ are the real-valued form of $\textbf{f}_{k,i}$, $\overline{\textbf{H}}_{k,u}$, $\textbf{u}_{k,u}$, ${\varvec{\Theta }}_{g,k}$ and ${\varvec{\Theta }}_{in,k}$, respectively.¹

Then, we have

$$\begin{aligned} \overline{\mathcal {H}}_{k,u} = \overline{\mathcal {H}}^0_{k,u} + \overline{\mathcal {H}}^1_{k,u} \end{aligned}$$

(61)

where $\overline{\mathcal {H}}^0_{k,u}$ and $\overline{\mathcal {H}}^1_{k,u}$ can be computed by

$$\begin{aligned} \overline{\mathcal {H}}^0_{k,u}=&\frac{2(\alpha {\varvec{\Theta }}_{g,k,r}-F_0(\textbf{f}_{k,u,r}){\varvec{\Theta }}_{in,k,r})}{\textbf{f}^T_{k,u,r}{\varvec{\Theta }}_{in,k,r}\textbf{f}_{k,u,r}}\nonumber \\&\quad -\frac{4\alpha ({\varvec{\Theta }}_{g,k,r}\textbf{f}_{k,u,r}\textbf{f}_{k,u,r}^T{\varvec{\Theta }}_{g,k,r}+{\varvec{\Theta }}_{in,k,r}\textbf{f}_{k,u,r}\textbf{f}^T_{k,u,r}{\varvec{\Theta }}_{g,k,r})}{(\textbf{f}^T_{k,u,r}{\varvec{\Theta }}_{in,k,r}\textbf{f}_{k,u,r})^2}\nonumber \\&\quad +\frac{8F_0(\textbf{f}_{k,u,r}){\varvec{\Theta }}_{in,k,r}\textbf{f}_{k,u,r}\textbf{f}^T_{k,u,r}{\varvec{\Theta }}_{in,k,r}}{(\textbf{f}^T_{k,u,r}{\varvec{\Theta }}_{in,k,r}\textbf{f}_{k,u,r})^2} \end{aligned}$$

(62)

and

$$\begin{aligned} \overline{\mathcal {H}}^1_{k,u}&=\frac{2\beta \overline{\textbf{H}}_{k,u,r}}{\left( \sum \limits _{i= 1}^U\textbf{f}^H_{k,i,r}\overline{\textbf{H}}_{k,u,r}\textbf{f}_{k,i,r}+\sigma ^2_{\hat{n}}\textbf{u}^H_{k,u,r}\textbf{u}_{k,u,r}\right) \text {ln}2}\nonumber \\&\quad -\frac{4\beta \overline{\textbf{H}}_{k,u,r}\textbf{f}_{k,u,r}\textbf{f}_{k,u,r}^T\overline{\textbf{H}}_{k,u,r}}{\left( \sum \limits _{i= 1}^U\textbf{f}^H_{k,i,r}\overline{\textbf{H}}_{k,u,r}\textbf{f}_{k,i,r}+\sigma ^2_{\hat{n}}\textbf{u}^H_{k,u,r}\textbf{u}_{k,u,r}\right) ^2\text {ln}2} \end{aligned}$$

(63)

Finally, we can get the complex Hessian matrix based on (57), and get the maximum eigenvalue of the complex Hessian matrix.

1.2 Derivation of (30)

Define $\nabla F(\textbf{f}^{(m)}_{k,u})=\textbf{t}_{k,u}=\textbf{t}_{re}+j\textbf{t}_{im}$, $\textbf{f}_{k,u}=\textbf{f}_{re}+j\textbf{f}_{im}$ and $\textbf{q}_{k,u} = \textbf{F}_{{\text{RF}}}\textbf{f}_{{{\text{BB}}_{k,u}}}-\textbf{d}_{k,u} = \textbf{q}_{re}+\textbf{q}_{im}$.

$$\begin{aligned}&F(\textbf{f}^{(m)}_{k,u})+2Re\left\{ \nabla ^HF(\textbf{f}^{(m)}_{k,u})(\textbf{f}_{k,u}-\textbf{f}^{(m)}_{k,u})\right\} \nonumber \\&\quad +\frac{\rho _{k,u}}{2}\Vert \textbf{f}_{k,u}-\textbf{F}_\text{RF}\textbf{f}_{\text{BB}_{k,u}}+\textbf{d}_{k,u} \Vert _2^2\nonumber \\&\quad = F(\textbf{f}^{(m)}_{k,u})-2Re\left\{ \nabla ^HF(\textbf{f}^{(m)}_{k,u})\textbf{f}^{(m)}_{k,u}\right\} +2Re\left\{ \nabla ^HF(\textbf{f}^{(m)}_{k,u})\textbf{f}_{k,u}\right\} \nonumber \\&\quad +\frac{\rho _{k,u}}{2}\Vert \textbf{f}_{k,u}-\textbf{F}_\text{RF}\textbf{f}_{\text{BB}_{k,u}}+\textbf{d}_{k,u} \Vert _2^2\nonumber \\&\quad =F(\textbf{f}^{(m)}_{k,u})-2Re\left\{ \nabla ^HF(\textbf{f}^{(m)}_{k,u})\textbf{f}^{(m)}_{k,u}\right\} +2(\textbf{t}_{re}^T\textbf{f}_{re}+\textbf{t}_{im}^T\textbf{f}_{im})\nonumber \\&\quad +\frac{\rho _{k,u}}{2}\left[ (\textbf{f}_{re}-\textbf{q}_{re})^T(\textbf{f}_{re}-\textbf{q}_{re})+(\textbf{f}_{im}-\textbf{q}_{im})^T(\textbf{f}_{im}-\textbf{q}_{im})\right] \nonumber \\&\quad =F(\textbf{f}^{(m)}_{k,u})-2Re\left\{ \nabla ^HF(\textbf{f}^{(m)}_{k,u})\textbf{f}^{(m)}_{k,u}\right\} +2Re\left\{ \textbf{t}^H_{k,u} \textbf{q}_{k,u}\right\} -\frac{2}{\rho _{k,u}}\Vert \textbf{t}_{k,u} \Vert _2^2\nonumber \\&\quad +\frac{\rho _{k,u}}{2}\left[ \left( \textbf{f}_{re}-\left( \textbf{q}_{re}-\frac{2\textbf{t}_{re}}{\rho _{k,u}}\right) \right) ^T\left( \textbf{f}_{re}-\left( \textbf{q}_{re}-\frac{2\textbf{t}_{re}}{\rho _{k,u}}\right) \right) \right. \nonumber \\&\quad \left. +\left( \textbf{f}_{im}-\left( \textbf{q}_{im}-\frac{2\textbf{t}_{im}}{\rho _{k,u}}\right) \right) ^T\left( \textbf{f}_{im}-\left( \textbf{q}_{im}-\frac{2\textbf{t}_{im}}{\rho _{k,u}}\right) \right) \right] \nonumber \\&\quad =\frac{\rho _{k,u}}{2}\Vert \textbf{f}_{k,u}-(\textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}-\textbf{d}_{k,u}-\frac{2\nabla F(\textbf{f}_{k,u}^{(m)})}{\rho _{k,u}}) \Vert _2^2\nonumber \\&\quad + F(\textbf{f}^{(m)}_{k,u})-2Re\left\{ \nabla ^HF(\textbf{f}^{(m)}_{k,u})\textbf{f}^{(m)}_{k,u}\right\} -\frac{2}{\rho _{k,u}}\Vert \nabla F(\textbf{f}_{k,u}^{(m)}) \Vert _2^2 \nonumber \\&\quad +2Re\left\{ \nabla F(\textbf{f}_{k,u}^{(m)})(\textbf{F}_\text{RF}\textbf{f}_{\text{BB}_{k,u}}-\textbf{d}_{k,u})\right\} \end{aligned}$$

(64)

Since the problem (29) w.r.t. $\textbf{f}_{k,u}$, the terms that do not contain $\textbf{f}_{k,u}$ in (64) can be regarded as constant terms when optimizing $\textbf{f}_{k,u}$. Therefore, the optimization problem for $\textbf{f}_{k,u}$ can be recast into the following form.

$$\begin{aligned}&\min \limits _{\textbf{f}_{k,u}}&\left\| \textbf{f}_{k,u}-\left( \textbf{F}_{\text{RF}}\textbf{f}_{\text{BB}_{k,u}}-\textbf{d}_{k,u}-\frac{2\nabla F(\textbf{f}_{k,u})}{\rho _{k,u}}\right) \right\| _2^2 \end{aligned}$$

(65a)

$$\begin{aligned}&\mathrm {s.t.}&\Vert \textbf{f}_{k,u}\Vert ^2_2=\frac{\mathcal {P}_k}{U} \end{aligned}$$

(65b)