### 2.1 Network description

We consider a downlink NOMA system as shown in Fig. 1, where the superposed signals from BS are reflected and transmitted to a pair of NOMA users,^{Footnote 1} i.e., the cell-edge user \(\textrm{U}_1\) and the cell-center user \(\textrm{U}_2\), simultaneously. The NOMA communication system is assisted by a STAR-RIS in the presence of two eavesdroppers^{Footnote 2} (i.e., \(\textrm{E}_1\), \(\textrm{E}_2\)), which are located on both sides of the STAR-RIS. Due to the serious blockage and complicated wireless environment, we assume that the links from the BS to both user \(\mathrm{U_1}\) and eavesdropper \(\mathrm{E_1}\) are negligible or even fall into complete outage status. Due to the long propagation distance between the BS and the cell-edge user, there will have severe obstacle blockage (buildings or trees) between the cell-edge user and the BS. Thus, it is reasonable to assume that the direct link is blocked for the cell-edge user. In the case that both the direct link and the reflecting link are considered for the cell-edge user, it will bring new challenges in the derivation of the security outage performance. Specifically, the distribution characteristics of the sum of the cascaded channel and the direct channel are reshaped. More specifically, the user \(\mathrm{U_2}\) and eavesdropper \(\mathrm{E_2}\) are located on the same side of the BS in comparison to STAR-RIS, which have ability to receive both the signal from the BS and the signal reflected by the STAR-RIS. However, the user \(\textrm{U}_1\) and eavesdropper \(\textrm{E}_1\) can receive only the signals refracted by the STAR-RIS. In the considered STAR-RIS-aided NOMA system, the direct links between the BS and the users located in the reflecting region are considered, whereas the direct links between the BS and the users located in the refracting area are neglected. This assumption is reasonable because the direct transmission links of the users in the refraction region will be blocked by the STAR-RIS.

It is assumed that the BS has *T* \(\ge\) 2 antennas, while both the NOMA users and the eavesdroppers are equipped with a single antenna. Besides, the STAR-RIS consists of \(M\times N\) (i.e., *M* rows and *N* columns) reconfigurable elements. In this paper, we employ the ES operating protocol for the STAR-RIS, in which the incident signal is split into two portions (refract and reflect signals) by each of the elements on the STAR-RIS. ES is a widely used protocol for STAR-RIS, which is considered in this paper. For the energy split mode, the incident signal can be divided into reflection and refraction signals through each reconfigurable unit. Each reconfigurable unit can adjust the energy ratio of reflection and refraction, i.e., the power split coefficient. Generally, the secrecy outage probability of the two NOMA users will be influenced by power split coefficients. We denote the channel vectors (or matrices) from BS to STAR-RIS, from STAR-RIS to user \(\mathrm{U_1}\), from STAR-RIS to eavesdropper \(\mathrm{E_1}\), from STAR-RIS to user \(\mathrm{U_2}\), from STAR-RIS to eavesdropper \(\mathrm{E_2}\), from BS to user \(\mathrm{U_2}\), and from BS to eavesdropper \(\mathrm{E_2}\) by \({\textbf{G}}\in {\mathbb {C}}^{M N\times T}\), \({\textbf{g}}_{1}\in {\mathbb {C}}^{1\times M N}\), \({\textbf{p}}_{1}\in {\mathbb {C}}^{1\times M N}\), \({\textbf{g}}_{2}\in {\mathbb {C}}^{1\times M N}\), \({\textbf{p}}_{2}\in {\mathbb {C}}^{1\times M N}\), \({\textbf{g}}_{3}\in {\mathbb {C}}^{1\times T}\), and \({\textbf{p}}_{3}\in {\mathbb {C}}^{1\times T}\), respectively. Moreover, the channel vectors (matrices) can be modeled as \({{\textbf{G}}}=\sqrt{\alpha _{SR} d_{SR}^{-\eta _{0}}}{\textbf{H}}\), \({{\textbf{g}}_k}=\sqrt{\alpha _{\Pi _1 \Pi _2} d_{\Pi _1 \Pi _2}^{-\eta _{0}}}{\textbf{h}}_k\) with \(k \in \{1, 2, 3 \}\), and \({{\textbf{p}}_k}=\sqrt{\alpha _{\Pi _1 \Pi _3} d_{\Pi _1 \Pi _3}^{-\eta _{0}}}{\textbf{e}}_k\), where \(\alpha _{SR}\), \(\alpha _{\Pi _1 \Pi _2}\), and \(\alpha _{\Pi _1 \Pi _3}\) are the amplitude coefficients,^{Footnote 3}\(d_{SR}\), \(d_{\Pi _1 \Pi _2}\), and \(d_{\Pi _1 \Pi _3}\) are the distances from the \(\textrm{BS}\) to the STAR-RIS, from the node \(\Pi _1\) to \(\Pi _2\), and from the node \(\Pi _1\) to \(\Pi _3\), with \(\Pi _1 \in \{\textrm{BS}, \mathrm{STAR-RIS} \}\), \(\Pi _2 \in \{\textrm{U}_1, \textrm{U}_2 \}\), and \(\Pi _3 \in \{\textrm{E}_1, \textrm{E}_2 \}\), \({\textbf{H}}\), \({\textbf{h}}_k\) and \({\textbf{e}}_k\) represent the small scale fading of the channels and \(\eta _0\) denotes the path loss exponent. Moreover, we adopt the independent Rayleigh fading for all the wireless communication links, i.e., the elements of \({\textbf{H}}\), \({\textbf{h}}_k\) and \({\textbf{e}}_k\) are independent and identically distributed (i.i.d.) following the distribution of \({{\mathcal {C}}}{{\mathcal {N}}}(0,1)\). Let \({\textbf{H}}=[{\textbf{h}}_{sr}^{1t},...,{\textbf{h}}_{sr}^{it},...,{\textbf{h}}_{sr}^{M N t}]^{H}\), where \({\textbf{h}}_{sr}^{it}=[{h}_{sr}^{i1},..., {h}_{sr}^{it} ,...,{h}_{sr}^{iT}]\), \(t \in \{ 1,...,t,...,T\}\), \({\textbf{h}}_{1}=[h_{rU_{1}}^{1},...,h_{rU_{1}}^{i},...,h_{rU_{1}}^{M N}]\), \({\textbf{h}}_{2}=[h_{rU_{2}}^{1},...,h_{rU_{2}}^{i},...,h_{rU_{2}}^{M N}]\), and \({\textbf{h}}_{3}=[h_{sU_{2}}^{1},...,h_{sU_{2}}^{t},...,h_{sU_{2}}^{T}]\) denote the channel coefficients from the *t*-th transmit antenna on BS to the *i*-th reconfigurable element on STAR-RIS, from the *i*-th reconfigurable element to user \(\mathrm{U_{1}}\), from the *i*-th reconfigurable element to user \(\mathrm{U_{2}}\), and from the *t*-th transmit antenna to user \(\mathrm{U_{2}}\), respectively. In addition, the fading gains \(h_{sr}^{it}\), \(h_{rU_{\nu }}^{i}\), and \(h_{sU_{2}}^{t}\) are complex Gaussian distributed with zero mean and unit variance, i.e., \(h_{sr}^{it}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )\), \(h_{rU_{\nu }}^{i}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )\), and \(h_{sU_{2}}^{t}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )\).

The effective cascaded channel coefficient from the BS to the STAR-RIS and from the BS to node \(\Pi _2\cup \Pi _3\) can be written as \(\mathbf {\Xi }_{k} \mathbf {\Phi } {\textbf{H}}\), with \(\Xi _{k}\in \{ {\textbf{h}}_{k},{\textbf{e}}_{k} \}\), where \(\mathbf {\Phi } = \text {diag} \{\sqrt{\beta _{1}^{\chi _{\nu }}}e^{j \theta _{1}^{\chi _{\nu }}},... ,\sqrt{\beta _{i}^{\chi _{\nu }}}e^{j \theta _{i}^{\chi _{\nu }}},... ,\sqrt{\beta _{M N}^{\chi _{\nu }}}e^{j \theta _{M N}^{\chi _{\nu }}}\}\) denotes the refractive/reflective coefficient matrix of the STAR-RIS. Here, the notation \(\chi _{\nu }\) represents the transmission mode for the signal with \(\nu \in \{ 1, 2\}\), i.e., \(\chi _{1} = tra.\) and \(\chi _{2} = ref.\) represent the refracting and reflecting modes for the signal, respectively. \(\sqrt{\beta _{i}^{\chi _{\nu }}}\in [0,1]\) and \(\theta _{i}^{\chi _{\nu }}\in [0,2\pi )\) denote the ES coefficient and phase shift of the *i*-th element for reflecting or refracting responses, respectively. On account that the STAR-RIS is passive and the energy consumption of circuits is assumed to be ignorable, we obtain \(\beta _{i}^{tra.}+\beta _{i}^{ref.}=1\). Since the channel estimation and feedback processes are not the focus of this manuscript, the STAR-RIS is assumed to apply adjustable phase shifts that are controlled through a communication-oriented software. Similar to [33], it is assumed that the perfect channel status information (CSI) from the BS to STAR-RIS links, from the STAR-RIS to two NOMA users links and from the BS to cell-center user links are available, which corresponds to the best scenario in terms of system operation and constitutes a performance benchmark for practical implementations.^{Footnote 4}

### 2.2 Secrecy capacity for STAR-RIS-aided NOMA

According to the downlink NOMA transmissions, the users’ messages are distinguished in the power domain. At the BS, the transmit signals are superposition coded and beamformed, which can be expressed as

$$\begin{aligned} {{\textbf{x}}}={\textbf{w}}\left( \sqrt{\alpha _o P_s} s_1+\sqrt{(1-\alpha _o)P_s} s_2 \right) , \end{aligned}$$

(1)

where \({\textbf{x}}\in {\mathbb {C}}^{T\times 1}\) is the transmit vector, \({\textbf{w}}\in {\mathbb {C}}^{T\times 1}\) is the beamforming vector, \(s_{\nu }\) is the signal for \(\textrm{U}_{\nu }\) with unit power, \(P_s\) is the total transmit power, the PA coefficient of \(s_1\) is \(\alpha _o\) with \(0.5< \alpha _o < 1\), and the rest transmit power is allocated to the signal \(s_2\) [9].

It is clear that the optimal beamforming design should consider both users in NOMA systems. However, to the best our knowledge, the optimal beamformer cannot be achieved in the closed-form for two-user multi-antenna NOMA systems currently. As a result, the secrecy performance cannot be analyzed for the optimal beamformer in the considered multi-antenna NOMA downlinks. In order to obtain an analytical solution of the secrecy performance, a sub-optimal beamformer is adopted in this paper, i.e., the beamformer based on the channel of the cell-center user is used. According to the benefits of NOMA over OMA, the system performance can also be improved when the channel difference between two NOMA users is exaggerated. In this work, since the direct link of user \(\mathrm{U_2}\) is more helpful to the capacity of \(\mathrm{U_2}\) than the reflected link created by the STAR-RIS when *N* is not so large, we adopt the matched filter (MF) beamforming with respect to the direct link of \(\mathrm{U_2}\).^{Footnote 5} Thus, the beamforming vector with normalized power, \({\textbf{w}}\), can be written as

$$\begin{aligned} {{\textbf{w}}}= \frac{{\textbf{h}}_{3}^{H}}{| | {\textbf{h}}_{3}| | }. \end{aligned}$$

(2)

The BS broadcasts the superposed signals \({\textbf{x}}\) to a pair of NOMA users, the signals received by both NOMA users, denoted by \(y_{U_{1}}\) and \(y_{U_{2}}\), respectively, can be given by

$$\begin{aligned} y_{U_{1}}=\left( {\textbf{g}}_{1}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{U_{1}}, \end{aligned}$$

(3)

and

$$\begin{aligned} y_{U_{2}}=\left( {\textbf{g}}_{3}+{\textbf{g}}_{2}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{U_{2}}, \end{aligned}$$

(4)

where \(n_{U_{\nu }}\) is the additive white Gaussian noise (AWGN) with mean power \(N_{U_{\nu }}\) at \(\textrm{U}_{\nu }\).

From (3), the signal-plus-interference-to-noise ratio (SINR) of user \(\textrm{U}_{1}\) to decode the information \(s_{1}\) can be given by

$$\begin{aligned} \gamma _{{U_{1}}, s_{1}}= \frac{\alpha _o \rho _{U_{1}}| {\textbf{g}}_{1}\mathbf {\Phi }{\textbf{G}} {\textbf{w}} |^2 }{(1-\alpha _o)\rho _{U_{1}} | {\textbf{g}}_{1}\mathbf {\Phi }{\textbf{G}} {\textbf{w}}|^2+1}, \end{aligned}$$

(5)

where \(\rho _{U_{1}}=P_s/N_{U_{1}}\) denotes the signal-to-noise ratio (SNR) of user \(\textrm{U}_{1}\).

In addition, the cell-center user \(\textrm{U}_2\) performs SIC to first detect the signal \(s_1\) of user \(\textrm{U}_1\), then proceeds to subtract \(s_1\) and decode its own signal \(s_2\). Hence, the SINR to decode the information \(s_1\) and the SNR to decode the information \(s_2\) of user \(\textrm{U}_2\) can be, respectively, obtained by

$$\begin{aligned} \gamma _{{U_{2}}, s_{1}}= \frac{\alpha _o \rho _{U_{2}}| {\textbf{g}}_{3}{\textbf{w}}+{\textbf{g}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2 }{(1-\alpha _o)\rho _{U_{2}} |{\textbf{g}}_{3}{\textbf{w}}+{\textbf{g}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}}|^2+1}, \end{aligned}$$

(6)

and

$$\begin{aligned} \gamma _{{U_2}, s_2}= (1-\alpha _o)\rho _{U_2} | {\textbf{g}}_{3}{\textbf{w}}+{\textbf{g}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(7)

where \(\rho _{U_{2}}=P_s/N_{U_{2}}\) denotes the SNR of user \(\textrm{U}_{2}\).

We also assume that the eavesdroppers can utilize parallel interference cancellation (PIC), i.e., eavesdroppers can detect \(s_1\) (or \(s_2\)) without being interfered by \(s_2\) (or \(s_1\)). This assumption may overestimate the eavesdropper’s detection ability, which refers to the worst-case scenario of the considered system model.

The received signals at the eavesdropper \(\textrm{E}_{\nu }\) can be, respectively, given by

$$\begin{aligned} y_{E_{1}}=\left( {\textbf{p}}_{1}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{E_{1}}, \end{aligned}$$

(8)

and

$$\begin{aligned} y_{E_{2}}=\left( {\textbf{p}}_{3}+{\textbf{p}}_{2}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{E_{2}}, \end{aligned}$$

(9)

where \(n_{E_{\nu }}\) is the AWGN with mean power \(N_{E_{\nu }}\) at eavesdropper \(\textrm{E}_{\nu }\).

Furthermore, the SNR at the eavesdropper \(\textrm{E}_{\nu }\) to detect \(s_{\nu }\) can be formulated as

$$\begin{aligned} \gamma _{{E_{1}}, s_{1}}= \alpha _{o}\rho _{E_{1}} |{\textbf{p}}_{1}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(10)

and

$$\begin{aligned} \gamma _{{E_{2}}, s_{2}}= (1-\alpha _o)\rho _{E_{2}} |{\textbf{p}}_{3}{\textbf{w}}+ {\textbf{p}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(11)

where \(\rho _{E_{\nu }}=P_s/N_{E_{\nu }}\) denotes the ratio of the transmitting power of the signal source to the noise power at the eavesdropper \(\textrm{E}_{\nu }\). Let \({\textbf{e}}_{1}=[h_{rE_{1}}^{1},...,h_{rE_{1}}^{i},...,h_{rE_{1}}^{M N}]\), \({\textbf{e}}_{2}=[h_{rE_{2}}^{1},...,h_{rE_{2}}^{i},...,h_{rE_{2}}^{M N}]\), and \({\textbf{e}}_{3}=[h_{sE_{2}}^{1},...,h_{sE_{2}}^{t},...,h_{sE_{2}}^{T}]\) denote the channel coefficients from the *i*-th reconfigurable element to eavesdropper \(\mathrm{E_{1}}\), from the *i*-th reconfigurable element to eavesdropper \(\mathrm{E_{2}}\), and from the *t*-th transmit antenna at the BS to eavesdropper \(\mathrm{E_{2}}\). The fading gains \(h_{rE_{\nu }}^{i}\) and \(h_{sE_{2}}^{t}\) are complex Gaussian distributed with zero mean and unit variance, i.e., \(h_{rE_{\nu }}^{i}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )\), and \(h_{sE_{2}}^{t}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )\).

Based on the SINR (or SNR) derived above, the capacity of the in node \(\Pi _2\) to decode signal \(s_{\nu }\) is expressed as

$$\begin{aligned} C_{{\Pi _2}, s_\nu }= \log _2 \left( {{1+\gamma _{{\Pi _2}, s_\nu }}}\right) . \end{aligned}$$

(12)

Furthermore, the secrecy capacities obtained at user \(\textrm{U}_\nu\) are, respectively, given by

$$\begin{aligned} C_{S, U_\nu ,s_{\nu } } = {\left[ { {C_{U_\nu , s_{\nu }}-C_{E_{\nu } , s_\nu }}}\right] ^{+}}, \end{aligned}$$

(13)

and

$$\begin{aligned} C_{S, U_2, s_1 } = {\left[ { {C_{U_2, s_1}-C_{E_2 , s_1}}}\right] ^{+}}, \end{aligned}$$

(14)

where \([x]^+=\max \left\{ {x,0}\right\}\).

### 2.3 STAR-RIS-OMA

In this subsection, the STAR-RIS-aided OMA scheme is selected as a baseline for comparison, in which the STAR-RIS is deployed to assist the BS to send the information to both user \(\textrm{U}_1\) and user \(\textrm{U}_2\). Thus, the detecting SNR of node \(\Pi _2\cup \Pi _3\) for STAR-RIS-OMA scheme can be, respectively, given by

$$\begin{aligned} \gamma _{{\Pi _4}, s_{1}}^{OMA}= \rho _{\Pi _4} | \omega _1 \mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(15)

and

$$\begin{aligned} \gamma _{{\Pi _5}, s_{2}}^{OMA}= \rho _{\Pi _5} |\omega _{3} {\textbf{w}}+\omega _{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(16)

where \(\Pi _4 \in \{\mathrm{U_1}, \mathrm{E_1} \}\), \(\Pi _5 \in \{\mathrm{U_2}, \mathrm{E_2} \}\), and \(\omega \in \{{\textbf{g}},{\textbf{p}} \}\).

The entire wireless communication process includes two time slots for STAR-RIS-OMA networks. More specifically, in the first time slot, the BS sends the information \(s_1\) through STAR-RIS to refract to user \(\textrm{U}_1\), and the BS sends \(s_2\) to reflect to user \(\textrm{U}_2\) by the aid of STAR-RIS in the second slot. As a result, the capacity of the node \(\Pi _3\) to decode signal \(s_{\nu }\) can be expressed as

$$\begin{aligned} C_{{\Pi _3}, s_\nu }^{OMA}= \frac{1}{2} \log _2 \left( {{1+\gamma _{{\Pi _3}, s_\nu }^{OMA}}}\right) . \end{aligned}$$

(17)

Thus, the secrecy capacity obtained at user \(\textrm{U}_\nu\) is given by

$$\begin{aligned} C_{S, U_\nu , s_\nu }^{OMA} = {\left[ { {C_{U_\nu , s_\nu }^{OMA}-C_{E_{\nu }, s_\nu }^{OMA}}}\right] ^{+}}. \end{aligned}$$

(18)

### 2.4 Channel statistical properties

In this subsection, the channel statistical properties of cascaded Rayleigh channels are provided, which will be employed to evaluate secrecy outage behaviors for STAR-RIS-aided NOMA systems in the following sections.

In view of the above discussions, \(h_{sr}^{it}\) and \(h_{rU_{\nu }}^{i}\) are the channel coefficients from the BS to STAR-RIS and from STAR-RIS to user \(\mathrm{U_{\nu }}\), respectively. Unfortunately, the exact distribution of cascaded Rayleigh channels \(X_{\nu ,t}=\sum _{t=1}^{ T}\sum _{j=1}^{ N}|h_{rU_{\nu }}^{i}||{{\textbf{h}}}_{sr}^{it}{\textbf{w}}|\) is difficult to derive, we propose to approximate the PDF of \(X_{\nu ,t}\) as a Gamma distribution by exploiting the Laguerre series approximation [34], which can be given by

$$\begin{aligned} f_{X}(x)= \frac{x^{a}}{b^{a+1}\Gamma (a+1)} \exp \left( -\frac{x}{b} \right) , \end{aligned}$$

(19)

where \(\Gamma (\cdot )\) denotes the Gamma function [35], \(a=\frac{c^2}{d}-1\), \(b =\frac{d}{c}\) with \(c=\frac{ N\pi }{2}\), \(d=4N\left( 1-\frac{\pi ^2}{16} \right)\).

Moreover, the exact distribution of \(Y=|{\textbf{h}}_{3}{\textbf{w}}|^2=||{\textbf{h}}_{3}||^2\) follows the Gamma distribution with shape parameter *T* and scale parameter 1. Thus, the PDF of *Y* can be written as

$$\begin{aligned} f_{Y}(y)= \frac{y^{T-1}}{\Gamma (T)} \exp \left( -y \right) . \end{aligned}$$

(20)

Additionally, we assume that the distance of each row on STAR-RIS is less than half wavelength, and the distance of each column on STAR-RIS is larger than half wavelength. Thus, the received signals between rows are correlated, whereas the received signals between any two columns are independent. As a result, the correlation matrix of the *n*-th column on STAR-RIS is defined as^{Footnote 6}

$$\begin{aligned} {\Upsilon _{n}}=\left( {\begin{array}{cccc} 1 &{} {{\Lambda ^{n} _{12}}} &{} \cdots &{} {{\Lambda ^{n} _{1M}}} \\ {{\Lambda ^{n} _{12}}} &{} 1 &{} \cdots &{} {{\Lambda ^{n} _{2M}}} \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ {{\Lambda ^{n} _{1M}}} &{} {{\Lambda ^{n} _{2M}}} &{} \cdots &{} 1 \end{array}}\right) , \end{aligned}$$

(21)

where \(\Upsilon \in \{ W, V, Q, L, G \}\), \(\Lambda \in \{ \eta , \xi , \zeta , \phi , \varsigma \}\), \(W_n\) represent the receiving correlation matrix of the *n*-th column on the STAR-RIS, \(V_n\) and \(Q_n\) represent the refracting correlation matrix of the *n*-th column on the STAR-RIS for \(\textrm{U}_1\) and \(\textrm{E}_1\), and \(L_n\) and \(G_n\) represent the reflecting correlation matrix of the *n*-th column on the STAR-RIS for \(\textrm{U}_2\) and \(\textrm{E}_2\), respectively. In addition, \(\Lambda ^{n} _{m,m^{'}}\) \((1\le m \le M, 1\le m^{'} \le M, m\ne m^{'} )\) denotes the correlation coefficient between the *m*-th and the \(m^{'}\)-th element of the *n*-th column on the STAR-RIS.

Referring to (5), the SINR of user \(\textrm{U}_{1}\) to decode \(s_1\) can be formulated as

$$\begin{aligned} \gamma _{U_{1}, s_1} =\frac{\alpha _o \rho _{U_{1}} A_1\left( X_{U_1,t}e^{j\theta _{j}^{\chi _{1}}}\right) ^{2}}{(1-\alpha _o) \rho _{U_{1}}A_1\left( X_{U_1,t}e^{j\theta _{j}^{\chi _{1}}}\right) ^{2}+1}, \end{aligned}$$

(22)

where \(A_1=\frac{ \beta _{j}^{\chi _{1}}}{({d_{SR}d_{RU_{1}}})^{\eta _{0}}}\left( 1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\xi _{mm^{'}}^{n}\right)\), \(X_{U_1,t}=\sum _{t=1}^{T}\sum _{j=1}^{N} h_{rU_{1}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}}\), and \(\beta _{j}^{\chi _{1}}\) is the transmitting coefficient.

In order to obtain better system security performance, we propose a new scheme, that is, each of the reconfigurable elements serves user \(\textrm{U}_1\) and user \(\textrm{U}_2\) at the same time. Specifically, the reflective links serve user \(\textrm{U}_2\) and the refractive links serve user \(\textrm{U}_1\). It is well known that, the STAR-RIS can adjust the phase shift of each reconfigurable element according to the legitimate user’s channel information, so that the sum of signal amplitudes obtained by the legitimate user can be maximized. Unfortunately, since it is hard to obtain the eavesdropping CSI, we only need to maximize the SINR or SNR received by the legitimate users. For the cell-edge user, the data transmission is through the STAR-RIS link only. According to the SINR expression in Eq. (5), we can get that if the cascaded channel gain is the largest (i.e., \(\left| {\textbf{g}}_{\textbf{1}}\mathbf {\Phi Gw}\right|\) is the maximum), the SINR is the maximum for the cell-edge user. By applying some mathematical operations, the sub-optimal phase shifts for maximizing the SINR can be obtained by \(\theta _{j,opt.}^{\chi _{1}}=-arg[[h_{rU_{1}}^{j}][{\textbf{h}}_{sr}^{jt}{\textbf{w}}]]\). It is shown that the design of phase shifts depends on the sub-optimal beamforming vector. Thus, the phase shifts obtained here is also suboptimal.

### Proposition 1

The CDF of \(\gamma _{{U_{1}}, s_1}\) in (22) can be written as

$$\begin{aligned} F_{\gamma _{U_{1}, s_1}}(z) =\frac{\gamma {\left( a+1,\frac{1}{b}\sqrt{\frac{z}{B_1(\alpha _o-(1-\alpha _o)z)}} \right) }}{\Gamma (a +1)}, \end{aligned}$$

(23)

where \(B_{1}= \alpha _o \rho _{U_{1}} A_1\), and \(\gamma (x,y)\) denotes the lower incomplete Gamma function [35].

### Proof

Please refer to “Appendix 1”.

For the cell-center user, there exist a direct link and a cascaded channel. According to the SINR expressions in Eqs. (6)–(7), it is obvious that when the overall channel gain (the direct link and the cascaded channel) is the largest, the SINR/SNR achieves the largest. In such case, it requires that \(\left| {\textbf{g}}_{\textbf{3}}{\textbf{w}}+{\textbf{g}}_{\textbf{2}}\mathbf {\Phi Gw}\right|\) is the maximum. Thus, the sub-optimal phase shifts for the cell-center user are denoted by \(\theta _{j,opt.}^{\chi _{2}}=arg[h_{sU_{2}}^{t}]-arg[[h_{rU_{2}}^{j}][{\textbf{h}}_{sr}^{j}{\textbf{w}}]]\). Let \(X_{2,t}=\sum _{t=1}^{T}\sum _{j=1}^{N}|h_{rU_{2}}^{j}||{\textbf{h}}_{sr}^{jt}{\textbf{w}}|\), \(Y_1=||{\textbf{h}}_{3}||\).

Based on NOMA, the SINR and SNR at cell-center user \(\textrm{U}_2\) can be formulated as

$$\begin{aligned} \gamma _{{U_{2}}, s_{1}}= \frac{\alpha _o \rho _{U_{2}}\left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2 }{(1-\alpha _o)\rho _{U_{2}} \left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2+1}, \end{aligned}$$

(24)

and

$$\begin{aligned} \gamma _{{U_{2}}, s_{2}}= (1-\alpha _o)\rho _{U_{2}} \left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2, \end{aligned}$$

(25)

where \(A_2=\frac{ \beta _{j}^{\chi _{2}}}{({d_{SR}d_{RU_{2}}})^{\eta _{0}}}(1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\phi _{mm^{'}}^{n})\), \(C_1= d_{SU_2}^{-\eta _{0}}\).

In order to obtain the CDF of \(\gamma _{{U_{2}}, s_{\nu }}\), we first derive a close-form expression of \(Z_{1}^2\), with \(Z_{1}^{2}=\left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2\), in the following proposition.

### Proposition 2

The CDF of \(Z_1^2\) can be written as

$$\begin{aligned} F_{Z_1^2}(z) = \frac{{\textbf{B}}(2T+1,a+1) z^{T+\frac{a+1}{2}} }{T\Gamma (T)C_1^T\Gamma (a +1)b^{a+1}A_2^{\frac{a+1}{2}}} {}_1 \!F_1 \left( {a+1; } { 2T+a+2}; {- \frac{1}{b} \sqrt{\frac{z}{A_2}} } \right) , \end{aligned}$$

(26)

where \({\textbf{B}}(x,y)\) is Beta function and \({}_1 \!F_1\) is degenerate hypergeometric function [35].

### Proof

Please refer to “Appendix 2”.

Based on (26), the CDF of \(\gamma _{U_{2}, s_1}\) is given by

$$\begin{aligned} F_{\gamma _{U_{2}, s_1}}\left( x\right) =F_{Z_{1}^2}\left( \frac{x}{\rho _{u_2} (\alpha _o-(1-\alpha _o)x)}\right) . \end{aligned}$$

(27)

In order to avoid redundancy, the detailed derivations are no longer presented.

Referring to the solution process of \(F_{\gamma _{U_{2}, s_{1}}}(x)\), the CDF of \(\gamma _{{U_2}, s_{2}}\) can be written as

$$\begin{aligned} F_{\gamma _{U_{2}, s_{2}}}(x) =F_{Z_1^2}\left( \frac{x}{\rho _{u_2}(1-\alpha _o)x} \right) . \end{aligned}$$

(28)

Furthermore, referring to (10), we rewrite \(\gamma _{{E_{1}}, s_{1}}\) as

$$\begin{aligned} \gamma _{{E_{1}}, s_{1}}= \alpha _{o}\rho _{E_{1}}A_3 Z_{1,t}^2 , \end{aligned}$$

(29)

where \(A_3=\frac{ \beta _{j}^{\chi _{1}} }{({d_{SR}d_{RE_{1}}})^{\eta _{0} }}\left( 1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\zeta _{mm^{'}}^{n}\right)\), \(Z_{1,t}= \sum _{t=1}^{T}\sum _{j=1}^{N} h_{rE_{1}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}} e^{j\theta _{j}^{\chi _{1}}}\).

According to the central limit theorem (CLT), \(Z_{1,t}\) can be approximated with a complex Gaussian-distributed random variable. As a result, the square of \(Z_{1,t}^2\) can be approximated with an exponential random variable with parameter *N*, respectively.

Based on the above operations, the CDF of \(\gamma _{{E_{1}}, s_{1}}\) can be expressed as

$$\begin{aligned} F_{\gamma _{E_{1}, s_{1}}}(z) =1-e^{-\frac{z}{\lambda _{E_{1}}}}, \end{aligned}$$

(30)

where \(\lambda _{E_{1}}= \alpha _{o}N\rho _{E_{1}}A_3\).

Referring to (11), we rewrite \(\gamma _{{E_{2}}, s_{2}}\) as

$$\begin{aligned} \gamma _{{E_{2}}, s_{2}}= (1-\alpha _{o})\rho _{E_{2}}\left( \sqrt{A_4} Z_{2,t}+\sqrt{C_2}Y_2\right) ^2 , \end{aligned}$$

(31)

where \(A_4=\frac{ \beta _{j}^{\chi _{2}}}{d_{SR}^{\eta _{o}}d_{RE_2}^{\eta _{1} }}\left( 1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\varsigma _{mm^{'}}^{n}\right)\), \(C_2=d_{SE_{2}}^{-\eta _{1}}\), \(Y_{2}= {\textbf{e}}_3 {\textbf{w}} e^{j\theta _{s}}\), \(Z_{2,t}=\sum _{t=1}^{T} \sum _{j=1}^{N} h_{rE_{2}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}} e^{j\theta _{j}^{\chi _{2}}}\).

According to the CLT, \(\left( \sqrt{A_4} Z_{2,t}+\sqrt{C_2}Y_2\right)\) can be approximated with a complex Gaussian-distributed random variable. As a result, the \(\left( \sqrt{A_4} Z_{2,t}+\sqrt{C_2}Y_2\right) ^2\) can be approximated with an exponential random variable with parameter \(NA_4+C_2\), respectively.

After the above operations, the CDF of \(\gamma _{{E_{2}}, s_{2}}\) can be expressed as

$$\begin{aligned} F_{\gamma _{E_{2}, s_{2}}}(z) =1-e^{-\frac{z}{\lambda _{E_{2}}}}, \end{aligned}$$

(32)

where \(\lambda _{E_{2}}= (1-\alpha _{o})\rho _{E_{2}}\left( NA_4+C_2\right)\).