On secrecy performance analysis of multi-antenna STAR-RIS-assisted downlink NOMA systems

Abstract

To accommodate the stringent requirements of enhanced coverage quality and improved spectral efficiency, simultaneous transmitting and reflecting reconfigurable intelligent surface (STAR-RIS)-aided communication has been perceived as an interesting research topic. This paper investigates a downlink STAR-RIS-aided non-orthogonal multiple access (NOMA) system, where a STAR-RIS is deployed to enhance the transmission qualities between users and a multiple-antenna base station (BS). The considered STAR-RIS utilizes the energy splitting (ES) protocol to serve a pair of NOMA users located at both sides of STAR-RIS. Based on the ES protocol, each reconfigurable element can operate in the modes of transmission and reflection simultaneously. In an effort to characterize the secrecy performance, we first derive the closed-form expressions of secrecy outage probability (SOP) for STAR-RIS-aided NOMA system. Then, the asymptotic performance of the derived SOP is analyzed. For gleaning further insights, secrecy diversity order (SDO) is derived according to the asymptotic approximation in the high signal-to-noise ratio and main-to-eavesdropper ratio regimes. As a further advance, the system parameters are optimized to minimize the SOP of the system. Our analytical results demonstrate that the multiple-antenna BS has almost no impact on SDO for STAR-RIS-aided NOMA system. In simulations, it is demonstrated that the theoretical results match with the simulation results very well and the SOP of STAR-RIS-aided NOMA is less than that of conventional orthogonal multiple access (OMA) system obviously.

1 Introduction

The upcoming sixth-generation (6G) wireless networks have advantages of supplying enhanced spectral efficiency (SE), global coverage and more intelligence as well as meeting the requirements of dynamic businesses [1]. Reconfigurable intelligent surface (RIS) has been emerged as a promising technique for 6G systems [2]. Specifically, by utilizing a large number of passive reconfigurable elements, a RIS can intelligently control the phase shifts of incident signals to satisfy the requirements of various wireless applications. Its typical applications include wireless coverage and network throughput enhancement, interference cancelation, and so on [3,4,5]. Different from the conventional full-duplex relay, RIS is capable of manipulating the phase shifts of the incident signal without introducing self-interference.

As a promising technology, non-orthogonal multiple access (NOMA) has advantages in supplying high SE and supporting massive connectivity [6]. Moreover, different from the conventional orthogonal multiple access (OMA), NOMA transmission strategies can achieve better user fairness, in which superposition coding (SC), successive interference cancellation (SIC), and message passing (MP) are employed at the transmitters or receivers. As a result, NOMA transmission schemes are widely considered to be promising solutions to handle the emerging heterogeneous services and applications, and thus have drawn significant research interests in the past few years [7, 8]. By extending NOMA to cooperative communication, cooperative NOMA scheme was proposed in [9]. To further enhance SE, the authors of [10] analyzed the outage performance of cooperative NOMA systems. For an in-depth performance evaluation, the closed-form expressions of the outage probability and ergodic capacity for NOMA systems with multiple users were surveyed in [11, 12]. Apart from the above treatises, NOMA technique has been widely applied to a lot of communication scenarios. Due to the fact that both RIS and NOMA techniques can be leveraged to enhance SE, the intrinsic integration of RIS and NOMA techniques was applied to improve the performance of wireless networks [13].

Up to now, from the perspective of performance analysis, a RIS-aided NOMA network has been discussed in [14,15,16]. Particularly, the influence of coherent/random phase shifts on the outage behavior was studied for RIS-NOMA networks in [14]. Furthermore, a simple design of RIS-NOMA transmission scheme was proposed, where an increasing number of reflection elements can effectively reduce the outage probability [15]. Inspired by the above works, the authors of [16] investigated the outage probability, ergodic rate and energy efficiency (EE) of RIS-aided NOMA systems. As a further development, by collaboratively optimizing the transmit power and the phase shifts, the transmit power efficiency for RIS-enabled multi-group NOMA networks were examined in [17]. Given the user’s rate requirement, the authors of [18] investigated the transmit power minimization problem with discrete phase shifts for RIS-aided NOMA comparing to OMA. According to whether there is a direct link between base station (BS) and users, the outage performance of multiple RISs-assisted NOMA networks with discrete phase shifting was analyzed in [19]. Overall, the findings in [14,15,16, 19] showed a significant performance improvement achieved by either RIS-assisted uplink or downlink NOMA. Furthermore, the ergodic rate of RIS-assisted uplink and downlink NOMA networks was examined in [20], which revealed the superiority of RIS over full-duplex decode and forward (DF) relaying.

Despite the aforementioned advantages, the conventional RIS can only serve the users located on the same side. That is to say, the reflecting-only RIS offers only half-space coverage, which limits the flexibility of deployments. To overcome this limitation, the novel concept of simultaneous transmitting and reflecting RIS (STAR-RIS) [21] or intelligent omni-surface (IOS) [22] has been proposed. More specifically, in contrast to conventional reflecting-only RIS, the incident signal on STAR-RIS is divided into the transmitted and reflected signals propagating into both sides of the surface, and thus STAR-RIS is capable of achieving a full-space reconfigurable wireless environment. Currently, depending upon usage scenarios and foreseeable requirements, a STAR-RIS can operate in three different modes, namely, energy splitting (ES), mode switching (MS), and time switching (TS). Particularly, for the ES mode, each element may have a different reflection-transmission amplitude coefficient ratio and all elements can operate in transmission and reflection mode simultaneously. For the MS mode, each element can operate in full transmission or reflection mode. In addition, in the TS mode, all elements periodically switch between the transmission and reflection modes in the different time slots. As a result, by deploying STAR-RIS, the transmitters and receivers do not have to be located on the same side of the surface comparing to the conventional RIS. Motivated by this characteristic, growing research efforts have been devoted to examining the benefits of deploying STAR-RIS in wireless networks. For instance, the authors proposed three practical operating protocols for STAR-RIS and studied the corresponding joint beamforming design problems in both unicast and multicast scenarios in [23]. Moreover, the fundamental coverage limits of STAR-RIS were characterized by the authors of [24], where both NOMA and OMA schemes were considered.

The deployment of STAR-RIS is beneficial to NOMA systems. On one hand, for NOMA users with weak channel conditions, STAR-RIS is able to create stronger transmission links. On the other hand, since STAR-RIS has the ability to adjust channel gains of different NOMA users, it can offer a flexible decoding order according to the priority of users. Nevertheless, the research on STAR-RIS-aided NOMA networks is still in its infancy. For example, three STAR-RIS operating protocols in a single-cell NOMA network were firstly evaluated in a recent work [25]. As a further advance, the authors in [26] investigated the performance of a pair of NOMA users for STAR-RIS networks in terms of outage probability and ergodic rate. However, secure communication is still an intractable problem when there exist several eavesdroppers in the considered system model.

Physical layer security (PLS) has gained widely attentions and several techniques have been proposed to improve the secrecy performance of wireless systems, such as cooperative diversity and spatial diversity. From the perspective of information theory, PLS has been widely used to enhance the security performance of wireless communication systems. The key idea behind the PLS is to use the different characteristics between the legitimate channel and the eavesdropper’s channel. The perfect security can be achieved when the quality of the eavesdropper’s channel is inferior to that of the legitimate channel. Since PLS depends on the physical characteristics of the propagation environment, it is interesting to research the PLS performance with the assistance of RIS [27, 28]. More specifically, with the goal of maximizing the secrecy rate, the authors of [27] introduced some low complexity approaches by jointly designing the access point’s transmit beamforming and the RIS’s reflect beamforming. Moreover, from the perspectives of hardware and system design, the authors of [28] proposed a practical design for active RIS, and the secrecy performance of RIS-aided networks was also investigated. However, few works have studied the PLS of RIS-assisted NOMA systems. For example, the authors of [29] investigated the PLS of the downlink in RIS-aided NOMA networks in the presence of an eavesdropper. Besides, the authors in [30] proposed a novel scheme to enhance the PLS in RIS-aided NOMA network.

Since the aforementioned significant literatures have laid a solid foundation for understanding of NOMA, STAR-RIS, and PLS techniques in communication networks, it is an effective approach to enhance the SE and EE by integrating these three promising technologies. In [26], the performance of a pair of users, i.e., the cell-edge user and cell-center user, was investigated for STAR-RIS-aided NOMA networks in terms of outage probability and ergodic rate. However, the PLS of such a system has never been reported. Additionally, the authors of [31] studied the secrecy performance of a RIS-aided wireless communication system in the presence of an eavesdropping user with Rayleigh fading channels. Furthermore, the average required transmit power, the outage probability, and the diversity order were investigated of multi-antenna RIS-assisted NOMA network in [32]. To the best of our knowledge, the combination of PLS with STAR-RIS has not been reported in multi-antenna NOMA system. The STAR-RIS can not only improve the channel gain of legitimate users, but also reduce the received signal strength at the eavesdropper. Thus, by carefully designing the phase shifts of the STAR-RIS, the secrecy performance of NOMA system can be boosted in a cost-effective manner. In addition, the STAR-RIS-assisted NOMA communication system has greater advantages than the reflecting-only RIS-aided NOMA system because it can provide the service for the users located within 360-degree coverage area. Motivated by the aforementioned observations, in this work, we propose a novel secure communication scheme for multi-antenna STAR-RIS-assisted NOMA system. More specifically, we investigate the secrecy performance of two NOMA users with different requirements of quality of services (QoSs) in the presence of an eavesdropping user located at the same side of each NOMA user. The direct links from the BS to both the cell-center user and the eavesdropper are taken into account. According to the aforementioned explanations, the key contributions of this work are summarized as follows:

• 1. We propose a STAR-RIS-assisted scheme to enhance PLS with a multi-antenna BS, a STAR-RIS, a pair of users and two eavesdroppers in NOMA system, where the channel correlation between reconfiguration elements of each column on STAR-RIS is taken into consideration.

• 2. We analyze the secrecy performance of the proposed scheme. More specially, we derive the exact SOP with closed-form expressions. For the sake of comparison, we also derive the expression of the exact SOP for the multi-antenna STAR-RIS-assisted OMA scheme as a benchmark. To provide further useful insights, we derive the asymptotic expressions of the SOP for multi-antenna STAR-RIS-assisted NOMA as well as OMA networks. Besides, we also derive the expression of secrecy diversity order (SDO) for multi-antenna STAR-RIS-aided NOMA and OMA systems.

• 3. Since the power allocation (PA) coefficient and the ES coefficient have significant impact on the secrecy performance for multi-antenna STAR-RIS-aided NOMA communication schemes, for the sake of giving scope to the positive influence of the parameters on secrecy performance, the PA and the ES parameters are studied and optimized to reduce the system SOP in our proposed scheme.

The remainder of this paper is organized as follows. In Sect. 2, the system model of multi-antenna STAR-RIS-aided NOMA system is introduced. The secrecy outage behaviors of multi-antenna STAR-RIS-assisted NOMA networks are examined in Sect. 3. More specially, the exact expressions of SOP for cell-edge user and cell-center user are provided. The asymptotic SOP for multi-antenna STAR-RIS-aided NOMA system is evaluated in Sect.  4. In Sect. 5, the SDO of multi-antenna STAR-RIS-aided NOMA communication system is derived. The parameters of the multi-antenna STAR-RIS-NOMA scheme are optimized in Sect. 6. Simulation results and their discussions are presented in Sect. 7. Conclusion is drawn in Sect. 8.

Notations: The main notations in this paper used are shown as follows. The probability density function (PDF) and cumulative distribution function (CDF) of a random variable X are denoted by $f_{X}(\cdot )$ and $F_{X}(\cdot )$, $\Gamma (\cdot )$ denotes the Gamma function, respectively. $\Gamma (\cdot , \cdot )$ denotes the upper incomplete Gamma function, $\gamma (\cdot ,\cdot )$ denotes the lower incomplete Gamma function, respectively. The superscript $(\cdot )^{H}$ denotes conjugate-transpose operation and $|| \cdot ||$ means the 2-norm of vectors, respectively. ${\textbf{B}}(\cdot ,\cdot )$ denotes Beta functions and ${}_p \!F_q$ denotes Generalized hypergeometric series, respectively.

The aim of this paper is to analyze the secrecy performance of downlink NOMA based on a multi-antenna BS and a STAR-RIS assistance, specifically, analyzing the secrecy outage probability and secrecy diversity order of this STAR-RIS-aided NOMA system. For multi-antenna STAR-RIS-NOMA systems, we propose a transmit antenna beamforming and ES protocol for STAR-RIS scheme, which minimizes the secrecy outage probability. There exists a direct link between the BS and the cell-center user $\textrm{U}_2$, while the BS can only communicate with the cell-edge users $\textrm{U}_1$ through STAR-RIS. Specifically, the user $\textrm{U}_2$ and the eavesdropper $\textrm{E}_2$ can receive the superposition of the transmitted signal from the BS and the reflected signal from STAR-RIS, while the user $\textrm{U}_1$ and the eavesdropper $\textrm{E}_1$ can only receive the refraction signal from STAR-RIS. A brief introduction to the design idea: The BS employs multi-antenna technique to process transmit data in order to satisfy the rate requirements of the cell-center user. The STAR-RIS utilizes ES protocol and adjustable phase shifts to meet the rate requirements of both NOMA users. Since the PA and the ES coefficients have significant impact on the secrecy performance for multi-antenna STAR-RIS-aided NOMA communication schemes. We propose an effective method to optimize the PA and the ES parameters.

2 System model and secrecy capacity

2.1 Network description

We consider a downlink NOMA system as shown in Fig. 1, where the superposed signals from BS are reflected and transmitted to a pair of NOMA users,¹ i.e., the cell-edge user $\textrm{U}_1$ and the cell-center user $\textrm{U}_2$, simultaneously. The NOMA communication system is assisted by a STAR-RIS in the presence of two eavesdroppers² (i.e., $\textrm{E}_1$, $\textrm{E}_2$), which are located on both sides of the STAR-RIS. Due to the serious blockage and complicated wireless environment, we assume that the links from the BS to both user $\mathrm{U_1}$ and eavesdropper $\mathrm{E_1}$ are negligible or even fall into complete outage status. Due to the long propagation distance between the BS and the cell-edge user, there will have severe obstacle blockage (buildings or trees) between the cell-edge user and the BS. Thus, it is reasonable to assume that the direct link is blocked for the cell-edge user. In the case that both the direct link and the reflecting link are considered for the cell-edge user, it will bring new challenges in the derivation of the security outage performance. Specifically, the distribution characteristics of the sum of the cascaded channel and the direct channel are reshaped. More specifically, the user $\mathrm{U_2}$ and eavesdropper $\mathrm{E_2}$ are located on the same side of the BS in comparison to STAR-RIS, which have ability to receive both the signal from the BS and the signal reflected by the STAR-RIS. However, the user $\textrm{U}_1$ and eavesdropper $\textrm{E}_1$ can receive only the signals refracted by the STAR-RIS. In the considered STAR-RIS-aided NOMA system, the direct links between the BS and the users located in the reflecting region are considered, whereas the direct links between the BS and the users located in the refracting area are neglected. This assumption is reasonable because the direct transmission links of the users in the refraction region will be blocked by the STAR-RIS.

It is assumed that the BS has T $\ge$ 2 antennas, while both the NOMA users and the eavesdroppers are equipped with a single antenna. Besides, the STAR-RIS consists of $M\times N$ (i.e., M rows and N columns) reconfigurable elements. In this paper, we employ the ES operating protocol for the STAR-RIS, in which the incident signal is split into two portions (refract and reflect signals) by each of the elements on the STAR-RIS. ES is a widely used protocol for STAR-RIS, which is considered in this paper. For the energy split mode, the incident signal can be divided into reflection and refraction signals through each reconfigurable unit. Each reconfigurable unit can adjust the energy ratio of reflection and refraction, i.e., the power split coefficient. Generally, the secrecy outage probability of the two NOMA users will be influenced by power split coefficients. We denote the channel vectors (or matrices) from BS to STAR-RIS, from STAR-RIS to user $\mathrm{U_1}$, from STAR-RIS to eavesdropper $\mathrm{E_1}$, from STAR-RIS to user $\mathrm{U_2}$, from STAR-RIS to eavesdropper $\mathrm{E_2}$, from BS to user $\mathrm{U_2}$, and from BS to eavesdropper $\mathrm{E_2}$ by ${\textbf{G}}\in {\mathbb {C}}^{M N\times T}$, ${\textbf{g}}_{1}\in {\mathbb {C}}^{1\times M N}$, ${\textbf{p}}_{1}\in {\mathbb {C}}^{1\times M N}$, ${\textbf{g}}_{2}\in {\mathbb {C}}^{1\times M N}$, ${\textbf{p}}_{2}\in {\mathbb {C}}^{1\times M N}$, ${\textbf{g}}_{3}\in {\mathbb {C}}^{1\times T}$, and ${\textbf{p}}_{3}\in {\mathbb {C}}^{1\times T}$, respectively. Moreover, the channel vectors (matrices) can be modeled as ${{\textbf{G}}}=\sqrt{\alpha _{SR} d_{SR}^{-\eta _{0}}}{\textbf{H}}$, ${{\textbf{g}}_k}=\sqrt{\alpha _{\Pi _1 \Pi _2} d_{\Pi _1 \Pi _2}^{-\eta _{0}}}{\textbf{h}}_k$ with $k \in \{1, 2, 3 \}$, and ${{\textbf{p}}_k}=\sqrt{\alpha _{\Pi _1 \Pi _3} d_{\Pi _1 \Pi _3}^{-\eta _{0}}}{\textbf{e}}_k$, where $\alpha _{SR}$, $\alpha _{\Pi _1 \Pi _2}$, and $\alpha _{\Pi _1 \Pi _3}$ are the amplitude coefficients,³$d_{SR}$, $d_{\Pi _1 \Pi _2}$, and $d_{\Pi _1 \Pi _3}$ are the distances from the $\textrm{BS}$ to the STAR-RIS, from the node $\Pi _1$ to $\Pi _2$, and from the node $\Pi _1$ to $\Pi _3$, with $\Pi _1 \in \{\textrm{BS}, \mathrm{STAR-RIS} \}$, $\Pi _2 \in \{\textrm{U}_1, \textrm{U}_2 \}$, and $\Pi _3 \in \{\textrm{E}_1, \textrm{E}_2 \}$, ${\textbf{H}}$, ${\textbf{h}}_k$ and ${\textbf{e}}_k$ represent the small scale fading of the channels and $\eta _0$ denotes the path loss exponent. Moreover, we adopt the independent Rayleigh fading for all the wireless communication links, i.e., the elements of ${\textbf{H}}$, ${\textbf{h}}_k$ and ${\textbf{e}}_k$ are independent and identically distributed (i.i.d.) following the distribution of ${{\mathcal {C}}}{{\mathcal {N}}}(0,1)$. Let ${\textbf{H}}=[{\textbf{h}}_{sr}^{1t},...,{\textbf{h}}_{sr}^{it},...,{\textbf{h}}_{sr}^{M N t}]^{H}$, where ${\textbf{h}}_{sr}^{it}=[{h}_{sr}^{i1},..., {h}_{sr}^{it} ,...,{h}_{sr}^{iT}]$, $t \in \{ 1,...,t,...,T\}$, ${\textbf{h}}_{1}=[h_{rU_{1}}^{1},...,h_{rU_{1}}^{i},...,h_{rU_{1}}^{M N}]$, ${\textbf{h}}_{2}=[h_{rU_{2}}^{1},...,h_{rU_{2}}^{i},...,h_{rU_{2}}^{M N}]$, and ${\textbf{h}}_{3}=[h_{sU_{2}}^{1},...,h_{sU_{2}}^{t},...,h_{sU_{2}}^{T}]$ denote the channel coefficients from the t-th transmit antenna on BS to the i-th reconfigurable element on STAR-RIS, from the i-th reconfigurable element to user $\mathrm{U_{1}}$, from the i-th reconfigurable element to user $\mathrm{U_{2}}$, and from the t-th transmit antenna to user $\mathrm{U_{2}}$, respectively. In addition, the fading gains $h_{sr}^{it}$, $h_{rU_{\nu }}^{i}$, and $h_{sU_{2}}^{t}$ are complex Gaussian distributed with zero mean and unit variance, i.e., $h_{sr}^{it}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )$, $h_{rU_{\nu }}^{i}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )$, and $h_{sU_{2}}^{t}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )$.

The effective cascaded channel coefficient from the BS to the STAR-RIS and from the BS to node $\Pi _2\cup \Pi _3$ can be written as $\mathbf {\Xi }_{k} \mathbf {\Phi } {\textbf{H}}$, with $\Xi _{k}\in \{ {\textbf{h}}_{k},{\textbf{e}}_{k} \}$, where $\mathbf {\Phi } = \text {diag} \{\sqrt{\beta _{1}^{\chi _{\nu }}}e^{j \theta _{1}^{\chi _{\nu }}},... ,\sqrt{\beta _{i}^{\chi _{\nu }}}e^{j \theta _{i}^{\chi _{\nu }}},... ,\sqrt{\beta _{M N}^{\chi _{\nu }}}e^{j \theta _{M N}^{\chi _{\nu }}}\}$ denotes the refractive/reflective coefficient matrix of the STAR-RIS. Here, the notation $\chi _{\nu }$ represents the transmission mode for the signal with $\nu \in \{ 1, 2\}$, i.e., $\chi _{1} = tra.$ and $\chi _{2} = ref.$ represent the refracting and reflecting modes for the signal, respectively. $\sqrt{\beta _{i}^{\chi _{\nu }}}\in [0,1]$ and $\theta _{i}^{\chi _{\nu }}\in [0,2\pi )$ denote the ES coefficient and phase shift of the i-th element for reflecting or refracting responses, respectively. On account that the STAR-RIS is passive and the energy consumption of circuits is assumed to be ignorable, we obtain $\beta _{i}^{tra.}+\beta _{i}^{ref.}=1$. Since the channel estimation and feedback processes are not the focus of this manuscript, the STAR-RIS is assumed to apply adjustable phase shifts that are controlled through a communication-oriented software. Similar to [33], it is assumed that the perfect channel status information (CSI) from the BS to STAR-RIS links, from the STAR-RIS to two NOMA users links and from the BS to cell-center user links are available, which corresponds to the best scenario in terms of system operation and constitutes a performance benchmark for practical implementations.⁴

2.2 Secrecy capacity for STAR-RIS-aided NOMA

According to the downlink NOMA transmissions, the users’ messages are distinguished in the power domain. At the BS, the transmit signals are superposition coded and beamformed, which can be expressed as

$$\begin{aligned} {{\textbf{x}}}={\textbf{w}}\left( \sqrt{\alpha _o P_s} s_1+\sqrt{(1-\alpha _o)P_s} s_2 \right) , \end{aligned}$$

(1)

where ${\textbf{x}}\in {\mathbb {C}}^{T\times 1}$ is the transmit vector, ${\textbf{w}}\in {\mathbb {C}}^{T\times 1}$ is the beamforming vector, $s_{\nu }$ is the signal for $\textrm{U}_{\nu }$ with unit power, $P_s$ is the total transmit power, the PA coefficient of $s_1$ is $\alpha _o$ with $0.5< \alpha _o < 1$, and the rest transmit power is allocated to the signal $s_2$ [9].

It is clear that the optimal beamforming design should consider both users in NOMA systems. However, to the best our knowledge, the optimal beamformer cannot be achieved in the closed-form for two-user multi-antenna NOMA systems currently. As a result, the secrecy performance cannot be analyzed for the optimal beamformer in the considered multi-antenna NOMA downlinks. In order to obtain an analytical solution of the secrecy performance, a sub-optimal beamformer is adopted in this paper, i.e., the beamformer based on the channel of the cell-center user is used. According to the benefits of NOMA over OMA, the system performance can also be improved when the channel difference between two NOMA users is exaggerated. In this work, since the direct link of user $\mathrm{U_2}$ is more helpful to the capacity of $\mathrm{U_2}$ than the reflected link created by the STAR-RIS when N is not so large, we adopt the matched filter (MF) beamforming with respect to the direct link of $\mathrm{U_2}$.⁵ Thus, the beamforming vector with normalized power, ${\textbf{w}}$, can be written as

$$\begin{aligned} {{\textbf{w}}}= \frac{{\textbf{h}}_{3}^{H}}{| | {\textbf{h}}_{3}| | }. \end{aligned}$$

(2)

The BS broadcasts the superposed signals ${\textbf{x}}$ to a pair of NOMA users, the signals received by both NOMA users, denoted by $y_{U_{1}}$ and $y_{U_{2}}$, respectively, can be given by

$$\begin{aligned} y_{U_{1}}=\left( {\textbf{g}}_{1}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{U_{1}}, \end{aligned}$$

(3)

and

$$\begin{aligned} y_{U_{2}}=\left( {\textbf{g}}_{3}+{\textbf{g}}_{2}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{U_{2}}, \end{aligned}$$

(4)

where $n_{U_{\nu }}$ is the additive white Gaussian noise (AWGN) with mean power $N_{U_{\nu }}$ at $\textrm{U}_{\nu }$.

From (3), the signal-plus-interference-to-noise ratio (SINR) of user $\textrm{U}_{1}$ to decode the information $s_{1}$ can be given by

$$\begin{aligned} \gamma _{{U_{1}}, s_{1}}= \frac{\alpha _o \rho _{U_{1}}| {\textbf{g}}_{1}\mathbf {\Phi }{\textbf{G}} {\textbf{w}} |^2 }{(1-\alpha _o)\rho _{U_{1}} | {\textbf{g}}_{1}\mathbf {\Phi }{\textbf{G}} {\textbf{w}}|^2+1}, \end{aligned}$$

(5)

where $\rho _{U_{1}}=P_s/N_{U_{1}}$ denotes the signal-to-noise ratio (SNR) of user $\textrm{U}_{1}$.

In addition, the cell-center user $\textrm{U}_2$ performs SIC to first detect the signal $s_1$ of user $\textrm{U}_1$, then proceeds to subtract $s_1$ and decode its own signal $s_2$. Hence, the SINR to decode the information $s_1$ and the SNR to decode the information $s_2$ of user $\textrm{U}_2$ can be, respectively, obtained by

$$\begin{aligned} \gamma _{{U_{2}}, s_{1}}= \frac{\alpha _o \rho _{U_{2}}| {\textbf{g}}_{3}{\textbf{w}}+{\textbf{g}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2 }{(1-\alpha _o)\rho _{U_{2}} |{\textbf{g}}_{3}{\textbf{w}}+{\textbf{g}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}}|^2+1}, \end{aligned}$$

(6)

and

$$\begin{aligned} \gamma _{{U_2}, s_2}= (1-\alpha _o)\rho _{U_2} | {\textbf{g}}_{3}{\textbf{w}}+{\textbf{g}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(7)

where $\rho _{U_{2}}=P_s/N_{U_{2}}$ denotes the SNR of user $\textrm{U}_{2}$.

We also assume that the eavesdroppers can utilize parallel interference cancellation (PIC), i.e., eavesdroppers can detect $s_1$ (or $s_2$) without being interfered by $s_2$ (or $s_1$). This assumption may overestimate the eavesdropper’s detection ability, which refers to the worst-case scenario of the considered system model.

The received signals at the eavesdropper $\textrm{E}_{\nu }$ can be, respectively, given by

$$\begin{aligned} y_{E_{1}}=\left( {\textbf{p}}_{1}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{E_{1}}, \end{aligned}$$

(8)

and

$$\begin{aligned} y_{E_{2}}=\left( {\textbf{p}}_{3}+{\textbf{p}}_{2}{\varvec{\Phi }}{\textbf{G}}\right) {\textbf{x}}+n_{E_{2}}, \end{aligned}$$

(9)

where $n_{E_{\nu }}$ is the AWGN with mean power $N_{E_{\nu }}$ at eavesdropper $\textrm{E}_{\nu }$.

Furthermore, the SNR at the eavesdropper $\textrm{E}_{\nu }$ to detect $s_{\nu }$ can be formulated as

$$\begin{aligned} \gamma _{{E_{1}}, s_{1}}= \alpha _{o}\rho _{E_{1}} |{\textbf{p}}_{1}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(10)

and

$$\begin{aligned} \gamma _{{E_{2}}, s_{2}}= (1-\alpha _o)\rho _{E_{2}} |{\textbf{p}}_{3}{\textbf{w}}+ {\textbf{p}}_{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(11)

where $\rho _{E_{\nu }}=P_s/N_{E_{\nu }}$ denotes the ratio of the transmitting power of the signal source to the noise power at the eavesdropper $\textrm{E}_{\nu }$. Let ${\textbf{e}}_{1}=[h_{rE_{1}}^{1},...,h_{rE_{1}}^{i},...,h_{rE_{1}}^{M N}]$, ${\textbf{e}}_{2}=[h_{rE_{2}}^{1},...,h_{rE_{2}}^{i},...,h_{rE_{2}}^{M N}]$, and ${\textbf{e}}_{3}=[h_{sE_{2}}^{1},...,h_{sE_{2}}^{t},...,h_{sE_{2}}^{T}]$ denote the channel coefficients from the i-th reconfigurable element to eavesdropper $\mathrm{E_{1}}$, from the i-th reconfigurable element to eavesdropper $\mathrm{E_{2}}$, and from the t-th transmit antenna at the BS to eavesdropper $\mathrm{E_{2}}$. The fading gains $h_{rE_{\nu }}^{i}$ and $h_{sE_{2}}^{t}$ are complex Gaussian distributed with zero mean and unit variance, i.e., $h_{rE_{\nu }}^{i}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )$, and $h_{sE_{2}}^{t}\sim {{\mathcal {C}}}{{\mathcal {N}}}(0,1 )$.

Based on the SINR (or SNR) derived above, the capacity of the in node $\Pi _2$ to decode signal $s_{\nu }$ is expressed as

$$\begin{aligned} C_{{\Pi _2}, s_\nu }= \log _2 \left( {{1+\gamma _{{\Pi _2}, s_\nu }}}\right) . \end{aligned}$$

(12)

Furthermore, the secrecy capacities obtained at user $\textrm{U}_\nu$ are, respectively, given by

$$\begin{aligned} C_{S, U_\nu ,s_{\nu } } = {\left[ { {C_{U_\nu , s_{\nu }}-C_{E_{\nu } , s_\nu }}}\right] ^{+}}, \end{aligned}$$

(13)

and

$$\begin{aligned} C_{S, U_2, s_1 } = {\left[ { {C_{U_2, s_1}-C_{E_2 , s_1}}}\right] ^{+}}, \end{aligned}$$

(14)

where $[x]^+=\max \left\{ {x,0}\right\}$.

2.3 STAR-RIS-OMA

In this subsection, the STAR-RIS-aided OMA scheme is selected as a baseline for comparison, in which the STAR-RIS is deployed to assist the BS to send the information to both user $\textrm{U}_1$ and user $\textrm{U}_2$. Thus, the detecting SNR of node $\Pi _2\cup \Pi _3$ for STAR-RIS-OMA scheme can be, respectively, given by

$$\begin{aligned} \gamma _{{\Pi _4}, s_{1}}^{OMA}= \rho _{\Pi _4} | \omega _1 \mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(15)

and

$$\begin{aligned} \gamma _{{\Pi _5}, s_{2}}^{OMA}= \rho _{\Pi _5} |\omega _{3} {\textbf{w}}+\omega _{2}\mathbf {\Phi }{\textbf{G}}{\textbf{w}} |^2, \end{aligned}$$

(16)

where $\Pi _4 \in \{\mathrm{U_1}, \mathrm{E_1} \}$, $\Pi _5 \in \{\mathrm{U_2}, \mathrm{E_2} \}$, and $\omega \in \{{\textbf{g}},{\textbf{p}} \}$.

The entire wireless communication process includes two time slots for STAR-RIS-OMA networks. More specifically, in the first time slot, the BS sends the information $s_1$ through STAR-RIS to refract to user $\textrm{U}_1$, and the BS sends $s_2$ to reflect to user $\textrm{U}_2$ by the aid of STAR-RIS in the second slot. As a result, the capacity of the node $\Pi _3$ to decode signal $s_{\nu }$ can be expressed as

$$\begin{aligned} C_{{\Pi _3}, s_\nu }^{OMA}= \frac{1}{2} \log _2 \left( {{1+\gamma _{{\Pi _3}, s_\nu }^{OMA}}}\right) . \end{aligned}$$

(17)

Thus, the secrecy capacity obtained at user $\textrm{U}_\nu$ is given by

$$\begin{aligned} C_{S, U_\nu , s_\nu }^{OMA} = {\left[ { {C_{U_\nu , s_\nu }^{OMA}-C_{E_{\nu }, s_\nu }^{OMA}}}\right] ^{+}}. \end{aligned}$$

(18)

2.4 Channel statistical properties

In this subsection, the channel statistical properties of cascaded Rayleigh channels are provided, which will be employed to evaluate secrecy outage behaviors for STAR-RIS-aided NOMA systems in the following sections.

In view of the above discussions, $h_{sr}^{it}$ and $h_{rU_{\nu }}^{i}$ are the channel coefficients from the BS to STAR-RIS and from STAR-RIS to user $\mathrm{U_{\nu }}$, respectively. Unfortunately, the exact distribution of cascaded Rayleigh channels $X_{\nu ,t}=\sum _{t=1}^{ T}\sum _{j=1}^{ N}|h_{rU_{\nu }}^{i}||{{\textbf{h}}}_{sr}^{it}{\textbf{w}}|$ is difficult to derive, we propose to approximate the PDF of $X_{\nu ,t}$ as a Gamma distribution by exploiting the Laguerre series approximation [34], which can be given by

$$\begin{aligned} f_{X}(x)= \frac{x^{a}}{b^{a+1}\Gamma (a+1)} \exp \left( -\frac{x}{b} \right) , \end{aligned}$$

(19)

where $\Gamma (\cdot )$ denotes the Gamma function [35], $a=\frac{c^2}{d}-1$, $b =\frac{d}{c}$ with $c=\frac{ N\pi }{2}$, $d=4N\left( 1-\frac{\pi ^2}{16} \right)$.

Moreover, the exact distribution of $Y=|{\textbf{h}}_{3}{\textbf{w}}|^2=||{\textbf{h}}_{3}||^2$ follows the Gamma distribution with shape parameter T and scale parameter 1. Thus, the PDF of Y can be written as

$$\begin{aligned} f_{Y}(y)= \frac{y^{T-1}}{\Gamma (T)} \exp \left( -y \right) . \end{aligned}$$

(20)

Additionally, we assume that the distance of each row on STAR-RIS is less than half wavelength, and the distance of each column on STAR-RIS is larger than half wavelength. Thus, the received signals between rows are correlated, whereas the received signals between any two columns are independent. As a result, the correlation matrix of the n-th column on STAR-RIS is defined as⁶

$$\begin{aligned} {\Upsilon _{n}}=\left( {\begin{array}{cccc} 1 &{} {{\Lambda ^{n} _{12}}} &{} \cdots &{} {{\Lambda ^{n} _{1M}}} \\ {{\Lambda ^{n} _{12}}} &{} 1 &{} \cdots &{} {{\Lambda ^{n} _{2M}}} \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ {{\Lambda ^{n} _{1M}}} &{} {{\Lambda ^{n} _{2M}}} &{} \cdots &{} 1 \end{array}}\right) , \end{aligned}$$

(21)

where $\Upsilon \in \{ W, V, Q, L, G \}$, $\Lambda \in \{ \eta , \xi , \zeta , \phi , \varsigma \}$, $W_n$ represent the receiving correlation matrix of the n-th column on the STAR-RIS, $V_n$ and $Q_n$ represent the refracting correlation matrix of the n-th column on the STAR-RIS for $\textrm{U}_1$ and $\textrm{E}_1$, and $L_n$ and $G_n$ represent the reflecting correlation matrix of the n-th column on the STAR-RIS for $\textrm{U}_2$ and $\textrm{E}_2$, respectively. In addition, $\Lambda ^{n} _{m,m^{'}}$ $(1\le m \le M, 1\le m^{'} \le M, m\ne m^{'} )$ denotes the correlation coefficient between the m-th and the $m^{'}$-th element of the n-th column on the STAR-RIS.

Referring to (5), the SINR of user $\textrm{U}_{1}$ to decode $s_1$ can be formulated as

$$\begin{aligned} \gamma _{U_{1}, s_1} =\frac{\alpha _o \rho _{U_{1}} A_1\left( X_{U_1,t}e^{j\theta _{j}^{\chi _{1}}}\right) ^{2}}{(1-\alpha _o) \rho _{U_{1}}A_1\left( X_{U_1,t}e^{j\theta _{j}^{\chi _{1}}}\right) ^{2}+1}, \end{aligned}$$

(22)

where $A_1=\frac{ \beta _{j}^{\chi _{1}}}{({d_{SR}d_{RU_{1}}})^{\eta _{0}}}\left( 1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\xi _{mm^{'}}^{n}\right)$, $X_{U_1,t}=\sum _{t=1}^{T}\sum _{j=1}^{N} h_{rU_{1}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}}$, and $\beta _{j}^{\chi _{1}}$ is the transmitting coefficient.

In order to obtain better system security performance, we propose a new scheme, that is, each of the reconfigurable elements serves user $\textrm{U}_1$ and user $\textrm{U}_2$ at the same time. Specifically, the reflective links serve user $\textrm{U}_2$ and the refractive links serve user $\textrm{U}_1$. It is well known that, the STAR-RIS can adjust the phase shift of each reconfigurable element according to the legitimate user’s channel information, so that the sum of signal amplitudes obtained by the legitimate user can be maximized. Unfortunately, since it is hard to obtain the eavesdropping CSI, we only need to maximize the SINR or SNR received by the legitimate users. For the cell-edge user, the data transmission is through the STAR-RIS link only. According to the SINR expression in Eq. (5), we can get that if the cascaded channel gain is the largest (i.e., $\left| {\textbf{g}}_{\textbf{1}}\mathbf {\Phi Gw}\right|$ is the maximum), the SINR is the maximum for the cell-edge user. By applying some mathematical operations, the sub-optimal phase shifts for maximizing the SINR can be obtained by $\theta _{j,opt.}^{\chi _{1}}=-arg[[h_{rU_{1}}^{j}][{\textbf{h}}_{sr}^{jt}{\textbf{w}}]]$. It is shown that the design of phase shifts depends on the sub-optimal beamforming vector. Thus, the phase shifts obtained here is also suboptimal.

Proposition 1

The CDF of $\gamma _{{U_{1}}, s_1}$ in (22) can be written as

$$\begin{aligned} F_{\gamma _{U_{1}, s_1}}(z) =\frac{\gamma {\left( a+1,\frac{1}{b}\sqrt{\frac{z}{B_1(\alpha _o-(1-\alpha _o)z)}} \right) }}{\Gamma (a +1)}, \end{aligned}$$

(23)

where $B_{1}= \alpha _o \rho _{U_{1}} A_1$, and $\gamma (x,y)$ denotes the lower incomplete Gamma function [35].

Proof

Please refer to “Appendix 1”.

For the cell-center user, there exist a direct link and a cascaded channel. According to the SINR expressions in Eqs. (6)–(7), it is obvious that when the overall channel gain (the direct link and the cascaded channel) is the largest, the SINR/SNR achieves the largest. In such case, it requires that $\left| {\textbf{g}}_{\textbf{3}}{\textbf{w}}+{\textbf{g}}_{\textbf{2}}\mathbf {\Phi Gw}\right|$ is the maximum. Thus, the sub-optimal phase shifts for the cell-center user are denoted by $\theta _{j,opt.}^{\chi _{2}}=arg[h_{sU_{2}}^{t}]-arg[[h_{rU_{2}}^{j}][{\textbf{h}}_{sr}^{j}{\textbf{w}}]]$. Let $X_{2,t}=\sum _{t=1}^{T}\sum _{j=1}^{N}|h_{rU_{2}}^{j}||{\textbf{h}}_{sr}^{jt}{\textbf{w}}|$, $Y_1=||{\textbf{h}}_{3}||$.

Based on NOMA, the SINR and SNR at cell-center user $\textrm{U}_2$ can be formulated as

$$\begin{aligned} \gamma _{{U_{2}}, s_{1}}= \frac{\alpha _o \rho _{U_{2}}\left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2 }{(1-\alpha _o)\rho _{U_{2}} \left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2+1}, \end{aligned}$$

(24)

and

$$\begin{aligned} \gamma _{{U_{2}}, s_{2}}= (1-\alpha _o)\rho _{U_{2}} \left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2, \end{aligned}$$

(25)

where $A_2=\frac{ \beta _{j}^{\chi _{2}}}{({d_{SR}d_{RU_{2}}})^{\eta _{0}}}(1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\phi _{mm^{'}}^{n})$, $C_1= d_{SU_2}^{-\eta _{0}}$.

In order to obtain the CDF of $\gamma _{{U_{2}}, s_{\nu }}$, we first derive a close-form expression of $Z_{1}^2$, with $Z_{1}^{2}=\left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} \right) ^2$, in the following proposition.

Proposition 2

The CDF of $Z_1^2$ can be written as

$$\begin{aligned} F_{Z_1^2}(z) = \frac{{\textbf{B}}(2T+1,a+1) z^{T+\frac{a+1}{2}} }{T\Gamma (T)C_1^T\Gamma (a +1)b^{a+1}A_2^{\frac{a+1}{2}}} {}_1 \!F_1 \left( {a+1; } { 2T+a+2}; {- \frac{1}{b} \sqrt{\frac{z}{A_2}} } \right) , \end{aligned}$$

(26)

where ${\textbf{B}}(x,y)$ is Beta function and ${}_1 \!F_1$ is degenerate hypergeometric function [35].

Proof

Please refer to “Appendix 2”.

Based on (26), the CDF of $\gamma _{U_{2}, s_1}$ is given by

$$\begin{aligned} F_{\gamma _{U_{2}, s_1}}\left( x\right) =F_{Z_{1}^2}\left( \frac{x}{\rho _{u_2} (\alpha _o-(1-\alpha _o)x)}\right) . \end{aligned}$$

(27)

In order to avoid redundancy, the detailed derivations are no longer presented.

Referring to the solution process of $F_{\gamma _{U_{2}, s_{1}}}(x)$, the CDF of $\gamma _{{U_2}, s_{2}}$ can be written as

$$\begin{aligned} F_{\gamma _{U_{2}, s_{2}}}(x) =F_{Z_1^2}\left( \frac{x}{\rho _{u_2}(1-\alpha _o)x} \right) . \end{aligned}$$

(28)

Furthermore, referring to (10), we rewrite $\gamma _{{E_{1}}, s_{1}}$ as

$$\begin{aligned} \gamma _{{E_{1}}, s_{1}}= \alpha _{o}\rho _{E_{1}}A_3 Z_{1,t}^2 , \end{aligned}$$

(29)

where $A_3=\frac{ \beta _{j}^{\chi _{1}} }{({d_{SR}d_{RE_{1}}})^{\eta _{0} }}\left( 1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\zeta _{mm^{'}}^{n}\right)$, $Z_{1,t}= \sum _{t=1}^{T}\sum _{j=1}^{N} h_{rE_{1}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}} e^{j\theta _{j}^{\chi _{1}}}$.

According to the central limit theorem (CLT), $Z_{1,t}$ can be approximated with a complex Gaussian-distributed random variable. As a result, the square of $Z_{1,t}^2$ can be approximated with an exponential random variable with parameter N, respectively.

Based on the above operations, the CDF of $\gamma _{{E_{1}}, s_{1}}$ can be expressed as

$$\begin{aligned} F_{\gamma _{E_{1}, s_{1}}}(z) =1-e^{-\frac{z}{\lambda _{E_{1}}}}, \end{aligned}$$

(30)

where $\lambda _{E_{1}}= \alpha _{o}N\rho _{E_{1}}A_3$.

Referring to (11), we rewrite $\gamma _{{E_{2}}, s_{2}}$ as

$$\begin{aligned} \gamma _{{E_{2}}, s_{2}}= (1-\alpha _{o})\rho _{E_{2}}\left( \sqrt{A_4} Z_{2,t}+\sqrt{C_2}Y_2\right) ^2 , \end{aligned}$$

(31)

where $A_4=\frac{ \beta _{j}^{\chi _{2}}}{d_{SR}^{\eta _{o}}d_{RE_2}^{\eta _{1} }}\left( 1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\varsigma _{mm^{'}}^{n}\right)$, $C_2=d_{SE_{2}}^{-\eta _{1}}$, $Y_{2}= {\textbf{e}}_3 {\textbf{w}} e^{j\theta _{s}}$, $Z_{2,t}=\sum _{t=1}^{T} \sum _{j=1}^{N} h_{rE_{2}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}} e^{j\theta _{j}^{\chi _{2}}}$.

According to the CLT, $\left( \sqrt{A_4} Z_{2,t}+\sqrt{C_2}Y_2\right)$ can be approximated with a complex Gaussian-distributed random variable. As a result, the $\left( \sqrt{A_4} Z_{2,t}+\sqrt{C_2}Y_2\right) ^2$ can be approximated with an exponential random variable with parameter $NA_4+C_2$, respectively.

After the above operations, the CDF of $\gamma _{{E_{2}}, s_{2}}$ can be expressed as

$$\begin{aligned} F_{\gamma _{E_{2}, s_{2}}}(z) =1-e^{-\frac{z}{\lambda _{E_{2}}}}, \end{aligned}$$

(32)

where $\lambda _{E_{2}}= (1-\alpha _{o})\rho _{E_{2}}\left( NA_4+C_2\right)$.

3 Secrecy outage probability analysis

In this section, the performance of STAR-RIS-assisted NOMA networks is investigated in terms of secrecy outage probability, where the exact expressions of SOP for both user $\textrm{U}_1$ and user $\textrm{U}_2$ are derived.

3.1 The SOP of the cell-edge user

According to the NOMA protocol, if the cell-edge user $\textrm{U}_1$ cannot detect safely the refracting signal $s_1$, the secrecy outage event will happen. The corresponding SOP can be written as

$$\begin{aligned} \begin{aligned} P_{U_1}= \Pr \left\{ C_{S,{U_1}, s_1}< R_{1}^s \right\} =\Pr \left\{ \frac{1+\gamma _{{U_1}, s_1}}{1+\gamma _{{E_1}, s_1}}< 2^{R_{1}^s} \right\} . \end{aligned} \end{aligned}$$

(33)

According to (23) and (33), we present a closed-form expression of user $\textrm{U}_1$ in the following lemma.

Lemma 1

The exact SOP expression of user $\textrm{U}_1$ in our proposed STAR-RIS-aided NOMA scheme is given by

$$\begin{aligned} \begin{aligned} P_{U_1}=&\frac{\pi (\gamma _{s,1}-1) }{\lambda _{E_1} N_0 \gamma _{s,1}} \exp \left( \frac{(\gamma _{s,1}-1)}{\lambda _{E_1}\gamma _{s,1} } \right) \sum _{l_0=1}^{N_0}\frac{\sqrt{1-I_1^2}}{(1+I_1)^2} \\&\times \exp \left( -\frac{2(\gamma _{s,1}-1)}{\lambda _{E_1}\gamma _{s,1}(I_1+1) } \right) F_{\gamma _{U_{1}, s_{1}}}\left( \frac{2(\gamma _{s,1}-1)}{I_1+1} \right) , \end{aligned} \end{aligned}$$

(34)

where $\gamma _{s,1}=2^{R_1^s}$, $I_1=\cos \left( \frac{2l_{0}-1}{2N_0}\pi \right)$, and $N_0$ denotes the number of terms for the Gauss–Chebyshev quadrature approximation.

Proof

Please refer to “Appendix 3”.

3.2 The SOP of the cell-center user

The SOP of user $\textrm{U}_2$ to detect $s_2$ is given by

$$\begin{aligned} P_{U_2}= {{\Pr \left( {{{C_{S,{U_{2}, s_1}}<R_{1}^{s} }}}\right) }} + {{\Pr \left( C_{S,U_{2}, s_2}<R_{2}^{s},{{{C_{S,{U_{2}, s_1}}\ge R_{1}^{s} }}} \right) }}. \end{aligned}$$

(35)

Since $C_{S,{U_{2}, s_1}} \ge R_{1}^{s}$ and $C_{S,U_{2}, s_2}<R_{2}^{s}$ are independent of each other, referring to [36], we can rewrite (35) as

$$\begin{aligned} P_{U_2}=1-{\underbrace{\Pr \left( C_{S,U_{2}, s_2}\ge R_{2}^{s},{{{C_{S,{U_{2}, s_1}}\ge R_{1}^{s} }}} \right) }_{\Psi _{1}}}. \end{aligned}$$

(36)

Referring to (26) and “Appendix 3”, we can easily obtain the approximate closed-form expression of $\Psi _1$ in the following lemma.

Lemma 2

The approximate closed-form expression of $\Psi _1$, which can be written as

$$\begin{aligned} \begin{aligned} {\Psi _{1}}=&1-\frac{\pi (\gamma _{s,2}-1) }{\lambda _{E_2} N_0 \gamma _{s,2}} \exp \left( \frac{(\gamma _{s,2}-1)}{\lambda _{E_2}\gamma _{s,2} } \right) \sum _{l_0=1}^{N_0}\frac{\sqrt{1-I_1^2}}{(1+I_1)^2} \\&\times \exp \left( -\frac{2(\gamma _{s,2}-1)}{\lambda _{E_2}\gamma _{s,2}(I_1+1) } \right) F_{\gamma _{U_{2}, s_{2}}}\left( \frac{2(\gamma _{s,2}-1)}{I_1+1} \right) , \end{aligned} \end{aligned}$$

(37)

where $\gamma _{s,2}=2^{R_2^s}$.

Then, by substituting (37) into (36), we obtain the approximate SOP of user $\textrm{U}_2$. Thus, the total approximate SOP can be expressed as

$$\begin{aligned} P_{out}= \frac{(P_{U_1}+P_{U_2})}{2}. \end{aligned}$$

(38)

3.3 The SOP of STAR-RIS-OMA

For STAR-RIS-OMA systems, the SOP of user $\textrm{U}_{\nu }$ can be expressed as

$$\begin{aligned} \begin{aligned} P_{U_{\nu }}^{OMA}= \Pr \left\{ C_{S,{U_{\nu }}, s_{\nu }}^{OMA}< R_{\nu }^s \right\} = \Pr \left\{ \frac{1+\gamma _{{U_{\nu }}, s_{\nu }}^{OMA}}{1+\gamma _{{E_{\nu }}, s_{\nu }}^{OMA}}< 2^{2R_{{\nu }}^s} \right\} . \end{aligned} \end{aligned}$$

(39)

Referring to (39), we can derive the approximate SOP of user $\textrm{U}_{\nu }$ for OMA systems in the following lemma.

Lemma 3

The approximate closed-form expression of $P_{U_{\nu }}^{OMA}$, which can be expressed as

$$\begin{aligned} \begin{aligned} P_{U_{\nu }}^{OMA}=&\frac{\pi (\gamma _{s,{\nu }}^{OMA}-1) }{\lambda _{E_{\nu +1}} N_0\gamma _{s,{\nu }}^{OMA}} \exp \left( \frac{(\gamma _{s,{\nu }}^{OMA}-1)}{\lambda _{E_{\nu +1}}\gamma _{s,{\nu }}^{OMA} } \right) \sum _{l_0=1}^{N_0}\frac{\sqrt{1-I_1^2}}{(1+I_1)^2} \\&\times \,F_{\gamma _{U_{\nu }}, s_{\nu }}^{OMA}\left( \frac{2(\gamma _{s,{\nu +1}}^{OMA}-1)}{I_1+1} \right) \exp \left( -\frac{2(\gamma _{s,{\nu }}^{OMA}-1)}{\lambda _{E_{\nu +1}}\gamma _{s,{\nu }}^{OMA}(I_1+1) } \right) , \end{aligned} \end{aligned}$$

(40)

where $\gamma _{s,1}^{OMA}=2^{2R_{1}^s}$, $\gamma _{s,2}^{OMA}=2^{2R_{2}^s}$, $\lambda _{E_3}=\rho _{E_1} A_3N$, and $\lambda _{E_4}=\rho _{E_2}A_4$.

Thus, the total approximate SOP for STAR-RIS-aided OMA can be expressed as

$$\begin{aligned} P_{out}^{OMA}= \frac{(P_{U_1}^{OMA}+P_{U_2}^{OMA})}{2}. \end{aligned}$$

(41)

From Lemmas 1, 2 and 3, we can observe that the closed expression of exact SOP has a high computation complexity, which is difficult to draw valuable insights. Motivated by this, in the following section, we will focus on the asymptotic SOP achieved by our proposed multi-antenna STAR-RIS-aided NOMA scheme. Based on the asymptotic SOP, we will provide an explicit evaluation on the secrecy performance of the proposed scheme.

4 The asymptotic secrecy outage probability analysis

In order to provide pivotal insights into the secrecy performance achieved by our proposed STAR-RIS-aided NOMA scheme, in this section, we will derive a closed-form expression for the asymptotic SOP in the high SNR regime. When $\rho _{s}\rightarrow \infty$ and $(1-\alpha _o)\rho _{U_{1}} A_1\left( \sum _{j=1}^{N} h_{rU_{1}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}} \right) ^2\ll 1$, we can obtain $\gamma _{U_{1}\leftarrow s_{1}}\approx \alpha _{o}\rho _{U_{1}} A_1\left( \sum _{j=1}^{N} h_{rU_{1}}^{j}{\textbf{h}}_{sr}^{jt}{\textbf{w}} \right) ^2$.

4.1 The asymptotic SOP of cell-edge user

Referring to (34), we first derive a closed-form expression to approximate $P_{U_1}$ in the following theorem.

Theorem 1

When $\rho _{s}\rightarrow \infty$, the approximate SOP of user $\textrm{U}_1$ to detect $s_1$ is given by

$$\begin{aligned} P_{U_1}^{\infty }\approx \frac{({\lambda _{E_1}\gamma _{s,1}})^{\frac{a + 1}{2}}e^{\frac{(\gamma _{s,{1}}-1)}{\lambda _{E_1} \gamma _{s,{1}} } }\Gamma \left( \frac{a+1}{2}+1, \frac{(\gamma _{s,{1}}-1)}{\lambda _{E_1} \gamma _{s,{1}} } \right) }{\Gamma (a+1)b^{a+1}(a +1)(\alpha _{o}N\rho _{U_{1}}A_1)^{\frac{a+1}{2}}}. \end{aligned}$$

(42)

4.2 The asymptotic SOP of cell-center user

Recalling to (34) and (42), we present a closed-form expression to approximate $P_{U_2}$ in the following theorem.

Theorem 2

The approximate SOP of user $\textrm{U}_2$ with closed-form expression is expressed as

$$\begin{aligned} \begin{aligned} P_{U_2}^{\infty }\approx&\frac{{\textbf{B}}\left( \frac{a+1}{2}+1,T \right) {\varphi _1}^{\frac{a + 1}{2}+T} \exp \left( \frac{(\gamma _{s,2}-1)}{\lambda _{E_2}\gamma _{s,2}} \right) }{\Gamma (T)\Gamma (a+1)b^{a+1}(a +1)C_{1}^{T}A_{2}^{\frac{a+1}{2}}} \Gamma \left( \frac{a+1}{2}+1+T, \frac{(\gamma _{s,2}-1)}{ \lambda _{E_2}\gamma _{s,2}}\right) , \end{aligned} \end{aligned}$$

(43)

where $\varphi _1=\frac{\rho _{E_{2}}\gamma _{s,2}(NA_4+C_2)}{\rho _{U_{2}}}$.

As a result, the total approximate SOP can be written as

$$\begin{aligned} P_{out}^{\infty }= \frac{P_{U_1}^{\infty }+P_{U_2}^{\infty }}{2}. \end{aligned}$$

(44)

Based on Theorems 1 and 2, it indicates that the asymptotic SOP of the multi-antenna STAR-RIS-aided NOMA scheme approaches a certain positive constant.

4.3 The asymptotic SOP of STAR-RIS-OMA

Referring to (40), we derive the SOP of the multi-antenna STAR-RIS-assisted NOMA scheme in the following theorem.

Theorem 3

The closed-form expression of SOP for user $\textrm{U}_{\nu }$, which is approximated as

$$\begin{aligned} \begin{aligned} P_{U_1}^{OMA,\infty }\approx&\frac{({\lambda _{E_3}\gamma _{s,1}^{OMA}})^{\frac{a + 1}{2}} \exp \left( \frac{(\gamma _{s,1}^{OMA}-1)}{\lambda _{E_3}\gamma _{s,1}^{OMA}} \right) }{\Gamma (a+1)b^{a+1}(a +1)(N\rho _{U_{1}} \gamma _{sru_{1}})^{\frac{a+1}{2}}} \Gamma \left( \frac{a+1}{2}+1, \frac{(\gamma _{s,1}^{OMA}-1)}{\lambda _{E_3}\gamma _{s,1}^{OMA}}\right) , \end{aligned} \end{aligned}$$

(45)

and

$$\begin{aligned} \begin{aligned} P_{U_2}^{OMA,\infty }\approx&\frac{{\textbf{B}}\left( \frac{a+1}{2}+1,T \right) {\varphi _2}^{\frac{a + 1}{2}+T} \exp \left( \frac{(\gamma _{s,2}^{OMA}-1)}{\lambda _{E_4}\gamma _{s,2}^{OMA}} \right) }{\Gamma (T)\Gamma (a+1)b^{a+1}(a +1)C_{1}^{T}A_{2}^{\frac{a+1}{2}}} \Gamma \left( \frac{a+1}{2}+T+1, \frac{(\gamma _{s,2}^{OMA}-1)}{\lambda _{E_4}\gamma _{s,2}^{OMA}}\right) , \end{aligned} \end{aligned}$$

(46)

where $\varphi _2=\frac{\rho _{E_{2}}\gamma _{s,2}^{OMA}(NA_4+C_2)}{\rho _{U_{2}}}$.

As a result, the total approximate SOP for STAR-RIS-aided OMA network can be written as

$$\begin{aligned} P_{out}^{OMA,\infty }= \frac{P_{U_1}^{OMA,\infty }+P_{U_2}^{OMA,\infty }}{2}. \end{aligned}$$

(47)

Based on Theorem 3, we also clarify that the asymptotic SOP of the multi-antenna STAR-RIS-aided OMA scheme approaches a certain positive value.

Comparing our proposed multi-antenna STAR-RIS-aided NOMA scheme with OMA scheme, we can obtain the following two remarks from the above analyses.

Remark 1

When $\rho _s \rightarrow \infty$, the average asymptotic SOP for our proposed multi-antenna STAR-RIS-aided NOMA system approaches a floor. This observation is also applicable to multi-antenna STAR-RIS-aided OMA system. It means that when the transmission power is large, blindly increasing the transmission power cannot improve the system security performance.

Remark 2

For multi-antenna STAR-RIS-aided NOMA system, the entire communication procedure occupies a total time slot. Thus, a factor of 1 is used in computing the channel capacities for the multi-antenna STAR-RIS-aided NOMA scheme. However, for multi-antenna STAR-RIS-aided OMA system, the entire communication procedure is completed by using two orthogonal time slots. A factor of $\frac{1}{2}$ is used for the multi-antenna STAR-RIS-aided OMA scheme. Besides, the value of Eq.(47) is always greater than that of Eq.(44) when the transmit power is in the high regime. It indicates that the advantage of the multi-antenna STAR-RIS-aided NOMA scheme over the multi-antenna STAR-RIS-aided OMA scheme.

5 Secrecy diversity order analysis

In order to gain more deep insights, the SDO is usually selected to evaluate the secrecy outage behaviors for communication networks, which is able to describe how fast the SOP decreases when both the transmitting SNR and main-to-eavesdropper ratio (MER) are sufficiently high [37,38,39]. Moreover, let $\lambda _{m,e}=\frac{\Omega _{main}}{\Omega _{eave}}$ denote the ratio of average channel gain between the source and the destination to that between the source and the eavesdropper, which is referred to the MER. Hence, the SDO can be defined as

$$\begin{aligned} d_{s}=-\mathop {\textrm{lim}}_{\begin{array}{c} \rho _{s}\rightarrow \infty \\ {\lambda _{m,e}}\rightarrow \infty \end{array}} \frac{log{P_{out}^{\infty }}}{log{\lambda _{m,e}}} , \end{aligned}$$

(48)

where $P_{out}^{\infty }$ denotes the total asymptotic SOP.

Furthermore, we define $\Omega _{RU_1}=\lambda _{m,e} \Omega _{RE_1}$, $\Omega _{SR}=\lambda _{1} \lambda _{m,e} \Omega _{RE_1}$, $\Omega _{RU_{2}}=\lambda _{2} \lambda _{m,e} \Omega _{RE_1}$, $\Omega _{SU_{2}}=\lambda _{3} \lambda _{m,e} \Omega _{RE_1}$, $\Omega _{RE_2}=\lambda _{4} \Omega _{RE_1}$, and $\Omega _{SE_2}=\lambda _{5} \Omega _{RE_1}$, where $\lambda _{1}$, $\lambda _{2}$, $\lambda _{3}$, $\lambda _{4}$, and $\lambda _{5}$ are positive constants, $\Omega _{RE_1}$, $\Omega _{RE_2}$, $\Omega _{RU_1}$, $\Omega _{RU_2}$, $\Omega _{SR}$, $\Omega _{SU_2}$, and $\Omega _{SE_2}$ stand for the channel gain of the link from the STAR-RIS to the eavesdropper $\textrm{E}_1$, the channel gain of the link from the STAR-RIS to the eavesdropper $\textrm{E}_2$, the channel gain of the link from the STAR-RIS to the user $\textrm{U}_1$, the channel gain of the link from the STAR-RIS to the user $\textrm{U}_2$, the channel gain of the link from the BS to the STAR-RIS, the channel gain of the link from the BS to the user $\textrm{U}_2$, and the channel gain of the link from the BS to the eavesdropper $\textrm{E}_2$, respectively.

5.1 The SDO of cell-edge user

Recalling (42), as $\rho _s \rightarrow \infty$, we derive the approximate SOP of user $\textrm{U}_1$ in the following corollary.

Corollary 1

The approximate SOP of cell-edge user $\textrm{U}_1$ can be given by

$$\begin{aligned} P_{U_1}^{\infty }\approx \frac{\Gamma \left( \frac{a + 1}{2}+1\right) \left( \frac{\rho _{E_1} \gamma _{s,1}D_2 }{\rho _{U_1}D_1\lambda _{m,e}}\right) ^{\frac{a + 1}{2}}}{\Gamma (a+1)b^{a+1}(a+1)} , \end{aligned}$$

(49)

where $D_1=1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\xi _{mm^{'}}^{n}$, $D_2=1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\zeta _{mm^{'}}^{n}$.

By substituting (49) into (48), we obtain the SDO of user $\textrm{U}_1$ as $\frac{a + 1}{2}$.

5.2 The SDO of cell-center user

Recalling (43), as $\rho _s \rightarrow \infty$, we present the approximate SOP of user $\textrm{U}_2$ in the following corollary.

Corollary 2

The approximate SOP of cell-center user $\textrm{U}_2$ can be given by

$$\begin{aligned} \begin{aligned} P_{U_2}^{\infty }\approx&\frac{{\textbf{B}}\left( \frac{a+1}{2}+1,T \right) \Gamma \left( \frac{a+1}{2}+1+T\right) }{\Gamma (T)\Gamma (a+1)b^{a+1}(a +1) } \left( \frac{\rho _{E_2}\gamma _{s,2}}{\rho _{U_2}} \cdot \frac{\varphi _3 D_4 \lambda _{m,e}+\varphi _6}{\varphi _5 D_3 \lambda _{m,e}^2} \right) ^{\frac{a+1}{2}} \\&\times \left( \frac{\rho _{E_2}\gamma _{s,2}}{\rho _{U_2}} \cdot \frac{\varphi _3 D_4 \lambda _{m,e}+\varphi _6}{\varphi _7 \lambda _{m,e}} \right) ^{T}, \end{aligned} \end{aligned}$$

(50)

where $\varphi _3=N\beta _{i}^{\chi _{2}}\varphi _4$, $\varphi _4=\lambda _1\lambda _4\Omega _{RE_1}$, $\varphi _5=\beta _{i}^{\chi _{2}}\lambda _1\lambda _2 \Omega _{RE_1}$, $\varphi _6=\lambda _5$, $\varphi _7=\lambda _3$, $D_3=1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\phi _{mm^{'}}^{n}$, $D_4=1+\sum _{m=1}^{M}\eta _{mm^{'}}^{n}\varsigma _{mm^{'}}^{n}$.

By substituting (50) into (48), we obtain the SDO of user $\textrm{U}_2$ as $\frac{a + 1}{2}$.

Based on Corollaries 1 and 2, it is obvious that the SDO of the multi-antenna STAR-RIS-aided NOMA scheme is a constant, i.e., $\frac{a+1}{2}=\frac{N}{2}\frac{\pi ^2}{\left( 16-\pi ^2 \right) }$, which depends on the number of columns on the STAR-RIS.

5.3 The SDO of STAR-RIS-OMA

Recalling (45) and (46), as $\rho _s \rightarrow \infty$, we also present the approximate SOP of user $\textrm{U}_2$ in the following corollary.

Corollary 3

The approximate SOP of user $\textrm{U}_{\nu }$ for STAR-RIS-OMA scheme can be expressed as

$$\begin{aligned} P_{U_1}^{OMA,\infty }\approx \frac{\Gamma \left( \frac{a + 1}{2}+1\right) \left( \frac{\rho _{E_1}D_2 \gamma _{s,1}^{OMA} }{\rho _{U_1}D_1\lambda _{m,e}}\right) ^{\frac{a + 1}{2}}}{\Gamma (a+1)b^{a+1}(a+1)} , \end{aligned}$$

(51)

and

$$\begin{aligned} \begin{aligned} P_{U_2}^{OMA,\infty }\approx&\frac{{\textbf{B}}\left( \frac{a+1}{2}+1,T \right) \Gamma \left( \frac{a+1}{2}+T+1\right) }{\Gamma (T)\Gamma (a+1)b^{a+1}(a +1) } \left( \frac{N\varphi _4 D_4 \lambda _{m,e}+\varphi _6}{\varphi _7\lambda _{m,e}}\cdot \frac{\rho _{E_2}\gamma _{s,2}^{OMA}}{\rho _{U_2}} \right) ^{T} \\&\times \left( \frac{N\varphi _4 D_4 \lambda _{m,e}+\varphi _6}{\lambda _1\lambda _2 \Omega _{RE_1} D_3 \lambda _{m,e}^2} \cdot \frac{\rho _{E_2}\gamma _{s,2}^{OMA}}{\rho _{U_2}} \right) ^{\frac{a+1}{2}} . \end{aligned} \end{aligned}$$

(52)

By substituting (51) and (52) into (48), we obtain the SDO of user $\textrm{U}_{\nu }$ as $\frac{a + 1}{2}$.

Based on Corollary 3, we also clarify that the SDO of the multi-antenna STAR-RIS-aided OMA scheme is a constant, i.e., $\frac{a+1}{2}=\frac{N}{2}\frac{\pi ^2}{\left( 16-\pi ^2 \right) }$, which is also dependent on the number of columns on STAR-RIS.

From Corollaries 1, 2 and 3, we have two remarks given as follows:

Remark 3

The SDO of our proposed system can be obtained, which is $\frac{N}{2} \frac{\pi ^2}{\left( 16-\pi ^2 \right) }$. This conclusion also applies to the STAR-RIS-aided OMA system. As expected, increasing the number of reconfigurable columns on the STAR-RIS will bring an increasing of the SDO and thus a decreasing of the SOP.

Remark 4

The SDO of the cell-edge user for STAR-RIS-aided NOMA and OMA systems is the same constant, namely $\frac{N}{2} \frac{\pi ^2}{\left( 16-\pi ^2 \right) }$. Interestingly, differing from the traditional SDO, the SDO of the cell-center user for STAR-RIS-aided NOMA and OMA systems is not impacted by the number of transmit antennas on the BS. This is because that the STAR-RIS-aided links of the eavesdropper $\textrm{E}_2$ (next to the cell-center user) include the links from the BS to the STAR-RIS by comparing with conventional cooperative communication links.

6 Parameter optimization

Unlike most articles only on performance analysis, this paper analyzes the secrecy performance and optimizes the parameters those affect system performance. As far as we are concerned, both the PA and ES coefficients affect the secrecy performance of the considered network. As a result, selecting appropriate parameters can improve effectively the possibility of secure communication.

The PA and ES coefficients can be jointly optimized. Then, the optimization problem can be formulated as⁷

$$\begin{aligned} \begin{aligned} \mathop {\textrm{max}}_{{\alpha _o,\beta ^{tra.}} }\qquad \ \ {}&\quad P_{out} \\ s.t.\quad \quad \ \ \ {}&0 \le \beta ^{tra.} \le 1, \\ \quad \quad \ \ \ {}&0.5< \alpha _o < 1, \end{aligned} \end{aligned}$$

(53)

where $\beta ^{tra.}$ represents the ES coefficient of each element and $\alpha _o$ represents PA coefficient, respectively. Due to the relationships between the cell-center and the cell-edge users for the PA and the ES coefficients, we only consider the PA and the ES coefficients of the cell-edge user as the optimization variables in the formulated optimization problem. After the optimal value of the cell-edge user is obtained, the optimal PA and ES coefficients of the cell-center user can be, respectively, obtained by $1- \alpha _o$ and $\beta ^{ref.}=1-\beta ^{tra.}$.

In this section, the optimization design of the two parameters is mainly carried out. It is clear that the optimal solution of the problem in (53) can be obtained by using two-dimension search or alternating optimization method with the one-dimension search. Due to higher computational complexity of two-dimension search, therefore, an alternating optimization method with the one-dimension search is invoked to find the optimal solution in this paper. The optimal parameters are obtained by alternating iteration method. More specifically, the detailed procedure is expressed as follows.

6.1 The PA coefficient optimization of STAR-RIS-NOMA

When the ES coefficient $\beta ^{tra.}$ is fixed, according to the expression of $P_{out}^{\infty }$, the derivative of SOP with respect to the PA coefficient $\alpha _o$ is obtained, and its first-order derivative is given as

$$\begin{aligned} \frac{d P_{out}^{\infty }}{d\alpha _o}= \frac{\Delta _1 \Delta _2}{(1-\alpha _o)^2}-\frac{\Delta _3 \Delta _4}{\alpha _{o}^2}, \end{aligned}$$

(54)

where

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta _1 = \frac{\gamma _{s,1}-1}{N\rho _{E_1}A_3\gamma _{s,1}}, \\ \Delta _3 = \frac{\gamma _{s,2}-1}{N\rho _{E_1}(NA_4+C_2)\gamma _{s,2}}, \\ \Delta _2 = \left( \frac{\rho _{E_1}A_3}{\rho _{U_1}A_1} \right) ^{\frac{a + 1}{2}} \frac{\Gamma \left( \frac{a+1}{2}+1\right) }{\Gamma \left( a+1\right) (a+1) b^{a+1}}, \\ \Delta _4 = \frac{{\textbf{B}}\left( \frac{a+1}{2}+1,T \right) {\varphi _1}^{\frac{a + 1}{2}+T} \Gamma \left( \frac{a+1}{2}+1+T\right) }{\Gamma (T)\Gamma (a+1)b^{a+1}(a +1)C_{1}^{T}A_{2}^{\frac{a+1}{2}}}. \end{array}\right. \end{aligned}$$

(55)

The optimal PA coefficient can be obtained by setting (54) to zero, which is represented by ${\alpha _{o}^{*}}=\frac{\sqrt{\Delta _1\Delta _2}}{\sqrt{\Delta _1\Delta _2}+\sqrt{\Delta _3\Delta _4}}$.

6.2 The ES coefficient optimization of STAR-RIS-NOMA

When the PA coefficient $\alpha _o$ is fixed, according to the expression of $P_{out}^{\infty }$, the derivative of SOP with respect to the ES coefficient $\beta ^{tra.}$ is obtained. However, it is difficult to determine the relationship between the derivative and zero, which cannot be determined as a unimodal function in the definition domain. In addition, the value of $\beta ^{tra.}$ is a consecutive positive number. To sum up, one-dimensional exhaustive search algorithm is adopted to find the optimal solution. The detailed steps of one-dimensional exhaustive search algorithm is omitted here. In such case, the two parameters $\alpha _o$, $\beta ^{tra.}$ of this system have been optimized.

Remark 5

It is clear that the PA and the ES coefficients will affect the system secrecy outage probabilities; thus selecting appropriate the PA and the ES coefficients is beneficial to the improvement of system security performance. It has certain guiding significance in practical engineering application.

7 Results and discussion

In this section, simulation results are presented to validate the theoretical expressions. Monte Carlo (MC) simulation parameters used in this section are presented in Table 1. For the other propagation distances between the STAR-RIS and the users, the behaviors of performance curves are similar. BPCU is the abbreviation for bit per channel use. All the noise powers are assumed to the same, which is $\sigma _{o}^{2}$.

Table 1 Simulation parameters

We also assume that the received correlation matrix of the n-th column $W_{n}$ is equal,⁸ and it is given by

$$\begin{aligned} W=W_n=\left( \begin{array}{ccc} 1 &{} 0.75 &{} -0.8\\ 0.75 &{} 1 &{} -0.7\\ -0.8 &{} -0.7 &{} 1\\ \end{array} \right) , \end{aligned}$$

(56)

where the coefficients $\eta _1$, $\eta _2$ and $\eta _3$ can be obtained in $W_n$. Moreover, the refractive correlation matrix of the n-th column on the STAR-RIS of user $\textrm{U}_1$ and eavesdropper $\textrm{E}_1$ are written as

$$\begin{aligned} V=V_n=\left( \begin{array}{ccc} 1 &{} -0.42 &{} 0.48\\ -0.42 &{} 1 &{} -0.56\\ 0.48 &{} -0.56 &{} 1\\ \end{array} \right) , \end{aligned}$$

(57)

where the coefficients $\xi _1$, $\xi _2$ and $\xi _3$ can be obtained in $V_n$.

$$\begin{aligned} Q=Q_n=\left( \begin{array}{ccc} 1 &{} -0.35 &{} 0.45\\ -0.35 &{} 1 &{} -0.63\\ 0.45 &{} -0.63 &{} 1\\ \end{array} \right) , \end{aligned}$$

(58)

where the coefficients $\zeta _1$, $\zeta _2$ and $\zeta _3$ can be obtained in $Q_n$.

Additionally, the reflective correlation matrix of the n-th column on STAR-RIS for user $\textrm{U}_2$, and eavesdropper $\textrm{E}_2$ can be given by

$$\begin{aligned} L=L_n=\left( \begin{array}{ccc} 1 &{} -0.54 &{} 0.72\\ -0.54 &{} 1 &{} -0.28\\ 0.72 &{} -0.28 &{} 1\\ \end{array} \right) , \end{aligned}$$

(59)

where the coefficients $\phi _1$, $\phi _2$ and $\phi _3$ can be obtained in $L_n$. Besides, we have

$$\begin{aligned} G=G_n=\left( \begin{array}{ccc} 1 &{} 0.72 &{} -0.63\\ 0.72 &{} 1 &{} -0.56\\ -0.63 &{} -0.56 &{} 1\\ \end{array} \right) , \end{aligned}$$

(60)

where the coefficients $\varsigma _1$, $\varsigma _2$ and $\varsigma _3$ can be obtained in $G_n$.

Fig. 1

System model. An illustration of downlink transmission of multi-antenna STAR-RIS-aided NOMA systems

Fig. 2

The SOP versus the transmit power for multi-antenna STAR-RIS-aided NOMA network. the SOP of three schemes: RAS, OAS and BF

Figure 2 plots the SOP of our proposed scheme versus transmit power $P_s$ for different transmit antenna selection schemes, i.e., beamforming (BF), optimal antenna selection (OAS) and random antenna selection (RAS) schemes. It can be observed that the secrecy outage behavior of STAR-RIS-aided NOMA system with BF scheme is superior to that of OAS and RAS schemes. This is due to the fact that multi-antenna beamforming can increase the system capacity. It also confirms that the three transmit antenna selection schemes obtain the same SOP when $P_s$ is in the high regime, which is consistent with Remark 1. The reason is that the SOP of the system is mainly determined by the rate requirements of the cell-edge user $\textrm{U}_1$. Furthermore, we can also observe a close agreement between the simulation results and the derived theoretical analysis expressions.

Fig. 3

The theoretical results and simulation results of SOP versus transmit power. the SOP of four schemes for NOMA versus [32]

Figure 3 depicts the SOP of multi-antenna STAR-RIS-NOMA networks versus transmit power $P_s$ with different benchmarks. It is observed that the SOP of the proposed scheme is less than various benchmark schemes. The reasons are given as follows. Compared with STAR-RIS-aided OMA, the benefit of our proposed scheme originates from the fact that our proposed scheme works in full-duplex (FD) mode and is not affected by self-interference. Compared with the RIS-NOMA, the SOP is decreased because of the advantage of STAR-RIS over RIS. The reason for this phenomenon that the proposed scheme is superior to the RIS-NOMA is that the signal incident on all STAR-RIS elements is divided into the transmitted and reflected signals, while a composite smart surface consist of reflecting-only and transmitting-only RIS. Moreover, we observe that the scheme with the aid of STAR-RIS for cooperative NOMA has a better secrecy performance compared to [32]. The reason is that the STAR-RIS design aims at maximizing the SINRs of both user $\textrm{U}_1$ and user $\textrm{U}_2$ simultaneously. What’s more, our proposed scheme outperforms the STAR-RIS-NOMA with random transmit antenna selection and random shift phase modulation (RR) scheme because the multiple transmit antennas and phase alignment can be used to increase the capacity of the main channel. Furthermore, when the transmit power is in the middle and high regions, the SOPs of STAR-RIS-NOMA as well as STAR-RIS-OMA networks tend to be flat, as discussed in Remarks 1 and 2.

Fig. 4

The SOP versus the different number of columns on STAR-RIS for different schemes. the SOP of the proposed scheme versus four other schemes

Figure 4 illustrates the SOP of STAR-RIS-aided cooperative NOMA network versus different number of columns on STAR-RIS. It can be seen from this figure that the SOP of all schemes except for the RR scheme of STAR-RIS-NOMA system decrease with the number of columns on STAR-RIS because the secrecy capacity of these schemes is enhanced by increasing the number of columns. However, with the number of columns increases slowly, the SOP of the STAR-RIS-NOMA with RR scheme is unchanged almost. This is because increasing the number of columns does not effectively increase the security capacity of two NOMA users.

Fig. 5

The SOP versus the different distances from the cell-edge user to eavesdropper, i.e., $d_{U_1 E_1}$. When the transmission power is constant, the influence of different distances on the SOP

Fig. 6

The SOP versus the different distances from the cell-center user to eavesdropper, i.e., $d_{U_2 E_2}$. When the transmission power is constant, the influence of different distances on the SOP

Figures 5 and 6 demonstrate the SOP of STAR-RIS-aided cooperative NOMA network versus different eavesdropping distances. Specifically, Fig. 5 examines the SOP of STAR-RIS-aided cooperative NOMA scheme versus the distance between cell-edge user $\textrm{U}_1$ and eavesdropper $\textrm{E}_1$. One can observe that the SOPs of all schemes are decreased with the distance between the STAR-RIS and the eavesdropper $\textrm{E}_1$ increases. The reason is that the capacity of the wiretap link is decreased when the propagation distance is increased. In addition, another observation is that the SOPs of all schemes are decreased when the distance between user $\textrm{U}_2$ and the eavesdropper $\textrm{E}_2$ increases in Fig. 6. The reason is similar to that in Fig. 5. Furthermore, compared with the impact of $d_{U_1 E_1}$ on SOP, we also observe that the SOP changes slowly when $d_{U_2 E_2}$ is in the medium and high regimes. This is due to the fact that the SOP of our proposed scheme is mainly determined by $d_{U_1 E_1}$, when $d_{U_2 E_2}$ tends to be larger.

Fig. 7

The SOP versus the secrecy rate requirement of the cell-edge user. When the transmission power is constant, the influence of different secrecy rate requirement of user $\textrm{U}_1$ on the SOP

Fig. 8

The SOP versus the secrecy rate requirement of the cell-center user. When the transmission power is constant, the influence of different secrecy rate requirement of user $\textrm{U}_2$ on the SOP

Figures 7 and 8 illustrate the SOP of multi-antenna STAR-RIS-aided cooperative NOMA system for different secrecy rate requirements. Specifically, on one hand, Fig. 7 examines the SOP of STAR-RIS-aided cooperative NOMA system versus the secrecy rate requirement of the cell-edge user $\textrm{U}_1$. In this figure, it can be observed that the SOP of the proposed STAR-RIS-aided NOMA scheme and other schemes become degraded as the secrecy rate increases. This is because the successful transmission occurs when the requirement of secrecy rate increases. On the other hand, Fig. 8 examines the SOP of STAR-RIS-aided cooperative NOMA system versus the secrecy rate requirement for the cell-center user $\textrm{U}_2$. We can observe that, as expected, the SOPs increase as the secrecy rate increases. The reason is present in the forgoing analysis. As a further development, our proposed scheme achieves a better secrecy performance than the scheme in STAR-RIS-OMA network. The reason is that our proposed scheme considers the secrecy rate requirements of two NOMA users at the same time, while the STAR-RIS-OMA scheme only meets the secrecy rate requirements of a single user.

Fig. 9

The theoretical results and simulation results of SOP versus $\lambda _{m,e}$. When the transmission power is constant, The impact of the $\lambda _{m,e}$ on the SOP

Figure 9 depicts the SOP of the STAR-RIS-aided NOMA scheme versus $\lambda _{m,e}$ for different transmission schemes. As it can be observed, the proposed STAR-RIS-aided scheme of cooperative NOMA and OMA system, the conventional RIS-NOMA, and the scheme in [32], have a better secrecy performance than the STAR-RIS-NOMA with RR scheme, when $\lambda _{m,e}$ is in the medium and high regimes. The reason is due to the fact that the SDO of the STAR-RIS-NOMA with RR scheme is one. Besides, it is noted that, the SDO of the conventional RIS-NOMA is a half of STAR-RIS-NOMA scheme. Interestingly, the SDOs of the proposed scheme, STAR-RIS-OMA scheme, and the scheme in [32] remain the same as discussed in Remark 3.

Fig. 10

The theoretical results of SOP for user $\textrm{U}_2$ versus $\lambda _{m,e}$. When the transmission power is constant, The impact of the $\lambda _{m,e}$ on the SOP for user $\textrm{U}_2$

Figure 10 illustrates the SOP of the cell-center user $\textrm{U}_2$ versus $\lambda _{m,e}$ for three transmission schemes, i.e., with the direct link for eavesdropper $\textrm{E}_2$, without the direct link for eavesdropper $\textrm{E}_2$ and without the STAR-RIS-aided link for eavesdropper $\textrm{E}_2$. In this figure, we observe that the SDO of user $\textrm{U}_2$ with the direct link for eavesdropper $\textrm{E}_2$ and without the direct link for eavesdropper $\textrm{E}_2$ schemes are the same, which is irrelevant to the number of transmit antennas on the BS. Moreover, we also observe that the scheme without the STAR-RIS-aided link for eavesdropper $\textrm{E}_2$ can obtain the full order of secrecy diversity, which is relevant to both the number of transmit antennas on the BS and the number of columns on the STAR-RIS as discussed in Remark 4. The reason is that the STAR-RIS-aided links of eavesdropper $\textrm{E}_2$ include the links from the BS to the STAR-RIS by comparing with conventional cooperative communication links.

8 Conclusion

In this paper, the multi-antenna STAR-RIS-assisted downlink NOMA schemes have been investigated thoroughly. More specifically, we have investigated the SOP of STAR-RIS-aided NOMA systems with a multi-antenna BS. The closed-form and asymptotic expressions of SOP for two NOMA users are derived, respectively. Based on the analytical results, the SDOs of cell-edge user and cell-center user are obtained, respectively. It has been also shown that the SOP of our proposed scheme outperforms that of STAR-RIS-OMA and other benchmark schemes. In our future research, on one hand, we will investigate the secrecy behaviors of a more complicated scenario with multiple STAR-RISs-aided NOMA communication networks. On the other hand, the channel estimation error was taken into consideration in our future work.

Appendix

1.1 Proof of Equation (23)

According to (22), we rewrite the expression of $\gamma _{U_{1}, s_1}$ as

$$\begin{aligned} \gamma _{U_{1}, s_1} =\frac{\alpha _o \rho _{U_{1}} A_1X_{1,t}^{2}}{(1-\alpha _o) \rho _{U_{1}}A_1X_{1,t}^{2}+1}. \end{aligned}$$

(61)

The CDF of $\gamma _{U_{1}, s_1}$ can be expressed as

$$\begin{aligned} \begin{aligned} F_{\gamma _{U_{1}, s_1}}(z) =&\Pr \left( \frac{\alpha _o \rho _{U_{1}} A_1X_{1,t}^{2}}{(1-\alpha _o) \rho _{U_{1}}A_1X_{1,t}^{2}+1}<z \right) \\ =&\Pr \left( X_{1,t}<\sqrt{\frac{z}{B_1(\alpha _o-(1-\alpha _o)z)}} \right) , \end{aligned} \end{aligned}$$

(62)

where $B_{1}= \alpha _o \rho _{U_{1}} A_1$.

By employing (19), (62) can be rewritten as

$$\begin{aligned} \begin{aligned} F_{\gamma _{U_{1}, s_1}}(z) =&F_{X_{1, t}}\left( \sqrt{\frac{z}{B_1(\alpha _o-(1-\alpha _o)z)}} \right) =\int _{0}^{\sqrt{\frac{z}{B_1(\alpha _o-(1-\alpha _o)z)}} }f_{X_{1,t}}(x)dx \\ =&\frac{\gamma \left( a+1, \frac{1}{b} \sqrt{\frac{z}{B_1(\alpha _o-(1-\alpha _o)z)}} \right) }{\Gamma (a+1)}. \end{aligned} \end{aligned}$$

(63)

Thus, the expression of $F_{\gamma _{U_{1}, s_1}}(z)$ is obtained.

1.2 Proof of Equation (26)

Since $X_{2,t}$ is approximated as a Gamma distribution, $Y_{1}$ is a squared Gamma distribution. Then, the PDF of $X_{2,t}$ and the CDF of $Y_{1}$ can be, respectively, expressed as

$$\begin{aligned} f_{X_{2,t}}(x)= \frac{x^{a}}{b^{a+1}\Gamma (a+1)} \exp \left( -\frac{x}{b} \right) , \end{aligned}$$

(64)

and

$$\begin{aligned} F_{Y_{1}}(y)=\frac{\gamma (T,y^2)}{\Gamma (T)}. \end{aligned}$$

(65)

Additionally, we can easily obtain that $Y_1$ is independent of $X_{2,t}$. It is because $E[X_{2,t} Y_1] = E[X_{2,t}]E[Y_1]=0$, thus, they are independent approximately.

Firstly, we can obtain the CDF of the sum of $\sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} =Z_1$, which can be written as

$$\begin{aligned} \begin{aligned} F_{Z_1}(z)=&\Pr \left( \sqrt{C_1} Y_1+\sqrt{A_2}X_{2,t} <z\right) \\ =&\int _{0}^{\frac{z}{\sqrt{A_2}}} F_{Y_{1}}\left( \frac{z-\sqrt{A_2}x}{\sqrt{C_1}}\right) f_{X_{2,t}}(x)dx. \end{aligned} \end{aligned}$$

(66)

Then, by substituting (64) and (65) into (68), we obtain the CDF of $Z_1$, which can be written as

$$\begin{aligned} \begin{aligned} F_{Z_1}(z) =&\frac{{\textbf{B}}(2T+1,a+1) z^{2T+a+1} }{T\Gamma (T)C_1^T\Gamma (a +1)b^{a+1}A_2^{\frac{a+1}{2}}} \\&\times \,{}_1 \!F_1 \left( {a+1; } { 2T+a+2}; {-\frac{z}{b\sqrt{A_2}} } \right) , \end{aligned} \end{aligned}$$

(67)

where ${\textbf{B}}(x,y)$ is Beta functions and ${}_1 \!F_1$ is Degenerate hypergeometric function [34].

As a result, the CDF of $Z_1^2$ can be expressed as

$$\begin{aligned} F_{Z_1^2}(z) =F_{Z_1}(\sqrt{z}), \end{aligned}$$

(68)

Here, the proof of (26) is completed.

1.3 Proof of Equation (34)

Referring to (33), we can rewrite the probability $P_{U_1}$ as

$$\begin{aligned} \begin{aligned} P_{U_1}=&\Pr \left( {C_{{U_1}, s_1}-C_{{E_1},s_1}<{R}_{1}^s}\right) \\ =&\int _{0}^{\infty }F_{\gamma _{U_{1}, s_1}}\left( \gamma _{s,1}x+\gamma _{s,1}-1\right) f_{{\gamma }_{{E_1}, s_1}}(x)dx\\=&\int _{\gamma _{s,1}-1}^{\infty }\frac{F_{\gamma _{U_{1}, s_1}}\left( {t_1}\right) f_{{\gamma }_{{E_1}, s_1}}\left( \frac{t_1+1-\gamma _{s,1}}{\gamma _{s,1}} \right) }{\gamma _{s,1}}dt_1. \end{aligned} \end{aligned}$$

(69)

Unfortunately, it is challenging to derive the exact closed-form of the integral in (69). By further applying Gaussian–Chebyshev quadrature into the above integral expression, we can obtain (34).

Here, the proof is completed.