Distributions of test statistics are essential in order to find analytically the probability of false alarm *p*
_{
fa
} and the detection probability *p*
_{
d
}. *T*
_{
p
} and *T*
_{
a
} have the same statistical distribution since *W*(*k*) is *i*.*i*.*d*. and *S*(*k*) is deterministic as *s*(*n*) is assumed to be deterministic. In the following, we develop the distribution of *T*
_{
p
} under *H*
_{0} and *H*
_{1} over a Gaussian channel, where the channel effect *h* is assumed to be constant. Similarly, the distribution of *T*
_{
a
} can be found. *T*
_{
or
} made its decision by applying the logical OR on the decisions of *T*
_{
p
} and *T*
_{
n
} which have the same distribution.

### 4.1 False alarm and detection probabilities of *T*
_{
p
}

The distribution of *T*
_{
p
} depends on \({\sum \nolimits }_{k=1}^{\frac {N}{2}} \Gamma _{p}(k)\) as presented in Eq. (20). A simplification of the term \({\sum \nolimits }_{k=1}^{\frac {N}{2}} \Gamma _{p}(k)\) can be obtained as follows (see Appendix 1):

$$\begin{array}{*{20}l} \sum\limits_{k=1}^{\frac{N}{2}} \Gamma_{p}(k)&=\frac{2}{N^{2}\sigma_{w}^{2}}\sum\limits_{k=1}^{\frac{N}{2}} \sum\limits_{u=1}^{k}|Y(u)|^{2} \\ &=\frac{2}{N^{2}\sigma_{w}^{2}}\sum\limits_{k=1}^{\frac{N}{2}} \left(\frac{N}{2}-k+1\right)|Y(k)|^{2} \end{array} $$

(29)

#### 4.1.1 False alarm probability of *T*
_{
p
}

Under *H*
_{0}, the test statistic *T*
_{
p
} is only related to the noise *w*(*n*). Using Eq. (29), *T*
_{
p
} can be written as follows:

$$\begin{array}{*{20}l} T_{p}&=\sum\limits_{k=1}^{\frac{N}{2}} \left[\frac{2}{N^{2}\sigma_{w}^{2}}\left(\frac{N}{2}-k+1\right)\left|W(k)\right|^{2}-R\left(1,\frac{N}{2};k\right)\right]\\ &=\sum\limits_{k=1}^{\frac{N}{2}}\left[\Gamma_{p}(k)-R\left(1,\frac{N}{2};k\right)\right] \end{array} $$

(30)

Being the discrete Fourier transform of a white noise *w*(*n*), *W*(*k*) asymptotically follows a normal distribution since it is the sum of independent terms. It is known that two Gaussian variables are independent if they are uncorrelated [22]. As *E*[*W*(*k*)*W*
^{∗}(*k*−*m*)]=*E*[*W*(*k*)]*E*[*W*
^{∗}(*k*−*m*)]=0 (see Appendix (2)), then *W*(*k*) becomes *i*.*i*.*d*.

Being the sum of independent terms and according to the central limit theorem, the distribution of *T*
_{
p
} tends towards \(\mathcal {N}(\mu _{0},V_{0})\) under *H*
_{0}. In this case, the probability of false alarm \(p_{fa}^{p}\) of *T*
_{
p
} can be found as follows:

$$ p_{fa}^{p}=Q\left(\frac{\lambda-\mu_{0}}{\sqrt{V_{0}}}\right) $$

(31)

where *Q*(.) is the *Q*-function ^{4}, and *λ* is the threshold of comparison. Since \(E\left [|W(k)|^{2}\right ]\ =N\sigma _{w}^{2}\) and based on Eq. (30), we can evaluate *μ*
_{0} as follows:

$$\begin{array}{*{20}l} \mu_{0}&=E[T_{p} ] \\ &=\frac{2}{N^{2}\sigma_{w}^{2}}\sum_{k=1}^{\frac{N}{2}} \left(\frac{N}{2}-k+1\right)E\left[|W(k)|^{2}\right]-\frac{N+2}{4} \\ &=0 \end{array} $$

(32)

In this case, the variance, *V*
_{0}, of *T*
_{
p
} becomes (see Appendix 4)

$$\begin{array}{*{20}l} V_{0}&=E\left[T_{p}^{2}\right]=\frac{(N+2)(N+1)}{6N} \end{array} $$

(33)

#### 4.1.2 Probability of detection of *T*
_{
p
}

Under *H*
_{1}, *Y*(*k*)=*hS*(*k*)+*W*(*k*), then Eq. (29) becomes as follows:

$$ \Gamma_{p}(k)=\frac{2}{N^{2}\sigma_{w}^{2}}\sum\limits_{k=1}^{\frac{N}{2}} \left(\frac{N}{2}-k+1\right)\left|hS(k)+W(k)\right|^{2} $$

(34)

Since *S*(*k*) is deterministic and the terms of *W*(*k*) are independent, the distribution of *T*
_{
p
} under *H*
_{1} tends also towards \(\mathcal {N}(\mu _{1},V_{1})\). In this case, the probability of detection \(p_{d}^{p}\), of *T*
_{
p
} can be found as follows:

$$ p_{d}^{p}=Q\left(\frac{\lambda-\mu_{1}}{\sqrt{V_{1}}}\right) $$

(35)

*μ*
_{1} and *V*
_{1} should be evaluated in order to find \(p_{d}^{p}\). Under *H*
_{1}, |*Y*(*k*)|^{2} becomes

$$\begin{array}{*{20}l} |Y(k)|^{2}&=Y(k)Y^{*}(k) \\ &=|hS(k)|^{2}+|W(k)|^{2}+2Re\{hS(k)W^{*}(k)\} \end{array} $$

(36)

where *hS*(*k*)=*DFT*{*hs*(*n*)} and *Re*{*X*} is the real part of *X*. The mean value of *T*
_{
p
} under *H*
_{1} can be found as follows (see Appendix 3):

$$ \mu_{1}=b\gamma $$

(37)

where \(b=\frac {2}{N^{2}}\sum _{k=1}^{\frac {N}{2}}(\frac {N}{2}-k+1)|S(k)|^{2}\), and *γ* is the SNR as defined by Eq. (4).

Under *H*
_{1}, the variance *V*
_{1} is given by the following equation (see Appendix 5):

$$\begin{array}{*{20}l} V_{1}&\,=\,V_{0}+\frac{8\gamma}{N^{3}}\sum_{k=1}^{\frac{N}{2}} \left(\frac{N}{2}-k+1\right)^{2}|S(k)|^{2} &=V_{0}+c\gamma \end{array} $$

(38)

where \(c=\frac {8}{N^{3}}{\sum \nolimits }_{k=1}^{\frac {N}{2}} \left (\frac {N}{2}-k+1\right)^{2}|S(k)|^{2}\).

*T*
_{
a
} is based on similar idea to *T*
_{
p
}, but it covers the *N* frequency points instead of just positive frequency points (\(\frac {N}{2}\) points). Since *W*(*k*) is *i*.*i*.*d*. and *S*(*k*) is deterministic, then by following the same process for \(p_{fa}^{p}\) and \(p_{d}^{p}\), the probability of false alarm \(p_{fa}^{a}\) and the probability of detection \(p_{d}^{a}\) of the detector *T*
_{
a
} can be found as follows:

$$\begin{array}{*{20}l} p_{fa}^{a}=Q\left(\frac{\lambda-\mu_{0}^{a}}{\sqrt{V_{0}^{a}}}\right) \end{array} $$

(39)

$$\begin{array}{*{20}l} p_{d}^{a}=Q\left(\frac{\lambda-\mu_{1}^{a}}{\sqrt{V_{1}^{a}}}\right) \end{array} $$

(40)

where \(\mu _{0}^{a}=0\); \( V_{0}^{a}=\frac {N}{3}+\frac {1}{2}+\frac {1}{6N}\); \(\mu _{1}^{a}=b_{a}\gamma \), where \(b_{a}=\frac {1}{N^{2}}\sum _{k=1}^{N}(N-k+1)|S(k)|^{2}\), and \(V_{1}^{a}=V_{0}^{a}+c_{a}\gamma \), where \(c_{a}=\frac {2}{N^{3}}\sum _{k=1}^{N} (N-k+1)^{2}|S(k)|^{2}\).

### 4.2 Probabilities of *T*
_{
or
}

*T*
_{
or
} applies the OR rule between the decisions of *T*
_{
p
} and *T*
_{
n
}, then *T*
_{
or
} can be considered as a hard cooperative detector of these two detectors. Since *T*
_{
p
} and *T*
_{
n
} are independent and have the same statistics as we defined previously, the probability of false alarm \(p_{fa}^{or}\) and the probability of detection \(p_{d}^{or}\) of *T*
_{
or
} can be found as follows [2]:

$$\begin{array}{*{20}l} p_{fa}^{or}&=1-\left(1-p_{fa}^{p} \right)^{2} \end{array} $$

(41)

$$\begin{array}{*{20}l} p_{d}^{or}&=1-\left(1-p_{d}^{p} \right)^{2} \end{array} $$

(42)

### 4.3 Probabilities of *T*
_{
av
}

*T*
_{
av
} can be developed following similar steps as Eq. (30).

$$\begin{array}{*{20}l} T_{av}&=\frac{2}{N^{2}\sigma_{w}^{2}}\sum\limits_{k=1}^{\frac{N}{2}}\left(\frac{N}{2}-k+1\right)\frac{|Y(k)|^{2}+|Y(-k+1)|^{2}}{2}\\ &\quad-\frac{N+2}{4} \end{array} $$

(43)

Under *H*
_{0}, *Y*(*k*)=*W*(*k*), then *T*
_{
av
} becomes the sum of independent terms. Based on CLT, *T*
_{
av
} asymptotically follows \(\mathcal {N}(\mu _{0}^{av},V_{0}^{av})\) under *H*
_{0}.

Under *H*
_{1}, *Y*(*k*)=*hS*(*k*)+*W*(*k*) and *S*(*k*) is deterministic, so *T*
_{
av
} is still following a normal distribution: \(\mathcal {N}(\mu _{1}^{av},V_{1}^{av})\) under *H*
_{1}.

The probability of false alarm, \(p_{fa}^{av}\), and detection, \(p_{d}^{av}\), of *T*
_{
av
} are expressed as follows:

$$\begin{array}{*{20}l} p_{fa}^{av}=Q\left(\frac{\lambda-\mu_{0}^{av}}{\sqrt{V_{0}^{av}}}\right) \end{array} $$

(44)

$$\begin{array}{*{20}l} p_{d}^{av}=Q\left(\frac{\lambda-\mu_{1}^{av}}{\sqrt{V_{1}^{av}}}\right) \end{array} $$

(45)

Since *W*(*k*) is *i*.*i*.*d*. and *P*
_{
s
}(*k*) is even, \(\mu _{0}^{av},V_{0}^{av}\), \(\mu _{1}^{av}\), \(V_{1}^{av}\) can be found by following similar steps to *μ*
_{0}, *V*
_{0}, *μ*
_{1} and *V*
_{1}, we can find that \(\mu _{0}^{av}=\mu _{0}\), \(V_{0}^{av}=\frac {V_{0}}{2}\), \(\mu _{1}^{av}=\mu _{1}\), and \(V_{1}^{av}=\frac {V_{1}}{2}\).

The theoretical and the simulated ROC curves of proposed detectors are with good agreement as shown in Figs. 3
a, b. Simulations were done under following conditions: 16-QAM modulation, *γ*=−12 dB, *N*=1000 samples, and *N*
_{
s
}=4 sps.

As shown in Fig. 3
a, *T*
_{
av
} is the most efficient detector for both considered shaping filters. For the simulations of Section 6 under Gaussian channel, only *T*
_{
av
} and *T*
_{
or
} are compared to other well-known detectors. The rest of simulations in this paper are done with a rectangular pulse-shaping filter.