Locally optimal detector design in impulsive noise with unknown distribution

Abstract

This paper designs the locally optimal detector (LOD) in additive white impulsive noise with unknown distribution. Unlike traditional LODs derived from a known or approximated noise probability density function (PDF), the LOD proposed in this paper is achieved by designing the zero-memory non-linearity (ZMNL) function based on real data. After the PDF estimation in a nonparametric way by a kernel method, the ZMNL function is designed as a piecewise differentiable function consisting of a polynomial function and inverse proportional functions. Then, we analyze the detection performance and develop the constant false alarm ratio technique. Simulation results show that the LOD design is near-optimal in α-stable noise and the optimal in real atmospheric data, compared with the maximum likelihood detector of α-stable distribution.

1 Introduction

Generally, signal detection in additive noise can be viewed as a problem of binary or multiple hypothesis testing [1]. Most existing digital systems use linear detectors which are the optimal in additive white Gaussian noise. However, nonlinear processing is required for optimal detection in impulsive noise with heavy PDF tails. This is necessary for many environments, such as vehicular communication, power line communication, low-frequency communication, and underwater communications [2–4]. In low signal-to-noise ratio (SNR), the LOD can be realized via a simple structure by a nonlinear preprocessor based on ZMNL processing followed by a linear correlator [5, 6]. As is well-known, the ZMNL function can be derived directly if the noise PDF in closed form is known.

However, the ZMNL function needs designing when the analytical PDF is unavailable. For instance, the α-stable noise, which is widely used for modeling the impulse noise, does not provide the PDF in closed form generally [4]. As a result, researchers have developed various ZMNL functions for different α-stable distributions [7–9]. However, real-data processing results demonstrate that the impulsive noise does not always strictly obey the α-stable distribution, though it generally has a unimodal PDF like the α-stable noise. Therefore, the ZMNLs which are designed for α-stable noise are not surely optimal in real-data processing, as illustrated by the simulations in Section 6.4.

This paper focuses on the LOD design when the noise distribution is unknown, which is not fully discussed before. Actually, this is a reasonable consideration since the noise in real world would probably be varying and disobey the assumed distributions, e.g., the α-stable distribution. To solve this problem, this paper proposes to develop a practical approach for designing the ZMNL based on the real-noise data instead of knowing the PDF or assuming the noise distribution.

In the LOD design, lessons are drawn from the existing ZMNL functions of the symmetric α-stable (S αS) noise which is practically useful as a heavy-tailed distribution. In this paper, the ZMNL is designed as a piecewise function which follows linearity, nonlinearity, and differentiability in different regions. Then, the detection performances are analyzed theoretically. In simulations, the proposed approach is demonstrated in simulated S αS noise and in real atmospheric noise. The detection performances will be compared with several existing detectors.

The remainder of this paper is organized as follows. Section 2 briefly reviews several detectors for known noise PDFs. Section 3 introduces the preparation works for unknown noise PDF. Then, the ZMNL function is designed in Section 4, and detection performances are analyzed in Section 5. Section 6 presents the simulations in α-stable noise and real atmospheric noise. Finally, conclusions are drawn in Section 7.

2 Detectors in white noise with known PDF

Detecting a deterministic signal in additive white noise is a classical problem in statistical signal processing. A general solution is by hypothesis testing. Under hypothesis H _i , the received signal is modeled as

$$\begin{array}{*{20}l} H_{i}: \boldsymbol{r} = \xi_{i} \cdot \boldsymbol{s}_{i}+ \boldsymbol{x} \end{array} $$

(1)

where s _i , ξ _i , and x denote the desired signal, the signal amplitude, and the additive white noise, respectively [1]. For the case of multiple sample detection, r, s _i , and x can be considered as M-dimensional row vectors.

Maximum likelihood detector (MLD) is a well-known detector and requires the noise PDF. Considering multiple sample detection in white noise with a PDF f(x), the MLD is formulated as

$$\begin{array}{*{20}l} \max_{H_{i}}\; \ln\text{Pr}(\boldsymbol{r}, \boldsymbol{s}_{i} | H_{i}) \doteq \max_{H_{i}} \;\sum_{m=1}^{M} \ln f\left(\boldsymbol{r} [\!m]- \xi_{i} \boldsymbol{s}_{i}[\!m] \right) \end{array} $$

(2)

where the symbol “ $\doteq$ ” denotes “being equivalent to.”

The decision rule (2) can be simplified for Gaussian noise. Under a realistic consideration of knowing the sign of ξ _i , without loss of generality, assume ξ _i >0. Then, simplifying (2) leads to the matched filter detector (MFD), formulated as

$$\begin{array}{*{20}l} \max_{H_{i}}\; \ln \text{Pr}(\boldsymbol{r}, \boldsymbol{s}_{i} ; H_{i})& \doteq \max_{H_{i}}\; -\sum_{m=1}^{M}\left(\boldsymbol{r}[\!m]- \xi_{i} \boldsymbol{s}_{i}[\!m]\right)^{2} \\ & \doteq \max_{H_{i}}\; \boldsymbol{r}\boldsymbol{s}_{i}^{\dag}, \quad \textrm{for Gaussian noise} \end{array} $$

(3)

where (·)^† denotes transpose. Obviously, the MFD depends on the linear correlation.

Decision rule (3) does not hold true for non-Gaussian noise. However, the MLD can be simplified in low SNR cases and results in the LOD, formulated as

$$\begin{array}{*{20}l} \max_{H_{i}} \;\ln \text{Pr}(\boldsymbol{r}, \boldsymbol{s}_{i} ; H_{i}) \doteq \max_{H_{i}} \; g(\boldsymbol{r})\boldsymbol{s}_{i}^{\dag},\quad \textrm{for low\, SNR} \end{array} $$

(4)

where

$$\begin{array}{*{20}l} g(x)=-{f^{\prime}}(x)/f(x) \end{array} $$

(5)

is called as the ZMNL function [1].

As can be seen from (5), g(x) is uniquely determined by f(x). Given analytically known f(x), g(x) is obtained directly in closed form. Otherwise, g(x) must be designed. Researchers usually chose to study g(x) by approximating the PDF. Figure 1 shows three reported ZMNL functions proposed for symmetric α-stable (S αS) noise, including the algebraic-tailed ZMNL (AZMNL, depending on α, for S αS distribution) [7], the Cauchy ZMNL (CZMNL, for Cauchy distribution α=1) [8], and the Gaussian-tailed ZMNL (GZMNL, independent on α, robust for S αS distribution) [9]. The ZMNL curves in Fig. 1 show that g(x) varies greatly for different distributions. This demonstrates the necessity of designing the optimal LOD for a unique distribution.

Fig. 1

The LODs for S αS noise and reported ZMNL design [7], for σ=1

3 Preparation of LOD design for unknown PDF

This paper proposes to design the LOD and g(x) based on real data when the PDF is unknown. Preparatory work raised by unknown PDF is introduced in this section. The detailed ZMNL design approach will be presented in the next section.

3.1 Guidelines for the LOD design

This paper focuses on impulsive noise which comes from unimodal heavy-tailed distributions and has a PDF similar to the PDFs of existing heavy-tailed distributions such as the S αS noise and the Middleton class A noise [2]. However, on the one hand, due to the remarkable differences among the PDFs, the ZMNL functions of existing noise models are possibly not the optimal for the impulsive noise in various scenarios, giving rise to the necessity of LOD design. On the other hand, since the PDF shapes are similar, the knowledge of S αS noise is referable for our LOD design.

Analysis on the existing ZMNL functions of impulsive noise shows that most of them can be regarded as piecewise functions which consist of the main body in the near-linear region and the tails in the non-linear regions. At the breakpoints, natural ZMNLs deduced from heavy-tailed distributions are generally differentiable, while designed ZMNLs may be continuous or discontinuous.

In this paper, g(x) is designed as a piecewise function, following three guidelines. (i) The main body of g(x) is estimated by polynomial fitting. (ii) The tails of g(x) are modeled as reciprocal functions. (iii) The breakpoints are localized for continuity and differentiability. The detailed reasons of the guidelines will be presented in Section 4 where we develop the LOD design procedures. Before that, the rest of this section will introduce the preparatory work for the LOD design.

3.2 ZMNL sample calculation via the KDE

When the noise distribution is unknown, the PDF can be estimated by nonparametric estimation methods. Herein, this paper proposes to measure the PDF samples by the kernel density estimation (KDE) method, i.e., the Parzen-Rosenblatt window method [10, 11].

Let {x₁,x₂,…,x _L } denote the amplitude vector with uniform increment Δx. Given the noise data $\widetilde {x}[n], n=1,2,\dots,N$, the PDF samples are modeled as

$$\begin{array}{*{20}l} \widetilde{f}[\!x_{l}]=\frac{1}{Nh}\sum_{n=1}^{N}\mathcal{K}\left(\frac{x_{l}-\widetilde{x}[\!n]}{h}\right) \end{array} $$

(6)

where $\mathcal {K}(\cdot)$ is the kernel function and h>0 is the smoothing parameter called the bandwidth.

Then, by formula (5), the ZMNL samples are calculated as

$$\begin{array}{*{20}l} \widetilde{g}[\!x_{l}]=-{\widetilde{f}_{\Delta}[\!x_{l}]}/{\widetilde{f}[\!x_{l}]} \end{array} $$

(7)

where

$$\begin{array}{*{20}l} \widetilde{f}_{\Delta}[\!x_{l}]=\left(\widetilde{f}[\!x_{l+1}]-\widetilde{f}[\!x_{l-1}]\right) /{(2\Delta x)} \end{array} $$

(8)

simulates f^′(x) at x=x _l . Specially, let $\widetilde {g}[\!x_{1}]=\widetilde {g}[\!x_{L}]=0$.

It is obvious that the value of bandwidth h is critical for the PDF estimation. In fact, since the PDF and ZMNL samples are simulated numerically in discrete points, the increment Δx is also very important. It must be evaluated reasonably so that $\widetilde {f}[\!x_{l}]$ can describe the shape of f(x) effectively.

As said before, this paper emphasizes on the impulsive noise from heavy-tailed distributions. Although the noise may disobey the α-stable distribution, the α-stable distribution characteristics are also referable in some aspects. In our practice, the α-stable methods are found to be efficient in the evaluation of h and Δx.

The evaluation approach is developed as follows. Firstly, estimate the α-stable distribution parameters based on the noise samples $\widetilde {x}[\!n]$. An estimation method can refer to paper [12] which is based on the quantile of samples.

Denote the estimated dispersion as $\widehat {\gamma }$, which has similar meaning to the variance in Gaussian distribution. Then, the parameters related to the KDE are set as

$$\begin{array}{*{20}l} h&=\Delta x=0.1 \widehat{\gamma}, \end{array} $$

(9)

$$\begin{array}{*{20}l} x_{1}&= \frac{1}{N}\sum\limits_{n=1}^{N} \widetilde{x}[\!n]-\frac{N}{2}\Delta{x}, \end{array} $$

(10)

$$\begin{array}{*{20}l} x_{L}&= \frac{1}{N}\sum\limits_{n=1}^{N} \widetilde{x}[\!n]+\frac{N}{2}\Delta{x}. \end{array} $$

(11)

By this approach, the PDF estimate $\widetilde {f}[\!x_{l}]$ can be smooth and well represent the probability density of noise samples.

3.3 ZMNL sample extraction

The ZMNL samples must be classified before using them for LOD design. The reason is as follows. As can be seen from (5), when $\widetilde {f}(x_{l})$ is very small, the error between $\widetilde {f}_{\Delta }[\!x_{l}]$ and f^′(x _l ) can be greatly increased when entering $\widetilde {g}[\!x_{l}]$, resulting in huge gap between $\widetilde {g}[\!x_{l}]$ and g(x) (see simulations in Section 6). Thus, $\widetilde {g}[\!x_{l}]$ samples for small $\widetilde {f}[\!x_{l}]$ are unreliable.

Considering the ZMNL curves are nearly straight in center where $\widetilde {f}[\!x_{l}]$ is large, we can extract the reliable ZMNL samples based on linearity, i.e., the cross-correlation coefficient between $\widetilde {g}[\!x_{i}]$ and x. The linearity of $\widetilde {g}[\!x_{i}]$ samples in any region $\widetilde {\Omega }$ is measured by

$$\begin{array}{*{20}l} \rho_{\widetilde{\Omega}}=\frac{\sum\limits_{x_{i} \in {\widetilde{\Omega}}}(x_{i}-\overline{x_{i}})\left(\widetilde{g}[\!x_{i}]-\overline{\widetilde{g}[\!x_{i}]}\right)}{\sqrt{\sum\limits_{x_{i} \in {\widetilde{\Omega}}}(x_{i}-\overline{x_{i}})^{2} \cdot \sum\limits_{x_{i} \in {\widetilde{\Omega}}}\left(\widetilde{g}[\!x_{i}]-\overline{\widetilde{g}[\!x_{i}]}\right)^{2}}} \end{array} $$

(12)

where $\overline {x_{i}}$ and $\overline {\widetilde {g}[\!x_{i}]}$ denote the mean of x _i and $\widetilde {g}[\!x_{i}]$, respectively, for $x_{i} \in {\widetilde {\Omega }}$.

The extraction algorithm is summarized in Table 1, where ρ_th is a threshold for linearity controlling. Generally, it is set 0.5≤ρ_th≤0.8.

Table 1 Algorithm of ZMNL sample extraction

The extracted ZMNL samples are denoted by

$$\begin{array}{*{20}l} \breve{{g}}[\!x_{i}]=\widetilde{g}[\!x_{i}],\; x_{i} \in \Omega=[\!x_{L_{\text{low}}}, x_{L_{\text{up}}}] \end{array} $$

(13)

which has I=L_up−L_low+1 samples.

4 Design of the LOD and ZMNL function

In the LOD design, the ZMNL function is modeled as a piecewise function. The main body is estimated by polynomial fitting. The tails and the breakpoints are determined for continuity and differentiability.

4.1 Polynomial fitting for near-linear region

The proposal of polynomial fitting is based on the basic property of ZMNL functions. As can be seen in (4), the ZMNL function transforms the observations r into the domain related to probability, acting like a weighting function. It may be linear (in Gaussian case) or nonlinear (in non-Gaussian cases). In either case, g(x) is supposed to keep close correlation to x for the majority of x, so that the test statistic $g(\boldsymbol {r}) \boldsymbol {s}_{i}^{\dag }$ could be optimized. Therefore, the near-linear region always exists in g(x) for heavy-tailed distributions.

This paper employs a polynomial function to describe the nonlinearity property and also keep certain linearity in the near-linear region. This is unlike traditional ZMNL designs which define the nonlinear region very differently from the linear region and produce unavoidable error to the real LOD.

Considering the degree P polynomial, the ZMNL function is modeled as

$$\begin{array}{*{20}l} \sum_{p=0}^{P} A_{p}x^{p}, \quad x\in \Omega \end{array} $$

(14)

where A _p , for p=0,1,…,P is the coefficient to be estimated. By using the ZMNL samples in (13), the linear equations can be written as

$$\begin{array}{*{20}l} \boldsymbol{A}\boldsymbol{X}_{\mathrm{v}}=\boldsymbol{G}_{x} \end{array} $$

(15)

where

$$\begin{array}{*{20}l} \boldsymbol{A}&=\left[A_{0} \quad A_{1} \quad \cdots \quad A_{P} \right], \end{array} $$

(16)

$$\begin{array}{*{20}l} \boldsymbol{G}_{x}&=\left[ \breve{g}[\!x_{1}] \quad \breve{g}[\!x_{2}] \quad \cdots \quad \breve{g}[\!x_{I}] \right], \end{array} $$

(17)

$$\begin{array}{*{20}l} \boldsymbol{X}_{\mathrm{v}}&=\left[{\begin{array}{cccc} 1 & 1& \cdots & 1\\ x_{1}^{1} &x_{2}^{1} & \cdots & x_{I}^{1} \\ x_{1}^{2} &x_{2}^{2} & \cdots & x_{I}^{2} \\ \vdots &\vdots & \ddots & \vdots \\ x_{1}^{P} &x_{2}^{P} & \cdots & x_{I}^{P} \end{array}} \right]. \end{array} $$

(18)

Since X_v is a Vandermonde matrix, a solution in least square (LS) sense is guaranteed to hold, provided that I≫P is satisfied generally. By the LS estimation, the parameter vector estimate is achieved by

$$\begin{array}{*{20}l} \widehat{\boldsymbol{A}}=\boldsymbol{G}_{x} \boldsymbol{X}_{\mathrm{v}}^{\dag} \left(\boldsymbol{X}_{\mathrm{v}} \boldsymbol{X}_{\mathrm{v}}^{\dag} \right)^{-1}. \end{array} $$

(19)

Plugging $\widehat {\boldsymbol {A}}$ into (14) yields

$$\begin{array}{*{20}l} \widehat{g}(x)= \sum\limits_{p=0}^{P} \widehat{A}_{p} x^{p}, \quad & x \in \Omega. \end{array} $$

(20)

Thus, the main body of $\widehat {g}(x)$ is achieved.

4.2 Tail design and breakpoint localization

Usually, the tail of g(x) functions as a controller which limits the effects of large-magnitude samples from heavy-tailed distributions. The tail function can be developed in various methods. For instance, Nikais and Shao proposed two kinds of nonlinearity for x∉Ω, including a “hole puncher” to null g(x) and a “clipper” to set g(x) equal to the nearest g(x _i ) as constants. The tail of g(x) is also modeled as x^1−α or x⁻¹ in α-stable noise [4, 7]. Besides, the breakpoints which connect the tails with the main body are also important for the LOD design.

This paper models the tail as a reciprocal function and localizes the breakpoint for differentiability everywhere. Hereby, g(x) is formulated as a piecewise function

$$\begin{array}{*{20}l} \widehat{g}(x)=\left\{{\begin{array}{lc} B_{ne}/x, & x < x_{ne} \\ \sum\limits_{p=0}^{P} \widehat{A}_{p} x^{p}, & x_{ne} \leq x \leq x_{po} \\ B_{po}/x, & x > x_{po} \end{array}} \right. \end{array} $$

(21)

where the breakpoint locations x _ne and x _po and two coefficients B _ne and B _po are the parameters to determine.

Under the consideration of continuity and differentiability, four equations of x _ne , x _po , B _ne , and B _po can be obtained, listed in a set

$$\begin{array}{*{20}l} \left\{{\begin{array}{c} \frac{B_{ne}}{x_{ne}}= \sum\limits_{p=0}^{P} \widehat{A}_{p} x_{ne}^{p}\\ - \frac{B_{ne}}{x^{2}_{ne}}=\sum\limits_{p=1}^{P} p\widehat{A}_{p} x_{ne}^{p-1}\\ \frac{B_{po}}{x_{po}}= \sum\limits_{p=0}^{P} \widehat{A}_{p} x_{po}^{p} \\ - \frac{B_{po}}{x^{2}_{po}}= \sum\limits_{p=1}^{P} p\widehat{A}_{p} x_{po}^{p-1} \\ \end{array}} \right.. \end{array} $$

(22)

The solution of (22) is included in Table 2 which summarizes the algorithm of designing ${\widehat {g}}(x)$. Consequently, the LOD can be achieved by using $\widehat {g}(x)$ in the decision rule (4).

Table 2 Algorithm of ZMNL function design

As far as we know, this is the first approach to use polynomial fitting for designing the ZMNL function $\widehat {g}(x)$. Herein, we call the proposed ZMNL design as polynomial ZMNL (PZMNL).

One merit of the PZMNL is that it can approximate the main body of real LOD very well. As shown in Fig. 1, the LOD g(x) changes smoothly without a breakpoint to clearly classify the near-linear region and the nonlinear region. However, traditional ZMNL functions change dramatically at the breakpoints, from proportional functions in the near-linear region into nonlinear functions in the nonlinear region. They are greatly different from g(x) and thus have an upper bound of being sub-optimal. Unlike them, the PZMNL can fit well around the near-linear region and achieve nearly optimal performances.

Notation: About the polynomial order P, practical experience shows that 10≤P≤20 is suitable for polynomial fitting. Otherwise, if P is too small, the designed polynomial cannot represent the nonlinearity beyond the near-linear region and also reduces the solution to Eq. (23). On the contrary, if P is too large, the designed polynomial is not smooth enough and also generates unfavorable solutions to (23), resulting in a narrow near-linear region. Examples can refer to the simulations in Section 6.

5 Detection performance analysis of the LOD

This section analyzes the detection performances of the detectors based on various ZMNL functions. Supposing h(x) as the ZMNL function before a linear detector, based on the signal model (1), the test statistic is

$$\begin{array}{*{20}l} T(\boldsymbol{r})= h(\boldsymbol{r})\boldsymbol{s}^{\dag}_{i}=\sum\limits_{m=1}^{M} h(r[\!m])s_{i}[\!m] \end{array} $$

(25)

for hypothesis H _i . If the PDF of T(r) is known, the detection performance can be deduced accordingly. However, the PDF of T(r) is uneasy to obtain generally.

Herein, we consider this topic in low SNR as a usual scenario in real world. Obviously, low SNR means that ξ _i is rather small. As a result, to detect a transmitted signal bit, communication systems need to accumulate massive samples as M is large. Besides, the noise samples are assumed to be independently and identically distributed (i.i.d.). Therefore, by using the central limit theorem, we suppose that the test statistic T(r) obeys the Gaussian distribution. Its PDF is available as long as we know its expectation and covariance.

Before the deduction, the following assumption is made to simplify analysis.

Assumption 1

The impulsive noise is zero-mean, symmetrically distributed; thus, f(x) is even and f^′(x) is odd. The function h(x) used for ZMNL processing is considered to be almost odd, leading to

$$\begin{array}{*{20}l} \int\limits_{-\infty}^{\infty}h(x)f(x)dx \approx 0,\, \int\limits_{-\infty}^{\infty}h^{2}(x)f'(x)dx \approx 0. \end{array} $$

(26)

Then, the asymptotic distribution of T(r) in low SNR is derived, concluded as Theorem 1.

Theorem 1

For the signal model in (1) under Assumption 1 and low SNR, the test statistic T(r) in (25) asymptotically obeys Gaussian distribution

$$\begin{array}{*{20}l} T({\boldsymbol{r}}) \,\substack{a\\\sim\\ \quad}\, \mathcal{N} \left(\mathcal{E}_{hg} \xi_{i} \boldsymbol{s}_{i} \boldsymbol{s}_{i}^{\dag}, \mathcal{E}_{hh}\boldsymbol{s}_{i} \boldsymbol{s}_{i}^{\dag} \right), \quad H_{i} \end{array} $$

(27)

for M→∞, where

$$\begin{array}{*{20}l} \mathcal{E}_{hg}&=\int\limits_{-\infty}^{\infty}h(x)g(x)f(x)dx, \end{array} $$

(28)

$$\begin{array}{*{20}l} \mathcal{E}_{hh}&=\int\limits_{-\infty}^{\infty}h^{2}(x)f(x)dx. \end{array} $$

(29)

The proof is referred in Appendix. Note that $\mathcal {E}_{hg}$ and $\mathcal {E}_{hh}$ are actually the expectations of h(x)g(x) and h²(x), respectively.

By Theorem 1, we can analyze the detection performances of a designed ZMNL function in a convenient way. For instance, in binary hypotheses

$$\begin{array}{*{20}l} \boldsymbol{r}=\left\{{\begin{array}{cl} \boldsymbol{x}, & H_{0}\\ \xi \cdot \boldsymbol{s}+ \boldsymbol{x}, & H_{1} \end{array}} \right. \end{array} $$

(30)

the asymptotic distribution of T(r) can be written as

$$\begin{array}{*{20}l} T({\boldsymbol{r}}) \,\overset{a}{\sim} \left\{ {\begin{array}{cl} \mathcal{N} \left(0, \mathcal{E}_{hh}\boldsymbol{s} \boldsymbol{s}^{\dag} \right), & H_{0}\\ \mathcal{N} \left(\mathcal{E}_{hg} \xi \boldsymbol{s} \boldsymbol{s}^{\dag}, \mathcal{E}_{hh}\boldsymbol{s} \boldsymbol{s}^{\dag} \right), & H_{1} \end{array}} \right. \end{array} $$

(31)

Then, the constant false alarm ratio (CFAR) technique can be achieved. A desired false alarm ratio (FAR) P _fa is available by the threshold

$$\begin{array}{*{20}l} \eta=\sqrt{\mathcal{E}_{hh}\boldsymbol{s} \boldsymbol{s}^{\dag} }Q^{-1}(P_{fa}) \end{array} $$

(32)

and the corresponding detection probability is given by

$$\begin{array}{*{20}l} P_{D}&=Q\left[\left(\eta-\mathcal{E}_{hg} \xi \boldsymbol{s} \boldsymbol{s}^{\dag}\right)/\sqrt{\mathcal{E}_{hh}\boldsymbol{s} \boldsymbol{s}^{\dag}}\right]\\ &=Q\left[Q^{-1}(P_{fa})-\mathcal{E}_{hg} \xi \sqrt{\boldsymbol{s} \boldsymbol{s}^{\dag}/\mathcal{E}_{hh}}\right] \end{array} $$

(33)

where Q(·) is the tail probability of the standard normal distribution, defined as

$$\begin{array}{*{20}l} Q(x)=\frac{1}{2\pi}\int\limits_{x}^{\infty} \exp\left(-\frac{u^{2}}{2}\right)du. \end{array} $$

(34)

As clearly shown in (29) and (32), the calculation of threshold η depends on the information of PDF f(x). Thus, for the LOD in the noise with unknown PDF, the CFAR technique is actually unavailable.

The following develops the CFAR technique for the noise with unknown PDF. Notice that false alarm is related to hypothesis H₀ and independent of hypothesis H₁. In (31), hypothesis H₀ requires f(x) to calculate the covariance coefficient $\mathcal {E}_{hh}$. Though g(x) is also unknown, it is unnecessary in hypothesis H₀. In fact, g(x) is only required in hypothesis H₁ to calculate the expectation coefficient $\mathcal {E}_{hg}$ in (28) and finally affects the detection probability P _D in (33).

Therefore, this paper proposes the CFAR technique by using the PDF estimate $\widetilde {f}[x_{l}]$ instead of the unknown real PDF f(x). By employing $\widetilde {f}[\!x_{l}]$ in (6), we can compute $\mathcal {E}_{hh}$ in (29) and then the threshold η by formula (32) for the desired FAR. Simulation results show that this approach can achieve a steady FAR which is very close to the desired FAR.

6 Results and discussion

This section presents the results of LOD design in the S αS noise and real atmospheric noise. Discussions about the proposed design of LOD and PZMNL are provided.

6.1 Detector and simulation settings

The general approach to suppress heavy-tailed interference is to pass the received observation through a ZMNL limiter. Two simple examples include a hole puncher and a clipper [4]. Some limiters are designed carefully according to the noise model.

This paper simulates three ZMNL methods which are proposed for S αS noise. The CZMNL is developed for the Cauchy distribution, i.e., S αS noise for α=1, formulated as

$$\begin{array}{*{20}l} g_{c}(x)=2x/\left(x^{2}+\sigma^{2}\right). \end{array} $$

(35)

The AZMNL is proposed for standard S αS noise, formulated as

$$\begin{array}{*{20}l} g_{a}(x)=\left\{ {\begin{array}{ll}{x\cdot\Gamma(3/\alpha)}/{\Gamma(1/\alpha)}, & |x| \leq \tau\\ K(\alpha)/{x}, & |x| > \tau \end{array}} \right. \end{array} $$

(36)

where $\tau = \sqrt {K(\alpha)\Gamma (1/\alpha)/\Gamma (3/\alpha)}$ uses K(α)=α². The GZMNL formula can refer to [7, 9] and is not introduced here for the sake of simplification.

As the analytical PDF of S αS noise is generally unavailable, the MLD is achieved by a numerical approach as follows. Firstly, the discrete PDF is computed by the inverse Fourier transform of the characteristic function. Then, assuming the signal amplitude is known, decision rule (2) is realized by linear interpolation. The ideal LOD by (5) is also obtained by linear interpolation of the discrete differential. Simulation results show that the MLD and the LOD perform very close to each other. Thus, the LOD and the MLD are depicted by the MLD curve. Obviously, linear interpolation bears expansive computational cost, but it is necessary for the MLD.

As S αS noise has an infinite covariance for α<2, herein, the SNR is measured by the generalized SNR (GSNR) as

$$\begin{array}{*{20}l} GSNR = 10\times \log_{10}\left({\xi^{2}}/{ \gamma }\right) \quad\textrm{in \; decibels} \end{array} $$

(37)

where γ denotes the dispersion of S αS noise. In simulations, γ is fixed as 1, so that the AZMNL in (36) can be used. The GSNR is changed by adjusting the signal amplitude ξ. The sinusoidal signal is transmitted, with uniform energy ss^†=M=1024. The probabilities of detection and false alarm are the results of 10⁵ Monte Carlo simulations.

6.2 ZMNL design in S αS noise

α-Stable distribution is widely used in impulse noise modeling. As its PDF is generally unavailable in closed form, the analytical LOD does not exist for most α-stable distributions. Herein, we simulate the LOD of S αS noise numerically. S αS noise data is simulated for α=1.5, N=10⁴. The KDE method uses the Gaussian kernel for $\mathcal {K}(\cdot)$.

The sample $\widetilde {g}[\!x_{l}]$ is depicted in Fig. 2. We can see that the curve $\widetilde {g}[\!x_{l}]$ maintains linearity around zero and changes dramatically when $\widetilde {f}[\!x_{l}]$ is rather small for |x|>5. Obviously, $\widetilde {g}[\!x_{l}]$ samples outside the near-linear region are unacceptable for polynomial fitting. It demonstrates the necessity of extraction process. By ρ_th=0.8, $\widetilde {g}[\!x_{l}]$ in Ω=[−5.7,6.5] are extracted for polynomial fitting.

Fig. 2

An example of the PZMNLs in S αS noise for α=1.5, N=10⁴

Then, $\widehat A_{p}$ is calculated for various order P. Figure 2 depicts the designed PZMNL functions, as well as the ZMNL of S αS distribution $\acute {g}(x)$ by numeric simulation. It can be seen that the PZMNL $\widehat {g}(x)$ for P=10, with the breakpoints at x _ne =− 3.3 and x _po =3.6, is almost the same as $\acute {g}(x)$. The PZMNLs for P=5 and 20 are less similar to $\acute {g}(x)$. Moreover, the PZMNL for P=40 is with a narrow near-linear region and quite different from $\acute {g}(x)$. Hence, setting P around 10 is suitable.

By comparison with Fig. 1, it is clear that the AZMNL, the CZMNL, and the GZMNL functions fit $\acute {g}(x)$ worse than the PZMNLs for P=5, 10, and 20. Note that the PZMNL generally changes in another simulation which contains different noise samples, and the PZMNLs for P=40 is not always as bad as that of Fig. 2.

6.3 Detection performances in S αS noise

Detection performances by the CFAR technique are simulated. The results of detection probability in various GSNRs are depicted in Fig. 3, for α=1.5 and P _fa =10⁻³. Figure 3a shows the curves for P _D increases from P _fa to 1 while the GSNR grows from low (− 25 dB) to high (− 10 dB). The curve shapes coincide with those of conventional CFAR cases since the theoretical P _D in formula (33) has a conventional form. Considering Fig. 3a curves are too close for observation, Fig. 3b draws their details in a limited GSNR range, to provide a clear presentation for P _D comparison. Considering the optimality of various ZMNLs keeps the same in different GSNRs, the following simulations will draw the figures of details and omit the figures of whole curves.

Fig. 3

Detection probability versus the GSNR, for α=1.5, P _fa =0.001: a curves for GSNR between − 25 and − 10 dB, b details for GSNR between − 21 and − 16 dB

As can be seen Fig. 3b, the CZMNL and the GZMNL curves are similar, worse than the other ZMNL detectors. The MLD is the best, and the AZMNL and the PZMNLs for P=5,10,20 are close to the MLD. Besides, the PZMNL for P=40 is worse, as a result of less fitting to $\acute {g}(x)$. It is worth noting that all the ZMNL functions achieve significant improvement compared to the MFD.

The detection performances for various values of α are also simulated. When α grows, the GZMNL performs better while the CZMNL performs worse. The AZMNL and the PZMNLs for P=5,10,20 are near-optimal. However, when α decreases, the AZMNL and the GZMNL become worse while the CZMNL gets better. The PZMNL design approach still works in a near-optimal way. The results for α=1.0 is shown in Fig. 4, where the CZMNL and the MLD are optimal. Here, we conclude that 10≤P≤20 is suitable in the S αS noise for α∈(1,2).

Fig. 4

Detection probability versus the GSNR, for α=1.0, P _fa =0.001

6.4 Experimental results on real data

Atmospheric noise is known to possess a significant impulsive nature. This paper uses it to demonstrate the proposed PZMNL method. The raw data is recorded by a magnetotelluric sounding system at sampling frequency 512 Hz and then whitened to eliminate the power line interference. The output shows typical characteristics of impulsive noise and called as “real data.” The real data used for illustration is recorded in a sunny day at about 10:20 a.m., June 7, 2017, in Qianjiang, Hubei province, China. Its estimate of α is 1.42, near to α=1.5 of the previously simulated S αS noise. In preprocessing, the power of real data is adjusted so that the estimate of dispersion γ is 1.

The PZMNLs are designed in Monte Carlo simulations where a piece of 10⁴-length samples is chosen randomly from the set of real data. One simulation result is depicted in Fig. 5. For comparison, we also simulate “ g(x) of S αS” which assumes S αS distribution, estimates the parameters, and generates the numeric ZMNL. As shown in Fig. 5, the PZMNL design produces smooth curves which fit the ZMNL samples well around the near-linear region, while the P=40 curve appears worse. Note that g(x) of S αS noise also performs well, suggesting its effectiveness in real-data processing.

Fig. 5

The PZMNLs in real atmospheric noise for N=10⁴. g(x) of S αS is under the assumption of S αS distribution, and the estimate of α is 1.42

The detection performances of PZMNLs are depicted in Fig. 6, by the CFAR technique proposed in Section 5, at P _fa =0.001. For comparison, other detectors are also simulated, under the assumption of S αS noise. As can be seen, the PZMNLs for P=10,20 are better than the MLD. The other ZMNLs are much worse. Besides, the simulated FARs do not equal to the set FAR P _fa =0.001 since the FAR and the threshold are calculated under the assumption of S αS noise and so are incorrect theoretically. However, the FARs of the PZMNLs are much close to 0.001 because of using the estimate $\widetilde {f}[x_{l}]$ by the KDE method instead of the assumption of S αS distribution.

Fig. 6

Detection probability versus the GSNR in real atmospheric noise. Except the PZMNLs, other detectors make the assumption of S αS distribution. Though P _fa is set as 10⁻³, the simulated FARs are 10⁻³×[0.8,1.2,1.3,0.8,0.9,1.0,1.0,1.0] for the detectors from top to bottom, respectively

To achieve a constant FAR, the CFAR technique can be simulated by a numeric method as the following. The threshold η is determined based on the simulated test statistics in H₀ and adjusted to fit the desired P _fa . Then, η is used in H₁ to evaluate the detection probability P _D . Finally, the detection results are shown in Fig. 7, for GSNR = − 18 dB. We can see that the PZMNL of P=10 is the optimal, better than the MLD and the PZMNLs of P=5,20. The AZMNL and the PZMNL of P=40 are similar, while the GZMNL and CZMNL are much worse. It shows that the PZMNL of a proper order can outperform other ZMNL functions when the impulsive noise disobeys the assumed distribution.

Fig. 7

Detection probability versus P _fa in real atmospheric noise, for GSNR =− 18 dB. Except the PZMNLs, other detectors make the assumption of S αS distribution

6.5 Extended discussion

The above results demonstrate our proposed methods of LOD and PZMNL design are effective based on noise samples, no matter whether their distribution model is known. This is a significant advantage over other methods which are developed based on priori known models. In fact, impulsive noise has several different models, but there is no clear agreement or conclusion about their optimality. Once any detectors make false assumptions about the model of real noise, their performances degrade greatly. However, our methods do not have such a risk.

As our methods are proposed for impulsive noise, they may be used in other applications. Some limitations are worthy noting. Firstly, the PZMNL algorithm is developed for unimodal noise with heavy tails. Otherwise, multimodal noise has more than one near-linear regions and complicates the polynomial fitting. Secondly, Assumption 1 is necessary for the CFAR technique. It is easily satisfied in impulsive noise whose PDF is generally symmetric to zero, but may not for others. Finally, the PZMNL’s order P may vary for different distributions. It can be optimized in preprocessing and updated continuously in real time.

7 Conclusions

This paper proposes a novel approach for the LOD design in impulsive noise from unknown distributions. The ZMNL function is designed based on noise samples without assuming distribution models. As it is mainly developed by polynomial fitting, we call it as PZMNL. Simulations on α-stable noise show that the PZMNL achieves the detection performances similarly to the maximum likelihood detector which knows the noise PDF. Experimental results on real atmospheric noise demonstrate that the PZMNL outperforms other detectors which make false assumptions on the distribution models of real noise.

8 \thelikesection Appendix

8.1 \thelikesubsection Proof of Theorem 1

By the central limit theorem, as the elements of r are i.i.d., T(r) asymptotically converges to obey the Gaussian distribution for M→∞. Under hypothesis H _i , the expectation of h(r) is calculated

$$\begin{array}{*{20}l} E\left[h(r)\right]&=\int\limits_{-\infty}^{\infty}h(r)f(r-\xi_{i} s_{i})dr\\&=\int\limits_{-\infty}^{\infty}h(r)f(r)dr-\int\limits_{-\infty}^{\infty}h(r)f'(r)\xi_{i} s_{i} dr \end{array} $$

(38)

where the first-order Taylor series is used for low SNR

$$\begin{array}{*{20}l} f(r-\xi_{i} s_{i})=f(r)-\xi_{i} s_{i}f'(r). \end{array} $$

(39)

By the formulas (5), (26), and (28), we achieve

$$\begin{array}{*{20}l} E\left[h(r);H_{i} \right] \approx \xi_{i} s_{i}\mathcal{E}_{hg}. \end{array} $$

(40)

Since r[ m] is i.i.d., the expectation of T(r) is given as

$$\begin{array}{*{20}l} E\left[T(\boldsymbol{r});H_{i} \right]=\mathcal{E}_{hg}\xi_{i} \boldsymbol{s}_{i}\boldsymbol{s}_{i}^{\dag}. \end{array} $$

(41)

The covariance can be derived as

$$\begin{array}{*{20}l} \mathcal{D}\left[T(\boldsymbol{r});H_{i} \right]=\sum\limits_{m=1}^{M} \mathcal{D}\left[h({\boldsymbol{r}[m]});H_{i} \right]=\mathcal{D}\left[h({r});H_{i} \right]\boldsymbol{s}_{i}\boldsymbol{s}_{i}^{\dag}. \end{array} $$

(42)

Then, in low SNR, the expectation of h²(r) is computed as

$$\begin{array}{*{20}l} E\left[h^{2}({r});H_{i} \right]&=\int\limits_{-\infty}^{\infty}h^{2}(r)f(r-\xi_{i} s_{i})dr \\&=\int\limits_{-\infty}^{\infty}h^{2}(r)f(r)dr-\int\limits_{-\infty}^{\infty}h^{2}(r)f'(r)\xi_{i} s_{i} dr\\&\approx \mathcal{E}_{hh}. \end{array} $$

(43)

Therefore, the covariance of h(r) is given as

$$\begin{array}{*{20}l} \mathcal{D}\left[h(r);H_{i} \right]&=E\left[h^{2}(r);H_{i} \right]-\left(E\left[h(r);H_{i} \right]\right)^{2} \\&= \mathcal{E}_{hh}- (\xi_{i} s_{i}\mathcal{E}_{hg})^{2} \\&\approx \mathcal{E}_{hh} \end{array} $$

(44)

since ξ _i is rather small. Then, the covariance of T(r) is

$$\begin{array}{*{20}l} \mathcal{D}\left[T(\boldsymbol{r});H_{i} \right]= \mathcal{E}_{hh}\boldsymbol{s}_{i}\boldsymbol{s}_{i}^{\dag}. \end{array} $$

(45)

Finally, the test statistic asymptotically obeys Gaussian distribution

$$\begin{array}{*{20}l} T(\boldsymbol{r}) \,\substack{a\\\sim}\, \mathcal{N} \left(\mathcal{E}_{hg} \xi_{i} \boldsymbol{s}_{i} \boldsymbol{s}_{i}^{\dag}, \mathcal{E}_{hh}\boldsymbol{s}_{i} \boldsymbol{s}_{i}^{\dag} \right), \quad H_{i}. \end{array} $$

(46)