Multichannel adaptive signal detection in space-time colored compound-gaussian autoregressive processes

Abstract

In this article, we consider the problem of adaptive detection for a multichannel signal in the presence of spatially and temporally colored compound-Gaussian disturbance. By modeling the disturbance as a multichannel autoregressive (AR) process, we first derive a parametric generalized likelihood ratio test against compound-Gaussian disturbance (CG-PGLRT) assuming that the true multichannel AR parameters are perfectly known. For the two-step GLRT design criterion, we combine the multichannel AR parameter estimation algorithm with three covariance matrix estimation strategies for compound-Gaussian environment, then obtain three adaptive CG-PGLRT detectors by replacing the ideal multichannel AR parameters with their estimates. Owing to treating the random texture components of disturbance as deterministic unknown parameters, all of the proposed detectors require no a priori knowledge about the disturbance statistics. The performance assessments are conducted by means of Monte Carlo trials. We focus on the issues of constant false alarm rate (CFAR) behavior, detection and false alarm probabilities. Numerical results show that the proposed adaptive CG-PGLRT detectors have dramatically ease the training and computational burden compared to the generalized likelihood ratio test-linear quadratic (GLRT-LQ) which is referred to as covariance matrix based detector and relies more heavily on training.

1 Introduction

In an airborne radar system, space-time adaptive processing (STAP) has been widely used in radar target detection; see [1–4] and references therein. Various well-known STAP based detectors have been extensively investigated under Gaussian assumption [4–7]. However, with the support of measured data, the Gaussian model is no longer suitable for background disturbance in many situations of practical interest. The conventional STAP detectors may suffer severe performance degradation when the disturbance is non-Gaussian. Instead, a compound-Gaussian model can successfully describe the non-Gaussian disturbance as a product of a spatially and temporally "slowly varying" texture and a locally "rapidly varying" Gaussian speckle component [8, 9]. The texture component accounts for random power variations over range cells. This model includes the so-called spherically invariant random processes (SIRPs). Working with the compound-Gaussian model, a multitude of adaptive detectors have been studied in the past few years, for instance, the generalized likelihood ratio test-linear quadratic (GLRT-LQ) which was independently derived in [10, 11], the detectors with Rao and Wald tests [12], the Bayesian optimum radar detector (BORD) [13], and so forth. Notice that all of the aforementioned STAP detectors proposed in Gaussian and compound-Gaussian environment can be considered as covariance matrix based detectors [8, 14]. Implementing these detectors involves estimating and inverting a space-time covariance matrix of the disturbance signal for each cell under test (CUT) utilizing independent and identically distributed (IID) target-free training data (or secondary data). Obviously, when the joint spatial-temporal dimension is large, the training and computational requirements will be quite onerous. Moreover, some practical situations may exacerbate the training data selection and collection problem and limit the amount of appropriate IID training data. The lack of training may lead to ill-conditioned covariance matrix estimate and significant degradation in the covariance matrix based detection procedure.

To overcome the above difficulty caused by large joint spatial-temporal dimension, the structural information about the disturbance space-time covariance matrix can be exploited. More precisely, a multichannel autoregressive (AR) process has been found to be able to model the spatial-temporal correlation of the disturbance efficiently [15–18]. In [15], based on approximating the disturbance spectrum with a multichannel AR model of low order, a parametric adaptive matched filter (PAMF) for STAP detection was presented for multichannel system in Gaussian environment. The PAMF detector, which has been proved to be equivalent to a parametric Rao detector in [16], has dramatically outperformed the conventional adaptive matched filter (AMF) [6, 7] with small training size. Also in Gaussian background, a parametric GLRT [17] and a simplified parametric GLRT [18] have been successively developed by utilizing a parametric model (multichannel AR model) in the GLR principle. Experimental results on simulated and real data show that two parametric GLRT detectors work well with limited or even no range training data [18–20]. However, under such conditions, the AMF detector and Kelly's GLRT cannot be implemented. Moreover, application of the multichannel AR model in non-stationary Gaussian clutter for STAP are investigated in [21–23].

For the corresponding problem in compound-Gaussian environment, a non-Gaussian parametric adaptive matched filter (NG-PAMF) has been derived in [24]. However, this test involves explicit knowledge of the disturbance statistics, which are not always available. Unlike the NG-PAMF, the normalized parametric adaptive matched filter (NPAMF) reported in [8, 14] requires no a priori knowledge about the disturbance statistics. This feature is rather important in real-time operation. However, Michels et al. [14] still combined the multichannel AR identification algorithm with the sample covariance matrix (SCM) even in compound-Gaussian background. The sample matrix is the maximum likelihood estimate (MLE) of the covariance matrix for Gaussian disturbance, but is no longer the MLE for compound-Gaussian disturbance. The covariance matrix estimation in compound-Gaussian environment is generally intractable. Conte et al. have advocated the use of a normalized sample covariance matrix (NSCM) in [25]. Also, considering the texture component as an unknown deterministic quantity, an approximate ML (AML) estimator has been derived by Gini et al. [26].

Motivated by the previous studies, the main purpose of this article is to derive a parametric GLRT (PGLRT) for detecting a multichannel signal in the presence of compound-Gaussian disturbance modeled as a multichannel AR process. Without any knowledge about the disturbance statistics, we resort to a suboptimal GLRT algorithm considering the texture components as unknown deterministic parameters. In further derivation, to get round the difficulty in performing the joint maximization for all the unknown parameters, a two-step GLRT design criterion is adopted in this article. We first derive the model-based parametric GLRT in compound-Gaussian environment (CG-PGLRT), which possesses the perfect knowledge about the multichannel AR parameters. We apply three covariance matrix estimation strategies, i.e., sample covariance matrix (SCM), normalized sample covariance matrix (NSCM) and approximate ML (AML) estimator, to the multichannel AR parameter estimation procedure for estimating the unknown AR coefficient matrices. Then three adaptive versions of the CG-PGLRT detector: CG-PGLRT-SCM, CG-PGLRT-NSCM, and CG-PGLRT-AML are obtained. Finally, the performance assessments are presented. Numerical results indicate that the CG-PGLRT-SCM detector has no texture CFAR property, while all of the CG-PGLRT, CG-PGLRT-NSCM, and CG-PGLRT-AML detectors ensure CFAR property with respect to the texture probability density function (PDF). Compared to the covariance matrix based detector, all of the proposed adaptive detectors can handle the training-limited case and significantly decreased the computation complexity in compound-Gaussian environment.

This article is organized as follows. The problem statement and the signal and disturbance models are presented in Section 2. The parametric GLRT detector for compound-Gaussian environment (CG-PGLRT) and adaptive CG-PGLRT detectors are derived in Section 3. Then the performance assessment of the proposed detectors is displayed in Section 4. Finally, conclusions are given in Section 5.

Notation: Vectors (matrices) are denoted by boldface lower (upper) case letters, all vectors are column ones, superscripts (•) ^T and (•) ^H denote transpose and complex conjugate transpose, respectively, $\mathcal{C}\mathcal{N}\text{(}\mu \text{,}R\text{)}$ denotes the multivariate complex Gaussian distribution with mean μ and covariance matrix R. denotes the complex number field and det{•} takes the determinant of a matrix.

2 Problem statement and signal model

2.1 Problem statement

Consider the scene that the radar transmits a coherent train of N pulses and receives the signal with a uniform linear array with J sensors. The received data collected over K range cells is organized in a J × N × K data cube. For the range cell under test (CUT), a binary hypothesis test is applied to the JN-dimensional complex baseband space-time vector of primary data $x_0={\left[x_0^T\left(0\right),x_0^T\left(1\right),\dots ,x_0^T\left(N-1\right)\right]}^T\in {ℂ}^{JN\times 1}$. Typically, x₀ contains an unwanted additive disturbance signal $d_0={\left[d_0^T\left(0\right),d_0^T\left(1\right),\dots ,d_0^T\left(N-1\right)\right]}^T\in {ℂ}^{JN\times 1}$ with unknown space-time covariance matrix $R\in {ℂ}^{JN\times JN}$ and may contain a target signal α s with deterministic but unknown complex amplitude, α, and known target space-time steering vector $s={\left[s^T\left(0\right),s^T\left(1\right),\dots ,s^T\left(N-1\right)\right]}^T\in {ℂ}^{JN\times 1}$. The space-time steering vector takes the form of the Kronecker product of the normalized spatial and temporal steering vectors. K IID complex baseband space-time vectors of target-free training data ${\left\{x_k=d_k\right\}}_{k=1}^K\in {ℂ}^{JN\times 1}$ exist for assisting the signal detection.

The detection problem at hand can be expressed as the following binary hypotheses test:

$$\left\{\begin{array}{cc}{H}_0:{\text{x}}_0=d_0,\hfill & {\text{x}}_k=d_k,k=1,\dots ,K,\hfill \\ H_1:{\text{x}}_{\text{0}}=\alpha s+{\text{d}}_{\text{0}},\hfill & {\text{x}}_k=d_k,k=1,\dots ,K,\hfill \end{array}\right.$$

(1)

where

$$\begin{array}{c}{d}_k={\left[d_k^T\left(0\right),d_k^T\left(1\right),\dots ,d_k^T\left(N-1\right)\right]}^T,\hfill \\ {\text{x}}_k={\left[{\text{x}}_k^T\left(0\right),{\text{x}}_k^T\left(1\right),\dots ,{\text{x}}_k^T\left(N-1\right)\right]}^T,\hfill \\ k=1,\dots ,K.\hfill \end{array}$$

(2)

The disturbance signals ${\left\{d_k\right\}}_{k=0}^K$ lump clutter, jamming, and thermal noise, and may be correlated in space and time.

2.2 Signal model

Herein, a multichannel AR process is applied to model the disturbance signal d _k (n) at time n, k = 0, 1, . . . , K:

$$\begin{array}{c}{\text{d}}_k\left(n\right)=-\sum _{p=1}^P{\text{A}}^H\left(p\right){\text{d}}_k\left(n-p\right)+{\epsilon }_k\left(n\right),\hfill \\ n=0,1,\dots ,N-1,\hfill \end{array}$$

(3)

where ${\left\{{\text{A}}^H\left(p\right)\right\}}_{p=1}^P$ denotes the unknown J×J coefficient matrices for AR process d _k (n) of known order P, and ε _k (n) denotes the J × 1 spatial noise driving vectors that are temporally white but spatially colored.

The non-Gaussian driving process ε _k (n) is modeled as a compound-Gaussian process. It follows that ε _k (n) can be thought of as zero-mean spherically invariant random vectors (SIRVs), i.e., they can be written in the form ${\epsilon }_k\left(n\right)=\sqrt{{\tau }_k}\cdot z_k\left(n\right),k=0,1,\dots ,K,n=0,1,\dots ,N-1$. Here the speckle z _k (n) are J × 1 complex, zero-mean, Gaussian vectors with unknown covariance matrix Q. The texture component τ _k is a positive random variable over range, but constant over time when it has long temporal coherent. The texture PDF f _τ (τ) is defined to be the characteristic PDF of the complex SIRV. Given a specific value of τ _k , we have ${\epsilon }_k\left(n\right)|{\tau }_k~\mathcal{C}\mathcal{N}\left(0,{\tau }_k\text{Q}\right)$. The covariance matrix of the driving process is E{ε _k (n)ε _k (n) ^H } = E(τ _k )Q where the mean value E(τ _k ) is also the average disturbance power.

Utilizing the multichannel parametric model allows signal whitening adopting a vector one-step linear prediction error filter (PEF), involving temporal whitening via an inverse moving-average (MA) filter followed by spatial whitening [27]. Thus, the temporally whitened versions of the steering vector, the primary data and the secondary data are obtained as following:

$$\tilde{\text{s}}\left(n\right)=\text{s}\left(n\right)+\sum _{p=1}^P{\text{A}}^H\left(p\right)\text{s}\left(n-p\right),$$

(4)

$${\tilde{\text{x}}}_0\left(n\right)={\text{x}}_0\left(n\right)+\sum _{p=1}^P{\text{A}}^H\left(p\right){\text{x}}_0\left(n-p\right),$$

(5)

$${\tilde{\text{x}}}_k\left(n\right)={\text{x}}_k\left(n\right)+\sum _{p=1}^P{\text{A}}^H\left(p\right){\text{x}}_k\left(n-p\right).$$

(6)

It is worth noting that the P th-order linear prediction coefficients ${\left\{{\text{A}}^H\left(p\right)\right\}}_{p=1}^P$ are identically equal to the AR(P) process coefficients.

The P th-order predictor error vector is given by

$${\epsilon }_k\left(n\right)={\tilde{\text{x}}}_k\left(n\right)\text{for}\text{the}\text{secondary}\text{data}$$

(7)

and

$${\epsilon }_0\left(n\right)={\tilde{\text{x}}}_0\left(n\right)-\alpha \tilde{\text{s}}\left(n\right)\text{for}\text{the}\text{primary}\text{data,}$$

(8)

respectively, where α = 0 under H₀ and α ≠ 0 under H₁. Equations (7) and (8) imply that ${\tilde{\text{x}}}_k(n)~\text{SIRV[0,}\text{Q},f_{\tau }({\tau }_k)]$ and ${\tilde{\text{x}}}_0\left(n\right)~\text{SIRV[}\alpha \tilde{\text{s}}\left(n\right),\text{Q},f_{\tau }\left({\tau }_0\right)]$, where SIRV[μ,Q,f _τ (τ)] denotes a complex SIRV obtained by sampling a SIRP with mean μ, covariance matrix Q, and a characteristic PDF f _τ (τ). As the distribution f _τ (τ) is unknown, the temporally whitened vectors ${\tilde{\text{x}}}_k\left(n\right),k=0,1,\dots ,K$ can be modeled as conditionally Gaussian with the unknown variance τ _k , i.e., ${\tilde{\text{x}}}_k\left(n\right)|{\tau }_k~\mathcal{C}\mathcal{N}\left(0,{\tau }_k\text{Q}\right)$ and ${\tilde{\text{x}}}_0\left(n\right)|{\tau }_0~\mathcal{C}\mathcal{N}\left(\alpha \tilde{\text{s}}\left(n\right),{\tau }_0\text{Q}\right)$.

For large data records, the first P values ${\left\{{\text{x}}_k\left(n\right)\right\}}_{n=0}^{P-1},k=0,1,\dots ,K$ can be ignored [28]. Thus, the joint conditional PDF or likelihood function fi(x _k (0),x _k (1), . . . , x _k (N - 1)|τ _k ;α,A^H ,Q) under hypothesis H _i , i = 0 or 1, can be suitably approximated as

$$\begin{array}{c}{f}_i({\text{x}}_k\left(P\right),{\text{x}}_k\left(P+1\right),\dots ,{\text{x}}_k\left(N-1\right)|{\text{x}}_k\left(0\right),{\text{x}}_k\left(1\right),\dots ,\hfill \\ {\text{x}}_k\left(P-1\right),{\tau }_k;\alpha ,{\text{A}}^H,\text{Q})\hfill \\ =\prod _{n=P}^{N-1}\frac{1}{{\left(\pi {\tau }_k\right)}^J\text{det}\left\{\text{Q}\right\}}\text{exp}\left\{-\frac{{\epsilon }_k^H\left(n\right){\text{Q}}^{-1}{\epsilon }_k\left(n\right)}{{\tau }_k}\right\}\hfill \end{array}$$

(9)

where ε _k (n), for k = 1, . . . , K, is a temporally whitened form of the training signals given by (6) and (7), whereas ε₀(n) is given by (4), (5), and (8) with α = 0 under H₀ and α ≠ 0 under H₁. Notice that the secondary data ${\left\{{\text{x}}_k\right\}}_{k=1}^K$ and the primary data x₀ are independent. Define $\text{X}\left(n\right)={\left[{\text{x}}_0^T\left(n\right),{\text{x}}_1^T\left(n\right),\dots ,{\text{x}}_K^T\left(n\right)\right]}^T$. The final joint conditional PDF is given by

$$\begin{array}{l}{f}_i(\text{X}(P),\text{X}(P+1),\dots ,\text{X}(N-1)|\text{X}(0),\text{X}(1),\dots ,\hfill \\ \text{X}(P-1),\tau ;\alpha ,{\text{A}}^H,\text{Q})\hfill \\ ={\displaystyle \prod _{k=0}^K{\displaystyle \prod _{n=P}^{N-1}\frac{1}{{(\pi {\tau }_k)}^J\det \left|\text{Q}\right\}}\exp \left\{-\frac{{\epsilon }_k^H(n){\text{Q}}^{-1}{\epsilon }_k(n)}{{\tau }_k}\right\}}}\hfill \\ ={\displaystyle \prod _{k=0}^K{\displaystyle \prod _{n=P}^{N-1}{f}_i({\epsilon }_k(n)|{\tau }_k;\alpha ,{\text{A}}^H,\text{Q})}}\hfill \end{array}$$

(10)

where τ = [τ₀, τ₁, . . . , τ _K ] ^T and ${\text{A}}^H=\left[{\text{A}}^H\left(1\right),{\text{A}}^H\left(2\right),\dots ,{\text{A}}^H\left(P\right)\right]\in {ℂ}^{J\times JP}$. Note that A^H (p) is expressed in terms of the Hermitian operation for notational convenience, but is not necessarily a Hermitian matrix.

3 Parametric GLRT derivation in compound-gaussian environment

In the Neyman-Pearson sense, the optimum solution for the composite hypothesis testing problem (1) is the likelihood ratio test (LRT). But for the case at hand, it cannot be implemented due to total ignorance of the signal parameter α, the multichannel AR parameters Q and A^H and the texture PDF f _τ (τ). Hence, we resort to a suboptimal GLRT algorithm where the τ _k s are modeled as unknown deterministic parameters, and perform the maximum likelihood estimation (MLE) for all the unknown parameters under each hypothesis. Unfortunately, the exact maximization with respect to the unknown parameters is rather difficult and does not exist a close-form expression. Therefore, to get round the above difficulty, the two-step GLRT design criterion is adopted. We first assume that the AR parameters Q and A^H are perfectly known, and derive the CG-PGLRT detector based on the primary data. The adaptive versions of CG-PGLRT detector are then obtained by substituting the unknown AR parameters with their estimates based on the training signals only.

3.1 CG-PGLRT detector for known AR parameters

The GLRT for the case at hand, under the assumption that Q and A^H are known as well as τ _k s are modeled as unknown deterministic parameters, is given by

$$\frac{{\text{max}}_{\alpha ,{\tau }_0}{\prod }_{n=P}^{N-1}f\left({\epsilon }_0\left(n\right)|{\tau }_0,H_1;\alpha ,{\text{A}}^H,\text{Q}\right)}{{\text{max}}_{{\tau }_0}{\prod }_{n=P}^{N-1}f\left({\epsilon }_0\left(n\right)|{\tau }_0,H_0;0,{\text{A}}^H,\text{Q}\right)}\underset{H_0}{\overset{H_1}{\gtrless }}\eta$$

(11)

where η is the threshold value to be set in order to ensure the desired probability of false alarm (P_fa). Under H₁ and H₀, the joint conditional PDFs of the primary data are

$$\begin{array}{l}{\displaystyle \prod _{n=P}^{N-1}f({\epsilon }_0(n)|{\text{τ}}_0,H_1;\alpha ,{\text{A}}^H,\text{Q})}\\ =\frac{1}{{(\pi {\text{τ}}_0)}^{(N-P)J}\det {\left\{\text{Q}\right\}}^{(N-P)}}\times \\ \exp \left\{-{\displaystyle \sum _{n=P}^{N-1}\frac{{[{\tilde{\text{x}}}_0(n)-\alpha \tilde{\text{s}}(n)]}^H{\text{Q}}^{-1}[{\tilde{\text{x}}}_0(n)-\alpha \tilde{\text{s}}(n)]}{{\tau }_0}}\right\}\end{array}$$

(12)

and

$$\begin{array}{c}\prod _{n=P}^{N-1}f\left({\epsilon }_0\left(n\right)|{\tau }_0,H_0;0,{\text{A}}^H,\text{Q}\right)\hfill \\ =\frac{1}{{\left(\pi {\tau }_0\right)}^{\left(N-P\right)J}\text{det}{\left\{\text{Q}\right\}}^{\left(N-P\right)}}\text{exp}\left\{-\sum _{n=P}^{N-1}\frac{{\tilde{\text{x}}}_0^H\left(n\right){\text{Q}}^{-1}{\tilde{\text{x}}}_0\left(n\right)}{{\tau }_0}\right\}\hfill \end{array}$$

(13)

respectively.

The ML estimates of τ₀ are easily derived, from the definition

$${\widehat{\tau }}_{0i}=\text{arg}\underset{{\tau }_{0i}}{\text{max}}\prod _{n=P}^{N-1}f\left({\epsilon }_0\left(n\right)|{\tau }_{0i},H_i;{\alpha }_i,{\text{A}}^H,\text{Q}\right),$$

i = 0 or 1,

$${\widehat{\tau }}_{0_0}=\frac{{\displaystyle {\sum }_{n=P}^{N-1}{\widetilde{\mathbf{\text{x}}}}_0^H(n){\text{Q}}^{-1}{\widetilde{\mathbf{\text{x}}}}_0(n)}}{(N-P)J}=\frac{\text{tr{}{\widetilde{\mathbf{\text{X}}}}_0^H{\mathbf{\text{Q}}}^{-1}{\widetilde{\mathbf{\text{X}}}}_0\}}{(N-P)J},$$

(14)

$$\begin{array}{c}{\widehat{\tau }}_{0_1}=\frac{{\displaystyle {\sum }_{n=P}^{N-1}{[{\tilde{x}}_0(n)-\alpha \tilde{s}(n)]}^H{Q}^{-1}[{\tilde{x}}_0(n)-\alpha }\tilde{s}(n)]}{(N-P)J}\\ =\frac{\text{tr}\{{({\tilde{X}}_0-\alpha \tilde{S})}^H{Q}^{-1}({\tilde{X}}_0-\alpha \tilde{S})\}}{(N-P)J};\end{array}$$

(15)

where

$$\widetilde{\mathbf{\text{S}}}=\left[\widetilde{\mathbf{\text{s}}}\left(P\right),\widetilde{\mathbf{\text{s}}}\left(P+1\right),\dots ,\widetilde{\mathbf{\text{s}}}\left(N-1\right)\right]\in {ℂ}^{J\times \left(N-P\right)},$$

(16)

$${\tilde{\text{x}}}_0=\left[{\tilde{\text{x}}}_0\left(P\right),{\tilde{\text{x}}}_0\left(P+1\right),\dots ,{\tilde{\text{x}}}_0\left(N-1\right)\right]\in {ℂ}^{J\times \left(N-P\right)}$$

(17)

and tr{·} denotes the trace operator.

Utilizing the two estimators in (14) and (15), the joint conditional PDFs under H₁ and H₀, respectively, are:

$$\begin{array}{l}{{\displaystyle \prod _{n=P}^{N-1}f({\epsilon }_0(n)|}{\tau }_0,H_1;\alpha ,A^H,Q)|}_{{\widehat{\tau }}_{0_1}}\\ ={\left[\frac{(N-P)J}{\pi e\text{tr}\{{({\tilde{X}}_0-\alpha \tilde{S})}^H{Q}^{-1}({\tilde{X}}_0-\alpha \tilde{S})\}\det {\left\{Q\right\}}^{\frac{1}{J}}}\right]}^{(N-P)J}\end{array}$$

(18)

$$\begin{array}{l}{{\displaystyle \prod _{n=P}^{N-1}f({\epsilon }_0(n)|}{\tau }_0,H_0;0,{\text{A}}^H,\mathbf{\text{Q}})|}_{{\widehat{\tau }}_{0_0}}\\ ={\left[\frac{(N-P)J}{\pi e\text{tr{(}{\widetilde{\mathbf{\text{X}}}}_0^H{\mathbf{\text{Q}}}^{-1}{\widetilde{\mathbf{\text{X}}}}_0\}\det {\left\{\mathbf{\text{Q}}\right\}}^{\frac{1}{J}}}\right]}^{(N-P)J}\end{array}$$

(19)

Performing the maximization of (18) with respect to α is tantamount to minimizing the following expression:

$$\begin{array}{c}\Psi (\alpha )=\text{tr}\{{({\tilde{X}}_0-\alpha \tilde{S})}^H{Q}^{-1}({\tilde{X}}_0-\alpha \tilde{S})\}\hfill \\ =\text{tr}\left\{{\tilde{X}}_0^H{Q}^{-1}{\tilde{X}}_0\right\}+\text{tr}\left\{{\tilde{S}}^H{Q}^{-1}\tilde{S}\right\}{\left|\alpha -\frac{\text{tr}\left\{{\tilde{S}}^H{Q}^{-1}{\tilde{X}}_0\right\}}{\text{tr}\left\{{\tilde{S}}^H{Q}^{-1}\tilde{S}\right\}}\right|}^2\hfill \\ -\frac{{\left|\text{tr}\left\{{\tilde{S}}^H{Q}^{-1}{\tilde{X}}_0\right\}\right|}^2}{\text{tr}\left\{{\tilde{S}}^H{Q}^{-1}\tilde{S}\right\}}.\hfill \end{array}$$

When the positive factor containing α is made to vanish, the minimum is clearly attained

$$\underset{\alpha }{\min }\Psi (\alpha )=\text{tr{}{\tilde{\text{X}}}_0^H{\text{Q}}^{-1}{\tilde{\text{X}}}_0\}-\frac{{\left|\text{tr{}{\tilde{\text{S}}}^H{\text{Q}}^{-1}{\tilde{\text{X}}}_0\}\right|}^2}{\text{tr{}{\tilde{\text{S}}}^H{\text{Q}}^{-1}\tilde{\text{S}}\}}$$

(20)

with

$$\widehat{\alpha }=\frac{\text{tr{}{\tilde{\text{S}}}^H{\text{Q}}^{-1}{\tilde{\text{X}}}_0\}}{\text{tr{}{\tilde{\text{S}}}^H{\text{Q}}^{-1}\tilde{\text{S}}\}}.$$

(21)

Substituting $\widehat{\alpha }$ as well as ${\widehat{\tau }}_{0_0}$ and ${\widehat{\tau }}_{0_1}$ into the decision rule (11), after some algebra manipulations, the detection statistic for parametric GLRT in compound-Gaussian environment (CG-PGLRT) can be recast as

$${\Lambda }_{\text{CG - PGLRT}}=-J\left(N-P\right)\text{ln}\left[1-\frac{{\left|\text{tr}\left\{{\tilde{\text{S}}}^H{\text{Q}}^{-1}{\tilde{\text{x}}}_0\right\}\right|}^2}{\text{tr}\left\{{\tilde{\text{S}}}^H{\text{Q}}^{-1}\tilde{\text{S}}\right\}\text{tr}\left\{{\tilde{\text{x}}}_0^H{\text{Q}}^{-1}{\tilde{\text{x}}}_0\right\}}\right]\underset{H_0}{\overset{H_1}{\gtrless }}{\eta }_{\text{CG - PGLRT}}$$

(22)

where η_CG-PGLRT is the appropriate modification of the original threshold in (11).

Note that the normalized parametric adaptive matched filter (NPAMF), originally developed in [8, 14] for compound-Gaussian environment, is closely related to the CG-PGLRT detector but replaces true Q and A^H with their estimates obtained from the training signals. In [8, 14, 29], several multichannel parameter estimation algorithms are considered in the NPAMF detector, including the Nuttall-Strand and the multichannel least squares methods [30]. However, we obtain herein adaptive version of the CG-PGLRT detector relying on maximum likelihood (ML) parameter estimation criterion. Specifically, our multichannel parameter estimation approach is similar to that proposed in [16, 17] for Gaussian parametric model-based STAP detectors, whereas we utilize only training signals for parameter estimation.

3.2 Adaptive CG-PGLRT detector for unknown AR parameters

To make the derived detector (22) fully adaptive, suitable estimates of the multichannel AR parameters Q and A^H must be acquired. Employing the target-free training signals to estimate the multichannel AR parameters allows to decouple from estimation of target parameters, such as complex amplitude α and the texture component τ₀. However, derivation of the MLEs of Q and A^H with K unknown textures ${\left\{{\tau }_k\right\}}_{k=1}^K$ is still a more challenging task. The similar covariance estimation problem in compound-Gaussian environment has been solved in [25, 26, 31], where three different estimation strategies are introduced, including SCM, NSCM, and AML estimators.

In our opinion, the derivation of these three covariance estimators depends upon two different clutter models, that is, the dependent interference model[31] and the independent interference model[26], respectively. The former assumes that the textures of training signals are completely correlated, i.e., τ₁ = τ₂ = ··· = τ _K , and adopts the SCM estimator. The latter, where the NSCM and AML estimators are usually applied, deals with the IID ${\left\{{\tau }_k\right\}}_{k=1}^K$ instead.

For our problem, as suggested in [26], a two-step person-by-person maximization is followed. Specifically, we first assume that the textures of the training signals are perfectly known for the realization under observation. Define the training data set ${\text{X}}_t\left(n\right)={\left[{\text{x}}_1^T\left(n\right),{\text{x}}_2^T\left(n\right),\dots ,{\text{x}}_K^T\left(n\right)\right]}^T,n=0,1,\dots ,N-1$. Conditioned on the first P values ${\left\{{\text{x}}_k\left(n\right)\right\}}_{n=0}^{P-1},k=1,2,\dots ,K$, the log-likelihood function based on training signals is proportional to

$$-K\left(N-P\right)\text{ln det}\left\{\text{Q}\right\}-J\left(N-P\right)\text{ln det}\left\{{\text{D}}_{{\tau }_t}\right\}-\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\epsilon }_k^H\left(n\right){\text{Q}}^{-1}{\epsilon }_k\left(n\right)}{{\tau }_k}$$

(23)

where ${\epsilon }_k\left(n\right)={\tilde{\text{x}}}_k\left(n\right)$ and ${\text{D}}_{{\tau }_t}\in {ℂ}^{K\times K}$ is a diagonal matrix whose diagonal entries are τ₁, . . . , T _K . We, next, use (23) to derive the MLEs of Q and A^H .

By setting to zero the derivative of (23) with respect to Q, we get the MLE of Q given τ₁ = [τ₁, . . . , τ _K ] ^T and A^H :

$$\begin{array}{ll}\hfill \tilde{Q}\left({\tau }_t,{\text{A}}^H\right)& =\frac{1}{L}\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\tilde{\text{x}}}_k\left(n\right){\tilde{\text{x}}}_k^H\left(n\right)}{{\tau }_k}\\ =\frac{1}{L}\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{\left[{\text{x}}_k\left(n\right)+{\text{A}}^H{\text{y}}_k\left(n\right)\right]{\left[{\text{x}}_k\left(n\right)+{\text{A}}^H{\text{y}}_k\left(n\right)\right]}^H}{{\tau }_k}\end{array}$$

(24)

where $y_k(n)=\left[x_k^T(n-1),\dots ,x_k^T{(n-P)}^T\right]\in {ℂ}^{JP\times 1}$ and L = (N - P) × K. Substituting the above $\widehat{\text{Q}}\left({\tau }_t,{\text{A}}^H\right)$ back in (23), we find that maximizing (23) with respect to A^H reduces to minimizing $\text{det}\left\{\widehat{\text{Q}}\left({\tau }_t,{\text{A}}^H\right)\right\}$. Therefore, the MLE of the AR coefficients matrix A^H can be obtained by minimizing $\text{det}\left\{\widehat{\text{Q}}\left({\tau }_t,{\text{A}}^H\right)\right\}$ with respect to A^H .

We further observe that [17]

$$\begin{array}{ll}\hfill L\widehat{\text{Q}}\left({\tau }_t,{\text{A}}^H\right)& ={\widehat{\text{R}}}_{xx}\left({\tau }_t\right)+{\widehat{\text{R}}}_{xy}\left({\tau }_t\right)\text{A}+{\text{A}}^H{\widehat{\text{R}}}_{yx}\left({\tau }_t\right)+{\text{A}}^H{\widehat{\text{R}}}_{yy}\left({\tau }_t\right)\text{A}\\ =\left[{\text{A}}^H+{\widehat{\text{R}}}_{yx}^H\left({\tau }_t\right){\widehat{\text{R}}}_{yy}^{-1}\left({\tau }_t\right)\right]{\widehat{\text{R}}}_{yy}\left({\tau }_t\right){\left[{\text{A}}^H+{\widehat{\text{R}}}_{yx}^H\left({\tau }_t\right){\widehat{\text{R}}}_{yy}^{-1}\left({\tau }_t\right)\right]}^H+\\ {\widehat{\text{R}}}_{xx}\left({\tau }_t\right)-{\widehat{\text{R}}}_{yx}^H\left({\tau }_t\right){\widehat{\text{R}}}_{yy}^{-1}\left({\tau }_t\right){\widehat{\text{R}}}_{yx}\left({\tau }_t\right)\end{array}$$

(25)

where the above correlation matrices conditioned on τ _t = [τ₁, . . . , τ _K ] ^T are expressed as

$${\widehat{\text{R}}}_{xx}\left({\tau }_t\right)=\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\text{x}}_k\left(n\right){\text{x}}_k^H\left(n\right)}{{\tau }_k}$$

(26)

$${\widehat{\text{R}}}_{yy}\left({\tau }_t\right)=\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\text{y}}_k\left(n\right){\text{y}}_k^H\left(n\right)}{{\tau }_k}$$

(27)

$${\widehat{\text{R}}}_{yx}\left({\tau }_t\right)=\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\text{y}}_k\left(n\right){\text{x}}_k^H\left(n\right)}{{\tau }_k}$$

(28)

and ${\widehat{\text{R}}}_{xy}\left({\tau }_t\right)={\widehat{\text{R}}}_{yx}^H\left({\tau }_t\right)$. Since ${\widehat{\text{R}}}_{yy}\left({\tau }_t\right)$ is non-negative definite and the second and third terms in (25) do not depend on A^H , it follows that [17]

$$\widehat{\text{Q}}\left({\tau }_t,{\text{A}}^H\right)\ge \widehat{\text{Q}}\left({\tau }_t,{\text{A}}^H\right){|}_{\text{A}=\widehat{\text{A}}\left({\tau }_t\right)}$$

(29)

where

$${\widehat{\text{A}}}^H\left({\tau }_t\right)=-{\widehat{\text{R}}}_{yx}^H\left({\tau }_t\right){\widehat{\text{R}}}_{yy}^{-1}\left({\tau }_t\right).$$

(30)

Inserting (30) into (25), we finally have

$$\widehat{\text{Q}}\left({\tau }_t\right)=\frac{1}{L}\left[{\widehat{\text{R}}}_{xx}\left({\tau }_t\right)-{\widehat{\text{R}}}_{yx}^H\left({\tau }_t\right){\widehat{\text{R}}}_{yy}^{-1}\left({\tau }_t\right){\widehat{\text{R}}}_{yx}\left({\tau }_t\right)\right].$$

(31)

Obviously, once the MLE of τ _t = [τ₁, . . . , τ _K ] ^T , namely ${\widehat{\tau }}_{t\text{ML}}={\left[{\widehat{\tau }}_{\text{1ML}},\dots ,{\widehat{\tau }}_{K\text{ML}}\right]}^T$, is available, the exact MLEs of A^H and Q can be readily obtained according to (30) and (31). However, so far the explicit solutions for ${\widehat{\tau }}_{t\text{ML}}$ are still inaccessible. Therefore, in the second step of the person-by-person maximization, we introduce the aforementioned two clutter models to motivate the use of the three different covariance estimation strategies.

For the dependent interference model, it is interesting that the MLE of Q with identical textures is proportional to the sample covariance matrix estimate of the temporally whitened training signals within a scalar multiplicative factor λ[26, 31]. More precisely, based on the classical sample covariance matrix (SCM) estimator ${\widehat{\text{Q}}}_{\text{SCM}}\left({\text{A}}^H\right)=\left(1/L\right){\sum }_{k=1}^K{\sum }_{n=P}^{N-1}{\tilde{\text{x}}}_k\left(n\right){\tilde{\text{x}}}_k^H\left(n\right)$, ${\widehat{\text{Q}}}_{\text{ML}}$ for the dependent interference model is given as

$$\begin{array}{ll}\hfill \widehat{\text{Q}}\left(\lambda ,{\text{A}}^H\right)& =\frac{1}{\lambda }{\widehat{\text{Q}}}_{\text{SCM}}\left({\text{A}}^H\right)\\ =\frac{1}{L}\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{\left[{\text{x}}_k\left(n\right)+{\text{A}}^H{\text{y}}_k\left(n\right)\right]{\left[{\text{x}}_k\left(n\right)+{\text{A}}^H{\text{y}}_k\left(n\right)\right]}^H}{\lambda }\end{array}$$

(32)

Likewise, following the derivation of (25), (30), and (31), the MLEs of A^H and Q for the dependent model can be straightforwardly expressed as

$${\widehat{\text{A}}}_{\text{SCM}}^H=-{\widehat{\text{R}}}_{yx}^H{\widehat{\text{R}}}_{yy}^{-1}$$

(33)

$${\widehat{\text{Q}}}_{\text{SCM}}\left(\lambda \right)=\frac{1}{\lambda }{\widehat{\text{Q}}}_{\text{SCM}}=\frac{1}{\lambda L}\left[{\widehat{\text{R}}}_{xx}-{\widehat{\text{R}}}_{yx}^H{\widehat{\text{R}}}_{yy}^{-1}{\widehat{\text{R}}}_{yx}\right]$$

(34)

where

$${\widehat{\text{R}}}_{xx}=\sum _{k=1}^K\sum _{n=P}^{N-1}{\text{x}}_k\left(n\right){\text{x}}_k^H\left(n\right)$$

(35)

$${\widehat{\text{R}}}_{yy}=\sum _{k=1}^K\sum _{n=P}^{N-1}{\text{y}}_k\left(n\right){\text{y}}_k^H\left(n\right)$$

(36)

$${\widehat{\text{R}}}_{yx}=\sum _{k=1}^K\sum _{n=P}^{N-1}{\text{y}}_k\left(n\right){\text{x}}_k^H\left(n\right)$$

(37)

Notice that an explicit solution for λ is not necessary since the AR coefficients estimate ${\widehat{\text{A}}}_{\text{SCM}}^H$ is λ-independent and the adaptive version of detector (22) only requires knowledge of ${\widehat{\text{Q}}}_{\text{SCM}}\left(\lambda \right)$ (λ) within a scale multiplicative constant.

Define ${\widehat{\tilde{\text{x}}}}_k\left(n\right)$ and $\widehat{\tilde{\text{s}}}\left(n\right)$ are the temporally whitened versions of x _k (n) and s(n), respectively, using ${\widehat{\text{A}}}_{\text{SCM}}^H$

$$\widehat{\tilde{\text{s}}}\left(n\right)=\text{s}\left(n\right)+\sum _{p=1}^P{\widehat{\text{A}}}_{\text{SCM}}^H\left(p\right)\text{s}\left(n-p\right),$$

(38)

$$\begin{array}{c}{\widehat{\tilde{\text{x}}}}_k\left(n\right)={\text{x}}_k\left(n\right)+\sum _{p=1}^P{\widehat{\text{A}}}_{\text{SCM}}^H\left(p\right){\text{x}}_k\left(n-p\right),\hfill \\ n=P,P+1,\dots ,N-1.\hfill \end{array}$$

(39)

Thus, we have $\widehat{\tilde{\text{S}}}=\left[\widehat{\tilde{\text{s}}}\left(P\right),\widehat{\tilde{\text{s}}}\left(P+1\right),\dots ,\widehat{\tilde{\text{s}}}\left(N-1\right)\right]\in {ℂ}^{J\times \left(N-P\right)}$ and ${\widehat{\tilde{\text{X}}}}_k=\left[{\widehat{\tilde{\text{x}}}}_k\left(P\right),{\widehat{\tilde{\text{x}}}}_k\left(P+1\right),\dots ,{\widehat{\tilde{\text{x}}}}_k\left(N-1\right)\right]\in {ℂ}^{J\times \left(N-P\right)}$.

Plugging ${\widehat{\text{Q}}}_{\text{SCM}},\widehat{\tilde{\text{S}}}$, and ${\widehat{\tilde{\text{X}}}}_0$ in place of Q, $\tilde{\text{S}}$, and ${\tilde{\text{X}}}_0$ into (22), respectively, leads to our adaptive CG-PGLRT-SCM detector given by

$${\Lambda }_{\text{CG - PGLRT - SCM}}=-J\left(N-P\right)\text{ln}\left[1-\frac{{\left|\text{tr}\left\{{\widehat{\tilde{\text{S}}}}^H{\widehat{\text{Q}}}_{\text{SCM}}^{-1}{\widehat{\tilde{\text{X}}}}_0\right\}\right|}^2}{\text{tr}\left\{{\widehat{\tilde{\text{S}}}}^H{\widehat{\text{Q}}}_{\text{SCM}}^{-1}\widehat{\tilde{\text{S}}}\right\}\text{tr}\left\{{\widehat{\tilde{\text{X}}}}_0^H{\widehat{\text{Q}}}_{\text{SCM}}^{-1}{\widehat{\tilde{\text{X}}}}_0\right\}}\right]\underset{H_0}{\overset{H_1}{\gtrless }}{\eta }_{\text{SCM}}$$

(40)

Actually, the dependent interference model can be considered as a simple extension of the partially-homogeneous environment under the assumption of the deterministic texture component. Recently, two different parametric Rao tests, referred to as the normalized parametric Rao (NPRao) test and the scale-invariant parametric Rao (SI-PRao) test, respectively, have been developed in [32] for the partially-homogeneous environment. The above parametric Rao tests can be candidates for multichannel parametric detection in the dependent interference model. Different to the CG-PGLRT-SCM detector, the NPRao and SI-PRao tests use both training and test signals for parameter estimation. However, the above techniques are no longer applicable to the independent interference model because of the model mismatch.

In the case of the independent interference model, the texture PDF f_τ(τ) is not always known a prior. To cope with this uncertainty, the random clutter texture component is treated as a deterministic and unknown parameter. The first candidate for the independent model is the normalized sample covariance matrix (NSCM) estimator [26, 33]. Here, we normalize each term $\left\{{\tilde{\text{x}}}_k\left(n\right){\tilde{\text{x}}}_k^H\left(n\right)\right\}$ in the sums in (24) by the data-dependent normalization factor ${\widehat{\tau }}_{k_{\text{NSCM}}}$ in place of true τ _k to obtain

$${\widehat{\text{Q}}}_{\text{NSCM}}\left({\text{A}}^H\right)=\frac{1}{L}\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\tilde{\text{x}}}_k\left(n\right){\tilde{\text{x}}}_k^H\left(n\right)}{{\widehat{\tau }}_{k_{\text{NSCM}}}},$$

(41)

where ${\widehat{\tau }}_{k_{\text{NSCM}}}=\frac{\text{tr{}{\tilde{\text{X}}}_k{\tilde{\text{X}}}_k^H\}}{J(N-P)}$ is the sample estimate of the local disturbance power in the reference range cells.

According to (30) and (31), we can also obtain ${\widehat{\text{A}}}_{\text{NSCM}}^H$ and ${\widehat{\text{Q}}}_{\text{NSCM}}$ as

$${\widehat{\text{A}}}_{\text{NSCM}}^H=-{\stackrel{⌣}{\mathbf{\text{R}}}}_{yx}^H{\stackrel{⌣}{\mathbf{\text{R}}}}_{yy}^{-1},$$

(42)

$${\widehat{\text{Q}}}_{\text{NSCM}}=\frac{1}{L}\left[{\stackrel{⌣}{\mathbf{\text{R}}}}_{xx}-{\stackrel{⌣}{\mathbf{\text{R}}}}_{yx}^H{\stackrel{⌣}{\mathbf{\text{R}}}}_{yy}^{-1}{\stackrel{⌣}{\mathbf{\text{R}}}}_{yx}\right]$$

(43)

where the covariance matrices are also normalized by ${\widehat{\tau }}_{k_{\text{NSCM}}}$

$${\stackrel{⌣}{\mathbf{\text{R}}}}_{xx}=\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\text{x}}_k\left(n\right){\text{x}}_k^H\left(n\right)}{{\widehat{\tau }}_{k_{\text{NSCM}}}},$$

(44)

$${\stackrel{⌣}{\mathbf{\text{R}}}}_{yy}=\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\text{y}}_k\left(n\right){\text{y}}_k^H\left(n\right)}{{\widehat{\tau }}_{k_{\text{NSCM}}}},$$

(45)

$${\stackrel{⌣}{\mathbf{\text{R}}}}_{yx}=\sum _{k=1}^K\sum _{n=P}^{N-1}\frac{{\text{y}}_k\left(n\right){\text{x}}_k^H\left(n\right)}{{\widehat{\tau }}_{k_{\text{NSCM}}}}.$$

(46)

From (41) and (42), it is worth observing that an interdependent relationship exists between the estimation of ${\widehat{\tau }}_{k_{\text{NSCM}}}$ and ${\widehat{\text{A}}}_{\text{NSCM}}^H$. Therefore, it is rather difficult to perform joint estimation for them. The idea pursued here is to obtain ${\widehat{\tau }}_{k_{\text{NSCM}}}$ by utilizing ${\widehat{\text{A}}}_{\text{SCM}}^H$, namely, we get

$${\widehat{\tau }}_{k_{\text{NSCM}}}=\frac{{\sum }_{n=P}^{N-1}{\widehat{\tilde{\text{x}}}}_k^H\left(n\right){\widehat{\tilde{\text{x}}}}_k\left(n\right)}{\left(N-P\right)J}=\frac{\text{tr}\left\{{\widehat{\tilde{\text{X}}}}_k{\widehat{\tilde{\text{X}}}}_k^H\right\}}{\left(N-P\right)J}.$$

(47)

In like manner, the adaptive CG-PGLRT-NSCM detector is given by

$$\begin{array}{c}{\Lambda }_{\text{CG - PGLRT - NSCM}}=\hfill \\ -J\left(N-P\right)\text{ln}\left[1=\frac{{\left|\text{tr}\left\{{\stackrel{⌣}{\widetilde{\mathbf{\text{S}}}}}^H{\widehat{\text{Q}}}_{\text{NSCM}}^{-1}{\stackrel{⌣}{\widetilde{\mathbf{\text{X}}}}}_0\right\}\right|}^2}{\text{tr}\left\{{\stackrel{⌣}{\widetilde{\mathbf{\text{S}}}}}^H{\widehat{\text{Q}}}_{\text{NSCM}}^{-1}\stackrel{⌣}{\widetilde{\mathbf{\text{S}}}}\right\}\text{tr}\left\{{\stackrel{⌣}{\widetilde{\mathbf{\text{X}}}}}_0^H{\widehat{\text{Q}}}_{\text{NSCM}}^{-1}{\stackrel{⌣}{\widetilde{\mathbf{\text{X}}}}}_0\right\}}\right]\underset{H_0}{\overset{H_1}{\gtrless }}{\eta }_{\text{NSCM}}\hfill \end{array}$$

(48)

where the matrices $\stackrel{⌣}{\widetilde{\mathbf{\text{S}}}}\in {ℂ}^{J\times \left(N-P\right)}$ and ${\stackrel{⌣}{\widetilde{\mathbf{\text{X}}}}}_0\in {ℂ}^{J\times \left(N-P\right)}$ consist of the temporally whitened versions of ${\left\{\text{x}\left(n\right)\right\}}_{n=P}^{N-1}$ and ${\left\{\text{s}\left(n\right)\right\}}_{n=P}^{N-1}$, respectively, using the coefficient estimate ${\widehat{\text{A}}}_{\text{NSCM}}^H$.

The second candidate for the independent model is the constrained approximate ML (AML) estimator, provided by the ML theory. Summarily, the constrained AML estimates of Q and A^H are derived by jointly solving iteratively the following equations:

$${\widehat{\tau }}_{k_{\text{AML}}}\left(i+1\right)=\frac{\text{tr}\left\{{\overline{\stackrel{̃}{\text{X}}}}_k^H\left({\widehat{\text{A}}}_{\text{AML}}^H\left(i\right)\right){\widehat{\text{Q}}}_{\text{AML}}^{-1}\left(i\right){\overline{\stackrel{̃}{\text{X}}}}_k\left({\widehat{\text{A}}}_{\text{AML}}^H\left(i\right)\right)\right\}}{\left(N-P\right)J},$$

(49)

$${\widehat{\text{A}}}_{\text{AML}}^H\left(i+1\right)=-{\bar{\text{R}}}_{yx}^H{\bar{\text{R}}}_{yy}^{-1}\Leftarrow {\widehat{\tau }}_{k_{\text{AML}}}\left(i+1\right),$$

(50)

$$\text{D}\left(i+1\right)=\frac{1}{L}\left[{\bar{\text{R}}}_{xx}-{\bar{\text{R}}}_{yx}^H{\bar{\text{R}}}_{yy}^{-1}{\bar{\text{R}}}_{yx}\right]\Leftarrow {\widehat{\tau }}_{k_{\text{AML}}}\left(i+1\right),$$

(51)

$${\widehat{\text{Q}}}_{\text{AML}}(i+1)=\frac{\text{tr{}\text{Q}\}}{\text{tr{}\text{D}(i+1)\}}\cdot \text{D}\text{(}i+1).$$

(52)

for i = 0, 1, 2, . . . , N_it, where N_it is the number of iterations. Equation (52) guarantees the constraint $\text{tr{}{\widehat{\text{Q}}}_{\text{AML}}(i+1)\}=\text{tr{}\text{Q}\}$ to be satisfied at each iteration. Note that the covariance matrices ${\bar{\text{R}}}_{xx},{\bar{\text{R}}}_{yy}$, and ${\bar{\text{R}}}_{yx}$ have the same structure as those in (44)-(46), the only difference being that the normalization factor ${\widehat{\tau }}_{k_{\text{NSCM}}}$ is replaced by ${\widehat{\tau }}_{k_{\text{AML}}}\left(i+1\right)$ which is refreshed at each iteration. Here, ${\widehat{\text{Q}}}_{\text{NSCM}}$ and ${\widehat{\text{A}}}_{\text{NSCM}}^H$ are used as the initialization matrix for this recursive estimator, i.e., ${\widehat{\text{Q}}}_{\text{AML}}\left(0\right)={\widehat{\text{Q}}}_{\text{NSCM}}$ and ${\widehat{\text{A}}}_{\text{AML}}^H\left(0\right)={\widehat{\text{A}}}_{\text{NSCM}}^H$.

The final estimates ${\widehat{\text{Q}}}_{\text{AML}}\left(N_{\text{it}}\right)$ and ${\widehat{\text{A}}}_{\text{AML}}^H\left(N_{\text{it}}\right)$ are denoted by ${\widehat{\text{Q}}}_{\text{AML}}$ and ${\widehat{\text{A}}}_{\text{AML}}^H$ for the sake of brevity. Utilizing the AR coefficient estimate ${\widehat{\text{A}}}_{\text{AML}}^H$, we obtain the temporally whitened data matrices $\overline{\stackrel{̃}{\text{S}}}$ and ${\overline{\stackrel{̃}{\text{x}}}}_0$ respectively. Inserting these estimates in (22), we finally come up with the adaptive CG-PGLRT-AML detector:

$$\begin{array}{c}{\Lambda }_{\text{CG - PGLRT - AML}}=\hfill \\ -J\left(N-P\right)\text{ln}\left[1-\frac{{\left|\text{tr}\left\{{\overline{\stackrel{̃}{\text{S}}}}^H{\widehat{\text{Q}}}_{\text{AML}}^{-1}{\overline{\stackrel{̃}{\text{X}}}}_0\right\}\right|}^2}{\text{tr}\left\{{\overline{\stackrel{̃}{\text{S}}}}^H{\widehat{\text{Q}}}_{\text{AML}}^{-1}\overline{\stackrel{̃}{\text{S}}}\right\}\text{tr}\left\{{\overline{\stackrel{̃}{\text{X}}}}_0^H{\widehat{\text{Q}}}_{\text{AML}}^{-1}{\overline{\stackrel{̃}{\text{X}}}}_0\right\}}\right]\underset{H_0}{\overset{H_1}{\gtrless }}{\eta }_{\text{AML}}\hfill \end{array}$$

(53)

In general, the adaptive versions of the CG-PGLRT detector can be defined as a parametric implementation of the following adaptive GLRT-LQ detector [34]

$${\Lambda }_{\text{GLRT - LQ}}=-JN\text{ln}\left[1-\frac{{\left|{\text{s}}^H{\widehat{\text{R}}}^{-1}{\text{x}}_0\right|}^2}{\left({\text{s}}^H{\widehat{\text{R}}}^{-1}\text{s}\right)\left({\text{x}}_0^H{\widehat{\text{R}}}^{-1}{\text{x}}_0\right)}\right]$$

(54)

where the original GLRT-LQ detector given in [10, 11] has been reintroduced in (54). The space-time covariance matrix estimate $\widehat{\text{R}}$ can be also obtained by exploiting the SCM, NSCM, and AML estimators, respectively. However, taking the SCM estimator as an example, we at least need K ≥ JN training signals to ensure a full-rank estimate of the JN × JN matrix R. Obviously, the large JN spatio-temporal product will impose excessive training and computational burdens to the detector.

3.3 Complexity issues

We provide the floating-point operations (flops) involved in each step of the parametric and non-parametric STAP detector with different covariance estimators in Tables 1 and 2, respectively. Only the complex multiplication and division are considered but the complex addition and subtraction are ignored.

Table 1 Complexity of the CG-PGLRT-SCM detector, CG-PGLRT-NSCM detector, and CG-PGLRT-AML detector for model order P (suppose KN >JP)

Table 2 Complexity of the GLRT-LQ detector with SCM, NSCM, and AML estimators (at least K ≥ JN)

Note that the adaptive parametric implementations differ mainly in parameter estimation, they share identical steps in signal whitening and calculating the test statistic. The CG-PGLRT-NSCM detector is slightly more complex than the CG-PGLRT-SCM detector since it requires evaluating ${\widehat{\tau }}_{k_{\text{NSCM}}}$ by utilizing ${\widehat{\text{A}}}_{\text{SCM}}^H$ for normalization. However, the CG-PGLRT-AML detector is the most complex detector among the three. Likewise, similar conclusion can be made for the adaptive versions of the GLRT-LQ detector. For a quick comparison, suppose KN > JP for the parametric detectors in Table 1. Clearly, it can be seen from Tables 1 and 2 that the parametric detectors can reduce the computational complexity of their non-parametric counterparts, especially when the spatial-temporal dimension JN is large.

4 Performance assessment

This section is devoted to the performance analysis of the proposed detectors: CG-PGLRT, CG-PGLRT-SCM, CG-PGLRT-NSCM, and CG-PGLRT-AML in terms of the probability of false alarm (P_fa) and the probability of detection (P _d ). For compound-Gaussian environment, the closed-form expressions for both P_fa as well as P _d are not available. Hence, we carry out the analysis via Monte-Carlo techniques based on 100/P_fa and 100/P _d independent trials, respectively. In order to limit the computational burden, we set the probability of false alarm P_fa = 10^-3 and the number of sensors J = 4 throughout the section.

Moreover, the following statements hold for this section:

(i) The spatial driving noise is K-distributed, which is obtained assuming that the texture component is Gamma distributed [35]

$$f_{\tau }\left(\tau \right)=\frac{1}{\Gamma \left(v\right)}{\left(\frac{v}{\mu }\right)}^v{\tau }^{v-1}\text{exp}\left(-\frac{v}{\mu }\tau \right)u\left(\tau \right)$$

(55)

where Г(·) is the gamma function, μ = E{τ} denotes the mean of the distribution, and ν is the shape parameter which provides a measure of noise spikiness. Lower ν means more spikes will appear in disturbance. Without loss of generality, we set μ = 1.

(ii) The disturbance signal is generated as a multichannel AR(2) compound-Gaussian process with the method in [36, 37]. We assume that the order P of the multichannel AR process is known. If P is unknown, it can be selected using some model order selection criterion [38]. The following correlation matrices are utilized to synthesize simulated data [39]

$$\begin{array}{c}\hfill {\left[{\text{R}}_{\text{d}}\left(l\right)\right]}_{mn}=E\left[\text{d}\left(t\right){\text{d}}^H\left(t-l\right)\right]=\rho {\rho }_s^{\left|m-n\right|}\times \\ \hfill \text{exp}\left\{-{\rho }_t{l}^2+j\left(m-n+l\right)\omega \right\},\\ \hfill m,n,=0,1,\dots ,J-1\end{array}$$

(56)

and

$${\text{R}}_{\text{d}}\left(l\right)={\text{R}}_{\text{d}}^H\left(-l\right),l=0,1,\dots .P$$

(57)

where ρ relates to the disturbance power, ρ _s decides the spatial correlation, ρ _t controls the temporal correlation, and {(m - n + l)ω} defines the phase of the correlation function (56). Here, we choose the value of ρ to satisfy tr{Q} = J.

(iii) The target steering vector sequence {s(n)} for a uniform linear array is of the form [15]

$$\text{s}\left(n\right)=\frac{1}{\sqrt{N}}{e}^{j2\pi n{f}_{td}}\text{s}\left(f_{ts}\right),n=0,1,\dots ,N-1$$

(58)

where f _ts and f _td denote the target normalized spatial and Doppler frequencies, respectively, and the target spatial steering vector s(f _ts ) is defined as $\text{s}\left(f_{ts}\right)=\left(1/\sqrt{J}\right){\left[1e^{j2\pi f_{ts}}\dots e^{j2\pi \left(J-1\right)f_{ts}}\right]}^T$. In simulation, we set f _ts = 0 and f _td = 0.25.

(iv) The SINR has been defined as

$$\text{SINR}=\frac{{\left|\alpha \right|}^2}{\mu }{\text{s}}^H{\text{R}}^{-1}\text{s}$$

(59)

For reader's convenience, we give an overview of relevant parameter settings for the simulation processes in Table 3.

Table 3 Relevant test parameters for the simulations

In Figures 1, 2, 3, 4, and 5 the CFAR properties of the proposed detectors: CG-PGLRT, CG-PGLRT-SCM, CG-PGLRT-NSCM and CG-PGLRT-AML are investigated. Figure 1 plots the detection threshold versus shape parameter ν for the proposed detectors with training size K = 2 and fixed P_fa = 10^-3. Relevant test parameters are taken from Table 3 and specified in the figures. Observe that the CG-PGLRT, CG-PGLRT-NSCM, and CG-PGLRT-AML thresholds are almost insensitive to changes in shape parameters ν. This reflects their texture CFAR behavior. However, for ν < 1, the thresholds of CG-PGLRT-SCM increases with decreasing ν (increasing disturbance spikiness). Thus, a loss of CFAR with respect to the texture PDF comes about for the CG-PGLRT-SCM detector.

Figure 1

Threshold versus shape parameter ( ν ) for fixed P _fa .

Figure 2

P _fa versus threshold for the CG-PGLRT with J = 4, N = 16, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.1, 0.5, 1, 4.5.

Figure 3

P _fa versus threshold for the CG-PGLRT-SCM with J = 4, N = 16, K = 2, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.1, 0.5, 1, 4.5.

Figure 4

P _fa versus threshold for the CG-PGLRT-NSCM with J = 4, N = 16, K = 2, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.1, 0.5, 1, 4.5.

Figure 5

P _fa versus threshold for the CG-PGLRT-AML with J = 4, N = 16, K = 2, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.1, 0.5, 1, 4.5.

Figures 2, 3, 4, and 5 depict plots of P_fa versus threshold corresponding to several shape parameter values (ν = 0.1, 0.5, 1, 4.5), for the proposed CG-PGLRT, CG-PGLRT-SCM, CG-PGLRT-NSCM, and CG-PGLRT-AML, respectively. The curves for the CG-PGLRT-SCM in Figure 3 show much higher variability compared to the CG-PGLRT, CG-PGLRT-NSCM, and CG-PGLRT-AML, which confirms its lack of CFAR with respect to the texture variations. The P_fa plots for the CG-PGLRT, CG-PGLRT-NSCM, and CG-PGLRT-AML also validate their robust texture CFAR performance.

In Figure 6, the effects of the disturbance spikiness upon the detection performance of the CG-PGLRT detector are analyzed by varying the shape parameter ν. Figure 6 displays P _d plots versus SINR for shape parameters ν = 0.1, 0.5, 1, 4.5, 10. The figure highlights that the detection performance is enhanced as ν decreases at the low SINR region. However, at the large SINR region (about > 10 dB), the CG-PGLRT detector performs worse with smaller ν. The lower the value of ν and the spikier the disturbance, for a given average disturbance power μ, at low SINR (small value of α), the signal energy that can be accumulated from the cells under test increases, hence, the detection performance is improved; at large SINR (large value of α), more spikes in disturbance may have a negative effect on the signal detection. In addition, the detector exhibits close performance for ν = 4.5 and 10, since the K distribution collapses into the Gaussian distribution when ν ≥ 4.5.

Figure 6

P _d versus SINR of CG-PGLRT for P _fa = 0.001, J = 4, N = 16, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.1, 0.5, 1, 4.5, 10.

In Section 3.2, we combined the multichannel AR parameter estimation algorithm with three covariance matrix estimation strategies: SCM, NSCM, and AML estimators, and then gave three adaptive CG-PGLRT detectors, where the true multichannel AR parameters Q and A^H are substituted with the estimated ones. Now various simulations are performed to evaluate the detection performance loss of the adaptive CG-PGLRT detectors with respect to the CG-PGLRT detector which possesses the perfect knowledge about the multichannel AR parameters Q and A^H .

In Figures 7, 8, and 9, the limited-training (K = 2) case is considered for different numbers of pulses N (N = 16, 32) and shape parameters ν = 0.1, 0.5 and 4.5. In particular, Figure 7 refers to the shape parameter ν = 0.5 whereas Figures 8 and 9 to ν = 0.1 and ν = 4.5. All the detection curves in Figures 7, 8, and 9 show that the detection performance is improved as N increases. For instance, at P _d = 0.9, the CG-PGLRT-AML is about 4 dB from the CG-PGLRT bound at N = 16; the gap reduces to about 1.5 dB at N = 32 for any value of ν in this case. This implies that we can remedy the performance degradation with respect to CG-PGLRT detector by increasing the temporal dimension N. In Figures 7 and 8, we find that the CG-PGLRT-NSCM bears almost the same performance as the CG-PGLRT-AML at N = 32 and slightly poorer than the CG-PGLRT-AML at N = 16. Figure 9 demonstrates that all the adaptive detectors: CG-PGLRT-SCM, CG-PGLRT-NSCM, and CG-PGLRT-AML perform almost the same in the homogeneous environment (ν ≥ 4.5), simply because the K distribution converges to the Gaussian distribution for increasing ν. Thus, the loss of the CG-PGLRT detector with the SCM estimator due to model mismatch decreases as ν increases. We also observe that the CG-PGLRT-AML detector outperforms the other two detectors for the heavy tailed disturbance (ν → 0). Thus, the CG-PGLRT-AML detector is the most suitable detector to implement the adaptive detection in the realistic spiky disturbance environment.

Figure 7

P _d versus SINR of CG-PGLRT, CG-PGLRT-SCM, CG-PGLRT-NSCM, and CG-PGLRT-AML for P _fa = 0.001, J = 4, K = 2, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.5.

Figure 8

P _d versus SINR of CG-PGLRT, CG-PGLRT-SCM, CG-PGLRT-NSCM and CG-PGLRT-AML for P _fa = 0.001, J = 4, K = 2, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.1.

Figure 9

P _d versus SINR of CG-PGLRT, CG-PGLRT-SCM, CG-PGLRT-NSCM and CG-PGLRT-AML for P _fa = 0.001, J = 4, K = 2, ρ _s = 0.99, ρ _t = 0.3, and ν = 4.5.

In Figures 10 and 11, we compare the performance of the proposed adaptive CG-PGLRT detectors with that of the GLRT-LQ with R_scm and with R_nscm. Figures 10 and 11 refers to the plots of P _d versus SINR of the adaptive CG-PGLRT detectors with training size K = 2, 3 and the GLRT-LQ with R_scm(R_nscm) with K = 128. The large K chosen for the GLRT-LQ with estimated R is to ensure a nonsingular estimate of R and obtain performance within 3 dB from the optimum bound. It is observed that, with the limited-training (K = 2), the performance of the CG-PGLRT-SCM is close to that of the GLRT-LQ with R_scm. And both CG-PGLRT-NSCM and CG-PGLRT-AML detectors can produce better detection performance with dramatically less training than the GLRT-LQ with R_scm. Though all the proposed adaptive CG-PGLRT detectors with K = 2 are inferior to the GLRT-LQ with R_nscm with K = 128, the performance degradation of the CG-PGLRT-NSCM and CG-PGLRT-AML detectors can be easily remedied only by increasing the training size to K = 3. Indeed, the parametric detectors are significantly less dependent on training than the covariance matrix based detector.

Figure 10

P _d versus SINR of CG-PGLRT-SCM, CG-PGLRT-NSCM, CG-PGLRT-AML and GLRT-LQ with R_scm for P _fa = 0.001, J = 4, N = 16, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.5.

Figure 11

P _d versus SINR of CG-PGLRT-SCM, CG-PGLRT-NSCM, CG-PGLRT-AML and GLRT-LQ with R_nscm for P _fa = 0.001, J = 4, N = 16, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.5.

So far the model order P of the multichannel AR process has been assumed to be known. In the following simulation example, we evaluate the performance of the proposed adaptive CG-PGLRT detectors when an model order estimation error occurs. Figure 12 depicts the performance of the proposed adaptive CG-PGLRT detectors using an overestimated model order. The curves indicate that some performance loss caused by mismatched model order P happens for N = 16 and K = 3. Since P samples have been discarded in the derivation of the proposed detectors as well as in calculating the test statistics, the detection performance is degraded as model order P is overestimated and the number of pulses N is not large enough.

Figure 12

P _d versus SINR of CG-PGLRT-SCM, CG-PGLRT-NSCM, CG-PGLRT-AML when model order P of multichannel AR process used for computing test statistic is true ( P = 2) and overestimated (assuming P = 3) for P _fa = 0.001, J = 4, N = 16, K = 3, ρ _s = 0.99, ρ _t = 0.3, and ν = 0.5.

5 Conclusions

In this article, multichannel signal detection problem in space-time colored compound-Gaussian environment is discussed. By exploiting the structural information about the disturbance space-time covariance matrix, we model the disturbance signal as a multichannel AR process to ease the training and computational burdens. Modeling the texture as an unknown deterministic parameter, we first derive the CG-PGLRT detector under the assumption that the multichannel AR parameters Q and A^H are perfectly known. For the two-step GLRT design criterion, we combine the multichannel AR parameter estimation algorithm with three covariance matrix estimation strategies, i.e., sample covariance matrix (SCM), normalized sample covariance matrix (NSCM) and approximate ML (AML) estimator, and then obtain the adaptive versions of the CG-PGLRT detector by substituting the true multichannel AR parameters with their estimates. Finally, we show the CFAR behavior and detection performances of the proposed detectors: CG-PGLRT, CG-PGLRT-SCM, CG-PGLRT-NSCM, and CG-PGLRT-AML by Monte Carlo trials.

Conclusions of this article are summarized as follows:

• The CG-PGLRT-SCM detector has no texture CFAR property, while the CG-PGLRT, CG-PGLRT-NSCM, and CG-PGLRT-AML detectors ensure texture CFAR property.

• The detection probability of the CG-PGLRT detector increases with increasing disturbance spikes (decreasing ν) at low SINR. However, at large SINR, we observe a degradation in performance as ν decreases.

• For the limited-training case, the detection performance loss of the adaptive CG-PGLRT detectors with respect to the CG-PGLRT detector can be remedied by increasing temporal dimension N.

• The CG-PGLRT-AML detector has the best detection performance in the heavy tailed disturbance (ν → 0). Thus, the CG-PGLRT-AML detector is the most suitable detector to implement the adaptive detection in the realistic spiky disturbance.

• Compared to the covariance matrix based detector, the proposed model-based adaptive detectors have significantly decreased the training requirements and the computation complexity.