Analysis of the Sign Regressor Least Mean Fourth Adaptive Algorithm

A novel algorithm, called the signed regressor least mean fourth (SRLMF) adaptive algorithm, that reduces the computational cost and complexity while maintaining good performance is presented. Expressions are derived for the steady-state excess-mean-square error (EMSE) of the SRLMF algorithm in a stationary environment. A sufficient condition for the convergence in the mean of the SRLMF algorithm is derived. Also, expressions are obtained for the tracking EMSE of the SRLMF algorithm in a nonstationary environment, and consequently an optimum value of the step-size is obtained. Moreover, the weighted variance relation has been extended in order to derive expressions for the mean-square error (MSE) and the mean-square deviation (MSD) of the proposed algorithm during the transient phase. Computer simulations are carried out to corroborate the theoretical findings. It is shown that there is a goodmatch between the theoretical and simulated results. It is also shown that the SRLMF algorithm has no performance degradation when compared with the least mean fourth (LMF) algorithm. The results in this study emphasize the usefulness of this algorithm in applications requiring reduced implementation costs for which the LMF algorithm is too complex.


Introduction
Reduction in complexity of the least mean square (LMS) algorithm has always received attention in the area of adaptive filtering [1][2][3]. This reduction is usually done by clipping either the estimation error or the input data, or both to reduce the number of multiplications necessary at each algorithm iteration. The algorithm based on clipping of the estimation error is known as the sign error or more commonly the sign algorithm (SA) [4][5][6][7][8], the algorithm based on clipping of the input data is known as the sign regressor algorithm (SRA) [9][10][11][12], and the algorithm based on clipping of both the estimation error and the input data is known as the sign sign algorithm (SSA) [13,14]. These algorithms result in a performance loss when compared with the conventional LMS algorithm [9,10]. However, significant reduction in computational cost and simplified hardware implementation can justify this poor performance in applications requiring reduced implementation costs [15,16].
The behavior of the SRA algorithm depends on the input data. It is shown in [11] that for some inputs, the LMS algorithm is stable while the SRA algorithm is unstable. This is a drawback of the SRA algorithm when compared with the SA algorithm since the latter is more stable than the LMS algorithm [4,16]. The SRA algorithm is always stable when the input data is Gaussian as in the case of speech processing. Also, the performance of the SRA algorithm is superior to that of the SA algorithm for Gaussian input data. It is shown in [10] that the SRA algorithm is much faster than the SA algorithm in achieving the desired steady-state mean-square error for white Gaussian data. Theoretical studies of the SRA algorithm with correlated Gaussian data in both stationary and nonstationary environments are found in [12].
The convergence rate and the steady-state mean-square error of the SRA algorithm is only slightly inferior to those of the LMS algorithm for the same parameter setting. In [10], the convergence rate of the SRA algorithm is compared with that of the LMS algorithm to show that the SRA algorithm converges slower than the LMS algorithm 2 EURASIP Journal on Advances in Signal Processing by a factor of 2/π for the same steady-state mean-square error.
It is shown in [17] that the SRA algorithm exhibits significantly higher robustness against the impulse noise than the LMS algorithm.
The above-mentioned advantages motivate us to analyze the proposed sign regressor least mean fourth (SRLMF) adaptive algorithm. In this paper, the mean-square analysis, the convergence analysis, the tracking analysis, and the transient analysis of the SRLMF algorithm are carried out. The framework used in this work relies on energy conservation arguments [18]. Expressions are evaluated for the steady-state excess-mean-square error (EMSE) of the SRLMF algorithm in a stationary environment. A condition for the convergence of the mean behavior of the SRLMF algorithm is also derived. Also, expressions for the tracking EMSE in a nonstationary environment are presented. An optimum value of the step-size μ is also evaluated. Moreover, an extension of the weighted variance relation is provided in order to derive expressions for the mean-square error (MSE) and the mean-square deviation (MSD) of the proposed algorithm during the transient phase. From the simulation results it is shown that both the SRLMF algorithm and the least mean fourth (LMF) algorithm [19] have a similar performance for the same steady-state EMSE. Moreover, the results show that the theoretical and simulated results are in good agreement.
The paper is organized as follows: following the Introduction is Section 2 where the proposed algorithm is developed, while the mean-square analysis of the proposed SRLMF algorithm is presented in Section 3. The convergence analysis of the proposed algorithm is presented in Section 4. Section 5 presents the tracking analysis of the proposed algorithm for random walk channels and as a by-product of this analysis the optimum value of step-size for these channels is derived. And Section 6 presents thoroughly the transient analysis of the proposed algorithm. The Computational Load is detailed in Section 7. To investigate the performance of the proposed algorithm, several simulation results for different scenarios are presented in Section 8. Finally, some conclusions are given in Section 9.

Algorithm Development
The SRLMF algorithm is based on clipping of the regression vector u i (row vector). Consider now the adaptive filter, which updates its coefficients according to the following recursion [18]: where w i (column vector) is the updated weight vector at time i, μ is the step-size, H[u i ] is some positive-definite Hermitian matrix-valued function of u i , g[e i ] denotes some function of the estimation error signal given by where d i is the desired signal. When the data is real-valued and g[e i ] = e 3 i , the general update form in (1) becomes Now if then the update form in (3) reduces to where M is the filter length. The SRLMF algorithm update recursion in (5) can be regarded as a special case of the general update form in (3) for some matrix data nonlinearity that is implicitly defined by the following relation:

Mean-Square Analysis of the SRLMF Algorithm
We wil assume that the data {d i , u i } satisfy the following conditions of the stationary data model [18,[20][21][22][23][24]. (A.4) The regressor covariance matrix is R = E[u * i u i ] > 0. For any adaptive filter of the form in (1), and for any data {d i , u i }, assuming filter operation in steady-state, the following variance relation holds [18]: Substituting (12) and (17) into (7) we get In order to simplify (18) and arrive at an expression for the steady-state EMSE ζ = E[e 2 ai ], we consider two cases.
(1) Sufficiently Small Step-Sizes. Small step-sizes lead to small values of E[e 2 ai ] and e ai in steady-state. Therefore, for smaller values of μ, the last two terms in (18) can be ignored, the steady-state EMSE is given by (2) Separation Principle. For larger values of μ, we resort to the separation assumption, namely, that at steady-state, u i 2 H is independent of e ai . In this case, the last term in (18) will be zero since e ai is zero mean, the steady-state EMSE can be shown to be

Convergence Analysis of the SRLMF Algorithm
Convergence analysis of the SRLMF algorithm is much more complicated than that of the LMS algorithm due to existence of the higher order estimation error signal in the coefficient update recursion. We thus make the following assumptions along with (A.2) to make the analysis mathematically more tractable [19][20][21][22][23][24]26]: (A.5) d i and u i are zero-mean, wide-sense stationary, and jointly Gaussian random variables.
Subtracting both sides of (5) from w o we get where w i = w o −w i . Taking expectations of both sides of (21) we obtain Using Price's theorem [25], we can conclude that Substituting (23) into (22) we get The expectation E[u T i e 3 i ] can be simplified using the fact that for zero-mean and jointly Gaussian random variables x 1 and

EURASIP Journal on Advances in Signal Processing
Thus, using (25) in conjunction with (A.5), it follows that where and from (9) Substituting (27) and (28) in (26) yields Substituting (29) into (24) we get Ultimately, it is easy to show that the mean behavior of the weight vector, that is E[w i ], converges to the optimal weight vector w o if μ is bounded by: where λ max represents the maximum eigenvalue of the regressor covariance matrix R. Notice, that there exists the time-varying function σ 2 e and the regressor variance σ 2 u in the upper bound for μ. Since σ 2 e is usually large at the beginning of adaptation processes, we can see that the convergence of the SRLMF algorithm strongly depends on the choice of initial conditions.

Tracking Analysis of the SRLMF Algorithm
Here, we assume that the data {d i , u i } satisfy the following assumptions of the nonstationary data model [18].
In this case, the following variance relation holds [18]: Tracking results can be obtained by inspection from the mean-square results as there are only minor differences. Therefore, by substituting (12) and (17) into (32), we get We again consider two cases for the evaluation of the tracking EMSE ζ of the SRLMF algorithm.
(1) Sufficiently Small Step-Sizes. Also, here, in this case we get (34) An optimum value of the step-size of the SRLMF algorithm is obtained by minimizing (34) with respect to μ. Setting the derivative of ζ with respect to μ equal to zero gives (2) Separation Principle. Similarly here as it was done for the derivation of (20), we obtain the following: and eventually the optimum step-size of the SRLMF algorithm is given by (37)

Transient Analysis of the SRLMF Algorithm
Here, we will assume that the data {d i , u i } satisfy the conditions of the stationary data model described in Section 3.

Weighted Energy-Conservation Relation
Theorem 1. For any adaptive filter of the form (1), any positive-definite Hermitian matrix Σ, and for any data {d i , u i }, it holds that [18]: Proof. Let us consider the adaptive filter updates of the generic form given in (1). Subtracting both sides of (1) from w o , we get If we multiply both sides of (39) by Two cases can be considered here.
Substituting (41) into (39), we get Expression (42) can be rearranged as Evaluating the energies of both sides of (43) results in After a straightforward calculation, the following weighted energy-conservation results: (46)

Weighted Variance Relation.
Here, the weighted variance relation presented in [18] has been extended in order to derive expressions for the MSE and the MSD of the SRLMF algorithm during the transient phase. (1), any positive-definite Hermitian matrix Σ, and for any data

Theorem 2. For any adaptive filter of the form
Similarly, for real-valued data, the above weighted variance relation becomes Proof. Squaring both sides of (40), we get Taking expectations of both sides of (52), we obtain or in the following format: For real-valued data, the weighted variance relation in (54) becomes

EURASIP Journal on Advances in Signal Processing
The transient analysis of the class of filters in (1) is more challenging due to the presence of the error nonlinearity. Nevertheless, by using some approximations, the analysis can be carried out to provide some useful insights about the performance of the SRLMF algorithm.
To start, the expectations E[ u i

HΣH g [e i ]] and E[e HΣ
ai g[e i ]] are evaluated in the ensuing analysis in terms of the weighted norm of w i−1 . Since these expectations are involved mathematically we will rely on the following assumption in order to facilitate their evaluation [18].
(A.10) The a priori estimation errors {e ai , e HΣ ai } are jointly circular Gaussian.

Evaluation of E[e HΣ
ai g[e i ]]. From Price's theorem, if x and y are jointly Gaussian random variables that are independent from a third random variable z, then it holds that [25]: Substituting (61) into (58), we get The ]] we use the separation principle, namely, we assume that the filter is long enough so that the following assumption holds [18].
(A.11) u i 2 HΣH is independent of e i . Therefore, Since e ai is Gaussian and independent of the noise, the expectation E[g 2 [e i ]] depends on e ai through its second moment only. Therefore, we can define the following function of E[e 2 ai ]: Substituting (63) and (67) into (55), we get Independence Assumption. If we assume that the regressor We thus find that studying the transient behavior of the SRLMF algorithm in effect has reduced to evaluating the functions Z 1 and Z 2 and studying the resulting variance relation (71). Let us now illustrate the application of the above results for white and correlated input data.
White Input Data. For white input data R is diagonal, say R = σ 2 u I. Therefore, if we select Σ = I, the variance relation (71) becomes Now since Substituting (73) into (66), we get Similarly by substituting (73) into (62), we get Substituting (74) and (75) into (72), we get Since E[ sign[u i ] 2 ] = M, the recursion in (76) becomes where We see that the transient behavior of the SRLMF algorithm is described by a nonlinear recursion in E[ w i 2 ] due to the presence of the factor E[ w i−1 2 ] inside f .

Correlated Input Data.
For uncorrelated data, the variance relation (72) shows that only unweighted norms of w i and w i−1 appear on both sides of the equation. However, for correlated data, different weighing matrices will appear on both sides of (72). If Σ = I in (71), we get

EURASIP Journal on Advances in Signal Processing
If Σ = R in (71), we get The term E[ w i 2 R M ] can be inferred from the prior weighting factors by expressing R M as a linear combination of its lower-order powers using the Cayley-Hamilton theorem. Thus let p(x) = det(xI − R) denote the characteristic polynomial of R, say Then we know that [18]: Using this fact, we have We can collect the above results into a compact vector notation by writing (79)-(81) as where the M × 1 vectors {W i , Y} are given by and the M × M coefficient matrix F is given by As can be seen from (86), the transient behavior of the SRLMF algorithm is described by an M-dimensional statespace recursion as opposed to one-dimensional in the white input case (72). We know that, the mean-square error is defined as and the excess mean-square error is defined as where The evolution of E[ w i 2 ] is described by the first entry of the state vector W i in (86).

Computational Load
Finally, the computational complexity of the LMF and SRLMF algorithms is discussed in this section. Tables 1 and 2 detail the estimated computational load per iteration for LMF and SRLMF algorithms, respectively, for real-and complex-valued data in terms of the number of real additions (+), real multiplications (×), and comparisons with zero (or sign evaluations). We know that one complex multiplication requires four real multiplications and two real additions, while one complex addition requires two real additions. As can be seen from these two tables, the computational complexity of the SRLMF algorithm becomes more interesting when the data is complex-valued. The case of fading channels in mobile communications is a good example where this scenario can bring drastic improvement in complexity of the SRLMF algorithm over the LMF algorithm.

Simulation Results
First, the performance analysis of the LMF and the SRLMF algorithms is investigated in an unknown system identification setup with w o = [0.227 0.460 0.688 0.460 0.227] T as far as convergence, steady-state and transient behaviors are concerned. Figure 1 depicts the convergence behavior of the two algorithms for a signal to noise ratio (SNR) of 10 dB in a uniform environment. This figure shows almost identical performance for the two algorithms; no deterioration has occurred to the SRLMF algorithm.
Second, in order to validate the theoretical findings, extensive simulations are carried out for different scenarios. While Figures 2-4 are for the case of the steady-state EMSE of the SRLMF algorithm in a stationary environment, Figure 5 is for the case of the tracking EMSE in a nonstationary  Step-size (µ)

MSE (dB)
Simulation Theory (small µ) Theory (separation) environment. In all of these figures the MSE is plotted versus the step-size μ with a SNR = 30 dB.
In the case of Figure 2, the regressors, with shift structure, are generated by feeding a unit-variance white process into a tapped delay line. However, in Figure 3, the regressors, with shift structure, are generated by passing correlated data into a tapped delay line. Here, the correlated data are obtained by passing a unit-variance i.i.d. Gaussian data through a first-order autoregressive model with transfer function Step-size (µ)

MSE (dB)
Simulation Theory (small µ) Theory (separation) Third, to further validate the theoretical results in a tracking scenario, the results of Figure 5 depicts this behavior. Here, the random-walk channel behaves according to where q i is a Gaussian sequence with zero mean and variance σ 2 q = 10 −9 and w o −1 = w o . As observed from Figure 5, the simulation results corroborate closely the theoretical results ((34) and (36)).
Finally, we examine the transient behavior of the SRLMF algorithm for the case of Gaussian data. Let us consider a real-valued regression sequence {u i } with covariance matrix Step-size (µ) Simulation Theory (small µ) Theory (separation) MSE (dB) R whose eigenvalue spread we set at ρ = 5. Let the SNR be 50 dB and the step-size is fixed at μ = 0.01. The results in Figures 6 and 7 show the theoretical and simulated MSD and MSE learning curves of the SRLMF algorithm using white Gaussian regressors and Gaussian regressors with an eigenvalue spread equal to 5. The theoretical values are obtained by using the expression (86). As can be seen here, There is an excellent match between the theoretical and simulated results.

Conclusions
A new adaptive algorithm, called the SRLMF algorithm, has been presented in this work. Expressions are derived for the steady-state EMSE in a stationary environment. A condition for the mean convergence is also found, and it turns out that the convergence of the SRLMF algorithm strongly depends on the choice of initial conditions. Also, expressions are obtained for the tracking EMSE in a nonstationary environment. An optimum value of the step-size μ is also evaluated. Moreover, an extension of the weighted variance relation is provided in order to derive expressions for the mean-square error (MSE) and the mean-square deviation (MSD) of the proposed algorithm during the transient phase. Monte Carlo simulations have shown that there is a good agreement between the theoretical and simulated results. The simulation results indicate that both the SRLMF algorithm and the LMF algorithm converge at the same rate resulting in no performance loss. The analysis developed in this paper is believed to make practical contributions to the design of adaptive filters using the SRLMF algorithm instead of the LMF algorithm in pursuit of the reduction in computational cost and complexity whilst still maintaining good performance.