- Research Article
- Open Access
Convergence Analysis of a Mixed Controlled
Adaptive Algorithm
- Abdelmalek Zidouri1Email author
https://doi.org/10.1155/2010/893809
© Abdelmalek Zidouri. 2010
- Received: 17 June 2010
- Accepted: 26 October 2010
- Published: 1 November 2010
Abstract
A newly developed adaptive scheme for system identification is proposed. The proposed algorithm is a mixture of two norms, namely, the
-norm and the
-norm (
), where a controlling parameter in the range
is used to control the mixture of the two norms. Existing algorithms based on mixed norm can be considered as a special case of the proposed algorithm. Therefore, our algorithm can be seen as a generalization to these algorithms. The derivation of the algorithm and its convexity property are reported and detailed. Also, the first moment behaviour as well as the second moment behaviour of the weights is studied. Bounds for the step size on the convergence of the proposed algorithm are derived, and the steady-state analysis is carried out. Finally, simulation results are performed and are found to corroborate with the theory developed.
Keywords
- Less Mean Square
- Less Mean Square Algorithm
- Step Size Parameter
- Uniform Noise
- Practical Instance
1. Introduction
The least mean square (LMS) algorithm [1] is one of the most widely used adaptive schemes. Several works have been presented using the LMS or its variants [2–14], such as signed LMS [8], the least mean fourth (LMF) algorithm and its variants [15], or the mixed LMS-LMF [16–18] all of which are intuitively motivated.
The LMS algorithm is optimum only if the noise statistics are Gaussian. However, if these statistics are different from Gaussian, other criteria, such as
-norm (
), perform better than the LMS algorithm. An alternative to the LMS algorithm which performs well when the noise statistics are not Gaussian is the LMF algorithm. A further improvement is possible when using a mixture of both algorithms, that is, the LMS and the LMF algorithms [16].









Adaptive filter algorithms designed through the minimization of equation (1) have a disadvantage when the absolute value of the error is greater than one. This makes the algorithm go unstable unless either a small value of the step size or a large value of the controlling parameter is chosen such that this unwanted instability is eliminated. Unfortunately, a small value of the step size will make the algorithm converge very slowly, and a large value of the controlling parameter will make the LMS algorithm essentially dominant.
The rest of the paper is organized as follows. In Section 2, the description of the proposed algorithm is addressed, while Section 3 deals with the convergence analysis. Section 4 details the derivation of the excess mean-square-error. The simulation results are reported in Section 5, and finally Section 6 concludes the main findings of the paper and outlines possible further work.
2. Proposed Algorithm




If
, the cost function defined by (4) reduces to the LMS algorithm whatever the value of
in the range
for which the unimodality of the cost function is preserved.
For
, the algorithm reduces to the
-norm adaptive algorithm, and moreover if
results in the familiar signed LMS algorithm [14].


- (1)
for
, the cost function may easily become large valued when the magnitude of the output error
, leading to a potentially considerable enhancement of noise, and
- (2)
for
, the gradient decreases in a positive direction, resulting in an obviously undesirable attribute for being used as a cost function. Setting the value of
within the range
provides a situation where the gradient at
is very much lower than that for the cases with
. This means that the resulting algorithm can be less sensitive to noise.
For
,
gives less weight for larger error and this tends to reduce the influence of aberrant noise, while it gives relatively larger weight to smaller errors and this will improve the tracking capability of the algorithm [19].
2.1. Convex Property of Cost Function
The cost function
is a convex function defined on
for
, where
and
are the dimensions of
and
, respectively.
Proof.




This shows that the cost function
is convex.
2.2. Analysis of the Error Surface
Case 1
.




Case 2
.
It can be shown as well that the error function for the feedback section will have a global minimum since the latter one is a convex function. As in the feedforward section, the adaptive process will continuously seek the bottom of the error function of the feedback section.
2.3. The Updating Scheme
where
is the trace operation of the autocorrelation matrix
.
In general, the step size is chosen small enough to ensure convergence of the iterative procedure and produce less misadjustment error.
3. Convergence Analysis

- (A1)
The input signal
is zero mean and having variance
.
- (A2)
The noise
is a zero-mean independent and identically distributed process and is independent of the input signal and having zero odd moments.
- (A3)
The step-size is small enough for the independence assumption [14] to be valid. As a consequence, the weight-error vector is independent of the input
.
While assumptions (A1-A2) can be justified in several practical instances, assumption (A3) can only be attained asymptotically. The independence assumption [14] is very common in the literature and is justified in several practical instances [21]. The assumption of small step size is not necessarily true in practice but has been commonly used to simplify the analysis [14].
During the convergence analysis of the proposed algorithm only the case of
is considered as it is carried out for the first time. Cases for
can be found, for example, in [16–18].
3.1. First Moment Behavior of the Weight Error Vector



note that in the second step of this equation the error
has been substituted.

where
is the largest eigenvalue of the autocorrelation matrix
, since in general
, and
is the minimum MSE.


(a) Discussion
Consequently, when
, the convergence for the LMS algorithm is proved.
3.2. Second Moment Behavior of the Weight Error Vector



Both of these expressions are substituted in (22) to result in its simplified form (27).
Now, denote by
and
the limiting values of
and
, respectively; then closed-form expressions for the limiting (steady-state) values of the second moment matrix and error power are derived next.






and
is the
th scalar element of the matrix
.
Two cases can be considered for the step size
so that the weight vector converges in the mean square sense.
(1) Case

(2) Case

(b) Discussion
Note that
will result in zero in the denominator of expression (41) and therefore will make
take any value in the range of positive numbers, a contradiction with the ranges of values for the step sizes of LMS and LMF algorithms. Moreover, any value for
in
will make of the step size
set by (41) less than zero, also this condition is discarded. This concludes that it is safer to use the more realistic bounds of (39) which will guarantee stability regardless of the value of
, and therefore will be considered here.
4. Derivation of the Excess Mean-Square-Error (EMSE)


Second, the following assumption is to be used in the following ensuing analysis:
(A4)The a priori estimation error
with zero-mean is independent of
.
where
.

where
and
are, respectively, the first-order and second-order derivatives of
with respect to
evaluated around
, and
denotes the third, and higher-order terms of
.
5. Simulation Results



Three different noise environments have been considered namely, Gaussian, uniform, and Laplacian. The length of the adaptive filter is the same as that of the unknown system. The learning curves are obtained by averaging 600 independent runs. Two scenarios are considered for the case of the value of
, that is,
and
. The performance measure considered here is the excess mean-square-error (EMSE).




Block diagram representation for the proposed algorithm.
Effect of α on the learning curves of the proposed algorithm in an AWGN noise environment scenario for p = 1.
Effect of α on the learning curves of the proposed algorithm in a Laplacian noise environment scenario for p = 1.
Effect of α on the learning curves of the proposed algorithm in a uniform noise environment scenario for p = 1.





Learning curves of the proposed algorithm in different noise environments scenarios for α = 0.2 and SNR of 0 dB.
Learning curves of the proposed algorithm in different noise environments scenarios for α = 0.8 and SNR of 0 dB.
Learning curves of the proposed algorithm in different noise environments scenarios for α = 0.2 and SNR of 10 dB.
Learning curves of the proposed algorithm in different noise environments scenarios for α = 0.8 and SNR of 10 dB.
Learning curves of the proposed algorithm in different noise environments scenarios for α = 0.2 and SNR of 20 dB.
Learning curves of the proposed algorithm in different noise environments scenarios for α = 0.8 and SNR of 20 dB.
In the case of an SNR of 20 dB, Figures 9 and 10 depict the results. The case of
is shown in Figure 9 while that of
is shown in Figure 10. One can see that, even though the proposed algorithm is still performing better in the uniform noise environment, as shown in Figure 9, for
, however, identical performance is obtained by the different noise environments when
as reported in Figure 10. The theoretical findings confirm these results as will be seen later.
From the above results, one can conclude that when
the proposed algorithm is biased towards the LMF algorithm, in contrast to the case when
, the proposed algorithm is biased towards the LMS algorithm.









Learning behavior of the proposed algorithm in the different noise environments scenarios for p = 4 and α = 0.2.
Learning behavior of the proposed algorithm in the different noise environments scenarios for p = 4 and α = 0.8.


Theoretical and simulation EMSE for
,
.
Gaussian | Laplacian | Uniform | ||||
---|---|---|---|---|---|---|
Theoretical | Simulation | Theoretical | Simulation | Theoretical | Simulation | |
0 dB | −16.9 | −16.85 | −9.62 | −9.82 | −22.81 | −22.6 |
10 dB | −26.02 | −26.53 | −19.33 | −19.99 | −31.64 | −31.29 |
20 dB | −44.14 | −43.93 | −40.34 | −40.55 | −45.14 | −45.43 |
Theoretical and simulation EMSE for
,
.
Gaussian | Laplacian | Uniform | ||||
---|---|---|---|---|---|---|
Theoretical | Simulation | Theoretical | Simulation | Theoretical | Simulation | |
0 dB | −22.47 | −22.6 | −14.26 | −16.02 | −27.65 | −26.59 |
10 dB | −28.7 | −28.64 | −26.41 | −26.32 | −29.15 | −29.57 |
20 dB | −39.28 | −39.87 | −39.24 | −39.58 | −39.28 | −39.92 |
6. Conclusion
A new adaptive scheme for system identification has been introduced, where a controlling parameter in the range
is used to control the mixture of the two norms. The derivation of the algorithm is worked out, and the convexity property is proved for this algorithm. Existing algorithms, for example [16–18] can be considered as a special case of the proposed algorithm. Also, the first moment behaviour as well as the second moment behaviour of the weights are studied. Bounds for the step size on the convergence of the proposed algorithm are derived. Finally, the steady-state analysis was carried out; simulation results performed for the purpose of validating theory are found to be in good agreement with the theory developed.
The proposed algorithm has been applied so far to a system identification scenario, for example, echo cancellation. As a future extension, recent work is going on the application of the proposed algorithm to mitigate the effects of intersymbol interference in a communication system.
Declarations
Acknowledgment
The author would like to acknowledge the support of King Fahd University of Petroleum and Minerals to carry out this research.
Authors’ Affiliations
References
- Widrow B, Stearns SD: Adaptive Signal Processing. Prentice-Hall, Englewood Cliffs, NJ, USA; 1985.MATHGoogle Scholar
- Sherman S: Non-mean-square error criteria. IRE Transactions on Information Theory 1958, 4(3):125-126. 10.1109/TIT.1958.1057451MathSciNetView ArticleGoogle Scholar
- Nagumo JI, Noda A: A learning method for system identification. IEEE Transactions on Automatic Control 1967, 12: 282-287.View ArticleGoogle Scholar
- Claasen TACM, Mecklenbraeuker WFG: Comparisons of the convergence of two algorithms for adaptive FIR digital filters. IEEE Transactions on Circuits and Systems 1981, 28(6):510-518. 10.1109/TCS.1981.1085011View ArticleMATHGoogle Scholar
- Gersho A: Adaptive filtering with binary reinforcement. IEEE Transactions on Information Theory 1984, 30(2):191-199. 10.1109/TIT.1984.1056890MathSciNetView ArticleMATHGoogle Scholar
- Feuer A, Weinstein E: Convergence analysis of LMS filters with uncorrelated data. IEEE Transactions on Acoustics, Speech, and Signal Processing 1985, 33(1):222-230. 10.1109/TASSP.1985.1164493View ArticleGoogle Scholar
- Bershad NJ: Behavior of the e-normalized LMS algorithm with Gaussian inputs. IEEE Transactions on Acoustics, Speech, and Signal Processing 1987, 35(5):636-644. 10.1109/TASSP.1987.1165197View ArticleGoogle Scholar
- Eweda E: Convergence of the sign algorithm for adaptive filtering with correlated data. IEEE Transactions on Information Theory 1991, 37(5):1450-1457. 10.1109/18.133267View ArticleMATHGoogle Scholar
- Douglas SC, Meng THY: Stochastic gradient adaptation under general error criteria. IEEE Transactions on Signal Processing 1994, 42(6):1335-1351. 10.1109/78.286951View ArticleGoogle Scholar
- Al-Naffouri TY, Zerguine A, Bettayeb M: A unifying view of error nonlinearities in LMS adaptation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '98), May 1998 1697-1700.Google Scholar
- Zhang H, Peng Y:
-norm based minimisation algorithm for signal parameter estimation. Electronics Letters 1999, 35(20):1704-1705. 10.1049/el:19991080View ArticleGoogle Scholar
- Siu S, Cowan CFN: Performance analysis of the lp norm back propagation algorithm for adaptive equalisation. IEE Proceedings, Part F: Radar and Signal Processing 1993, 140(1):43-47. 10.1049/ip-f-2.1993.0006Google Scholar
- Vargas RA, Burrus CS: The direct design of recursive or IIR digital filters. Proceedings of the 3rd International Symposium on Communications, Control, and Signal Processing (ISCCSP '08), March 2008 188-192.Google Scholar
- Haykin S: Adaptive Filter Theory. 4th edition. Prentice-Hall, Upper-Saddle River, NJ, USA; 2002.MATHGoogle Scholar
- Walach E, Widrow B: The least mean fourth (LMF) adaptive algorithm and its family. IEEE Transactions on Information Theory 1984, 30(2):275-283. 10.1109/TIT.1984.1056886View ArticleGoogle Scholar
- Tanrikulu O, Chambers JA: Convergence and steady-state properties of the least-mean mixed-norm (LMMN) adaptive algorithm. IEE Proceedings Vision, Image & Signal Processing 1996, 143(3):137-142. 10.1049/ip-vis:19960449View ArticleGoogle Scholar
- Zerguine A, Cowan CFN, Bettayeb M: LMS-LMF adaptive scheme for echo cancellation. Electronics Letters 1996, 32(19):1776-1778. 10.1049/el:19961202View ArticleGoogle Scholar
- Zerguine A, Cowan CFN, Bettayeb M: Adaptive echo cancellation using least mean mixed-norm algorithm. IEEE Transactions on Signal Processing 1997, 45(5):1340-1343. 10.1109/78.575705View ArticleGoogle Scholar
- Siu S, Gibson GJ, Cowan CFN: Decision feedback equalisation using neural network structures and performance comparison with standard architecture. IEE Proceedings, Part I: Communications, Speech and Vision 1990, 137(4):221-225. 10.1049/ip-i-2.1990.0031Google Scholar
- Price R: A useful theorem for non-linear devices having Gaussian inputs. IEEE Transactions on Information Theory 1958, 4: 69-72. 10.1109/TIT.1958.1057444View ArticleMATHGoogle Scholar
- Mazo JE: On the independence theory of equalizer convergence. The Bell System Technical Journal 1979, 58(5):963-993.MathSciNetView ArticleMATHGoogle Scholar
- Macchi O: Adaptive Processing: The Least Mean Squares Approach with Applications in Transmission. John Wiley & Sons, West Sussex, UK; 1995.MATHGoogle Scholar
- Sayed AH: Fundamentals of Adaptive Filtering. Wiley-Interscience, New York, NY, USA; 2003.Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.