A variable step-size diffusion LMS algorithm with a quotient form

A new variable step-size strategy for the least mean square (LMS) algorithm is presented for distributed estimation in adaptive networks using the diffusion scheme. This approach utilizes the ratio of filtered and windowed versions of the squared instantaneous error for iteratively updating the step-size. The result is that the dependence of the update on the power of the error is reduced. The performance of the algorithm improves even though it is at the cost of added computational complexity. However, the increase in computational complexity can be minimized by careful manipulation of the update equation, resulting in an excellent performance-complexity trade-off. Complete theoretical analysis is presented for the proposed algorithm including stability, transient and steady-state analyses. Extensive experimental analysis is then done to show the performance of the proposed algorithm under various scenarios.


Introduction
The past decade has seen a lot of research in the area of distributed estimation for wireless sensor networks [1][2][3][4].The diffusion scheme using the least mean square (LMS) algorithm was first introduced by Lopes and Sayed in [2].The first algorithm to propose a variable step-size for the diffusion LMS algorithm was proposed by Saeed et al. in [5] and extended by Saeed et al. in [6].Since then, several variable step-size algorithms have been proposed [7][8][9][10][11][12].
While each algorithm improves in performance, the improvement comes at the cost of increased computational complexity.The variable step-size strategy proposed by Kwong and Johnston in [13] was extended to the distributed network scenario by Saeed et al. in [5,6].This work was further expanded by Saeed and Zerguine in [7] by using the distributed nature of the network to improve performance.A noise constraint was introduced by Saeed et al. in [8] for improved performance.An optimal step-size for the distributed scenario was derived by Ghazanfari-Rad and Labeau in [9], based on which they derived a variable step-size for each iteration.A variable step-size as well as combination weights were derived using an upper bound for the mean square deviation (MSD) by Jung et al. in [10].Lee et al. derived an individual step-size for each sensor by minimizing the local MSD in [11].Saeed et al. proposed a variable step-size strategy for distributed estimation in a compressible system in [12].
While all these algorithms provide excellent performance, they are all based on the l 2 -norm, except for the algorithm in [12].This means that the power of the instantaneous error is the key factor for all these algorithms.This means that the performance of these algorithms suffers significantly in low signal-to-noise ratio (SNR) scenarios.Furthermore, there is significant increase in computational complexity as a trade-off for improved performance.
This work proposes a method where the step-size is varied using a quotient of the filtered error power.The algorithm is inspired from the work of Zhao et al. in [14].The quotient form combined with the windowing effect reduces the effect of the power of the instantaneous error, making the algorithm more robust to low SNR scenarios.
The quotient form of [14] is combined with the diffusion scheme in this work.A theoretical analysis of the algorithm has been presented.The energy conservation approach used for the analysis in this work was not used in [14].Saeed presented a generalized analysis for variable step-size schemes using the energy conservation approach in [15].This work was extended to the diffusion scheme by Saeed et al. in [16].However, the variable step-size using the quotient form was not included in [16].The analysis presented here utilizes the approach of [16] and gives stability, transient, and steady-state analyses for the proposed algorithm.Simulation comparisons with the variable stepsize diffusion LMS (VSSDLMS) algorithm of [6] show that the proposed algorithm achieves significant improvement with a reasonably acceptable increase in computational complexity.
The rest of the paper is divided as follows.Section 2 presents the system model and defines the problem statement.Section 3 proposes the new algorithm.The theoretical analysis is given in Section 4, followed by simulation results in Section 5. Section 6 concludes this work.

System model
A geographical area of N sensor nodes, each connected to its closest neighbors as shown in Fig. 1, is being considered.The unknown parameters are modeled as a vector, w o , of size (M × 1).The input to a node at any given time instant, i, is a (1 × M) regressor vector, u k (i), where k ∈ N is the node index.The resulting observed output for the node is a noise corrupted scalar, d k (i), given by where v k (i) is the zero-mean additive noise.The generic form of the Adapt-then-Combine (ATC) variable step-size diffusion LMS (VSDLMS) algorithm is given by [16] where  2)-( 4).The new function is derived in the next section, followed by theoretical analysis and experimental results.

Proposed algorithm
The proposed update equation for the step-size is given by where α and γ are positive parameters such that 0 < α < 1 and γ > 0 and a and b are forgetting factors for the decaying exponential windows in the numerator and denominator and are bounded as 0 < a < b < 1.
Although (6) represents the complete window for both the numerator and denominator, a more suitable and iterative representation is given by Combining ( 5) and ( 9) with ( 2) and ( 3) gives the proposed variable step-size diffusion LMS algorithm with a quotient form (VSQDLMS).

Theoretical analysis
Recently, the authors of [16] proposed a unified theoretical analysis for VSDLMS algorithms.Following the same procedure as [16], the following global variables are introduced to represent the entire network: The combination weight matrix is given by {C} lk = c lk .When expanded to the entire network, this becomes G = C ⊗ I M , where ⊗ represents the Kronecker product.The unknown parameters vector w o is expanded for the entire network using w The resulting global set of equations representing the VSQDLMS algorithm are given by

Mean analysis
Introducing the global weight-error vector as Using (15), the update in ( 2) is rewritten as where the relation in ( 15) is extended for k (i).Expanding the error term, e k (i), and rearranging gives Applying the expectation operator to (17) and using the independence assumption gives, after simplification where is the autocorrelation of the input data at node k.
The algorithm will be stable in the mean sense if the term This means that the algorithm will not diverge if this term is bounded.Thus, for the algorithm to be stable in the mean, the following condition needs to be satisfied: where λ max R u,k is the maximum eigenvalue of the autocorrelation matrix R u,k .

Mean-square analysis
The mean-square analysis in [16] assumes Gaussian input data.The auto-correlation matrix is decomposed into its component eigenvector and eigenvalue matrices as R U = T T T , with being the diagonal eigenvalue matrix and T being the eigenvector matrix, such that T T T = I.
The eigenvector matrix T is then used to transform the remaining variables as follows: where and are weighting matrices.The final iterative update equation for the mean-square transient analysis is given by [16] where  where denotes the block Kronecker product, R v = v I M , v is a diagonal noise variance matrix for the network and σ = bvec .The matrix A is given in [2,16] as A = diag {A 1 , A 2 , . . ., A N }, with each component matrix defined as where k is the diagonal eigenvalue matrix and λ k is the corresponding eigenvalue vector for node k.
In order to find the mean square deviation (MSD), the weighting matrix is chosen as To get the excess mean square error (EMSE), the weighting matrix is chosen as = .

Steady-state analysis
At steady-state, (20) becomes where To get the steady-state MSD, we choose σ = bvec I N 2 M 2 , and to get the steady-state EMSE, we choose σ = bvec { }.

Steady-state step-size analysis
The step-size matrices, D ss = diag μ k,ss I M and D 2 ss = diag μ 2 k,ss I M , are defined by the individual node values for the steady-state step-size.The step-size update for the proposed algorithm is defined by ( 5), ( 7)- (9).Taking the expectation gives At steady-state, the EMSE value is assumed to go to 0, the expectation operator is removed and after simple manipulations, we get where the term ss indicates steady-state value.Thus, the steady-state step-size value is given by (36) and the value for μ 2 k,ss is simply the square of (36).

Results and discussion
This section presents experimental results for the proposed algorithm.The algorithm is tested in four different scenarios.First, the proposed algorithm is compared with the algorithm from [6] as well with the no cooperation case, in which each sensor applies the variable step-size quotient strategy for estimation without cooperating with any other sensor.The reason for the comparison with only the algorithm of [6] is that the computational complexity for both algorithms is similar.However, it is shown that with a small increase in computational complexity, the proposed algorithm performs much better.Next, the theoretical transient analysis of (20) is compared with simulation results.In the third experiment, the steady-state results from (26) are compared with steady-state simulation results.Finally, the performance of the proposed algorithm is tested in a scenario where the exact value of M is unknown.

Experiment I
In the first experiment, the proposed VSQDLMS algorithm is compared with the variable step-size diffusion LMS (VSSDLMS) algorithm from [6].A comparison of computational complexity is given in Table 1.As can  2.The values of the parameters are chosen such that the convergence speed is the same for all algorithms and the performance is measured on the basis of the steady-state MSD, as shown in the figures.Results are shown for white Gaussian input regressor data as well as colored input regressor data, with the correlation factor of 0.5.As can be seen from Figs. 2 and 3, the proposed algorithm clearly outperforms the VSSDLMS algorithm from [6].At low SNR, the performance of the proposed algorithm at 0 dB is even better than the performance of the VSSDLMS algorithm at 10 dB.The steady-state results for these experiments are given in Table 3.As can be seen, the proposed algorithm gives an improvement in performance of almost 10 dB for the case of white input and 5 dB for the case of colored input.This significant improvement in performance clearly justifies the trade-off between complexity and performance.

Experiment II
In this experiment, the performance of the proposed algorithm is tested for different network sizes with a fixed unknown vector length, M = 5.The performance of the proposed algorithm is then tested for a fixed network size,  As can be seen, the theoretical results closely match the simulation results.The mismatch during the transient stage is a result of the assumptions from [16].However, this is expected, as shown in the results of [16], and is acceptable as the two curves are closely matched at steady-state.

Experiment IV
Next, we compare the steady-state results obtained using (26) with the steady-state results obtained through There is a very close match for all results.For an exact comparison, some of these results have been tabulated in Tables 4 and 5.As can be seen, the results for the theory match closely with those obtained through simulation.

Experiment V
In the last experiment, the robustness of the proposed algorithm is tested in the case that the exact number of unknown parameters are not known.This means that the exact value for M is unknown.In such a case, there are three possible scenarios.First, an underdetermined case, where the value of M is chosen as less than the actual value.Second, an overdetermined case, where the value of M is chosen as more than the actual value.Finally, the case where the value of M is chosen correctly.For this experiment, the network size is chosen as N = 10 and the unknown parameter vector length as M = 10.The range for M is varied between 5 and 15, and the steady-state results are compared for SNR values of 0 dB, 10 dB, and 20 dB.The results are shown in Fig. 13.It can be seen that for the underdetermined case, the value of SNR does not matter, as the algorithm converges very quickly to give a large error.The best result is achieved for an exact match, as expected.However, the results for the overdetermined case are very close to the best result.As the value of M increases, the value of the steady-state error also increases.However, it is clear from these results that in case of a mismatch, it is better to have an overdetermined system rather than an underdetermined system.

Conclusion
This work proposes a new variable step-size diffusion least mean square algorithm, based on the quotient form, for distributed estimation in adaptive networks.The mean, mean-square, and steady-state analyses have presented.The proposed algorithm has been compared with the algorithm from [6], and a computational complexity analysis has been shown to justify this comparison.It has been shown that the proposed algorithm easily outperforms the algorithm from [6] and provides a much better trade-off between complexity and performance.
Results show an improvement in steady-state MSD of almost 10 dB for white input data and almost 5 dB for correlated input.Results are also shown for different network sizes as well as for different sizes of the unknown vector.A comparison between theoretical and simulation results has been shown and the two were found to be closely matched, as shown by the figures as well as the tables.Finally, the performance of the algorithm has been tested in case of a mismatch in the value of M. The proposed algorithm has been shown to perform well in a Gaussian noise environment.However, for non Gaussian environments, a variable step-size algorithm based on the least mean fourth algorithm be developed.

Fig. 1
Fig. 1 An illustration of an adaptive network of N nodes

Fig. 2
Fig.2Performance comparison for the proposed VSQDLMS vs VSSDLMS[6] algorithms for white input

Fig. 4
Fig. 4 Steady-state results for varying N with M = 5 and SNR 20 dB

Fig. 5 of 12 Fig. 6
Fig. 5 Steady-state results for varying M with N = 20 and SNR 20 dB

Table 1
Computational complexity comparison for step-size update equations

Table 2
Parameter values for Experiment 1

Table 3
Steady-state results for Figs.1 and 2

Table 4
Steady-state results for N = 5 and N = 10 simulations.The SNR values are varied from 0 dB to 30 dB.The network size is varied from N = 5 to N = 20.The results are shown in Figs. 9, 10, 11, and 12.

Table 5
Steady-state results for N = 15 and N = 20 Fig. 13 Steady-state results for a mismatch in the value of M for a network of size N = 10