Stochastic analysis of neural network modeling and identification of nonlinear memoryless MIMO systems

Abstract

Neural network (NN) approaches have been widely applied for modeling and identification of nonlinear multiple-input multiple-output (MIMO) systems. This paper proposes a stochastic analysis of a class of these NN algorithms. The class of MIMO systems considered in this paper is composed of a set of single-input nonlinearities followed by a linear combiner. The NN model consists of a set of single-input memoryless NN blocks followed by a linear combiner. A gradient descent algorithm is used for the learning process. Here we give analytical expressions for the mean squared error (MSE), explore the stationary points of the algorithm, evaluate the misadjustment error due to weight fluctuations, and derive recursions for the mean weight transient behavior during the learning process. The paper shows that in the case of independent inputs, the adaptive linear combiner identifies the linear combining matrix of the MIMO system (to within a scaling diagonal matrix) and that each NN block identifies the corresponding unknown nonlinearity to within a scale factor. The paper also investigates the particular case of linear identification of the nonlinear MIMO system. It is shown in this case that, for independent inputs, the adaptive linear combiner identifies a scaled version of the unknown linear combining matrix. The paper is supported with computer simulations which confirm the theoretical results.

Introduction

Neural network[1] approaches have been extensively used in the past few years for nonlinear MIMO system modeling, identification and control where they have shown very good performances compared to classical techniques[2–6].

If these NN approaches are to be used in real systems, it is important for the algorithm designer and the user to understand their learning behavior and performance capabilities. Several authors have analyzed NN algorithms during the last two decades which considerably helped the neural network community to better understand the mechanisms of neural networks[1, 7–15]. For example, the authors in[13] have studied a simple structure consisting of two inputs and a single neuron. The authors in[8] studied a memoryless single-input single-output (SISO) system identification model for the single neuron case. In[9] the authors proposed a stochastic analysis of gradient adaptive identification of nonlinear Wiener systems composed of a linear filter followed with a Zero-memory nonlinearity. The model was composed of a linear adaptive filter followed by an adaptive parameterized version of the nonlinearity. This study has been later generalized[16] for the analysis of stochastic gradient tracking of time-varying polynomial Wiener systems. In[12] the author analyzed NN identification of nonlinear SISO Wiener systems with memory for the case where the adaptive nonlinearity is a memoryless NN with an arbitrary number of neurons. The case of a nonlinear SISO Wiener-Hammerstein system (i.e., an adaptive filter followed by an adaptive Zero-memory NN followed by an adaptive filter) has been analyzed in[11].

This paper deals with a typical class of nonlinear MIMO systems (Figure 1) which is composed of M inputs, M memoryless nonlinearities, a linear combiner, and L outputs. This corresponds, for example, to MIMO channels used in wireless terrestrial communications[17–22], satellite communications[23, 24], amplifier modeling[25], control of nonlinear MIMO systems[6], etc. Recently, a neural network approach has been proposed to adaptively identify the overall input–output transfer function of this class of MIMO systems and to characterize each component of the system (i.e., the memoryless nonlinearities and the linear combiner)[4]. The proposed NN model is composed of a set of memoryless NN blocks followed by an adaptive linear combiner. Each part of the adaptive system aims at identifying the corresponding part in the unknown MIMO system. The algorithm has been successfully applied to system modeling, channel tracking, and fault detection.

Figure 1

Nonlinear MIMO system.

The purpose of this paper is to provide a stochastic analysis of NN modeling of this class of MIMO systems. The paper provides a general methodology that may be used to solve other problems in stochastic NN learning analysis. The methodology consists of splitting the study into simple structures, before studying the complete structure. Here, as a first step we start by analyzing a simple linear adaptive MIMO scheme (consisting of an adaptive matrix) that identifies the nonlinear MIMO system (i.e., problem of linear identification of a nonlinear MIMO system). Then we analyze a nonlinear adaptive system in which the nonlinearities are assumed to be known and frozen during the learning process, only the linear combiner is made adaptive. Finally, the complete adaptive scheme is analyzed taking into account the insights given by the analysis of the simpler structures. In our analytical approach, we derive the general formulas and recursions, which we apply to a case study that we believe is illustrative to the reader.

The paper is organized as follows. The problem statement is given in Section 2. The study of the simple structures is detailed in Section 3. Section 4 presents the analysis for the complete structure. Simulation results and illustrations are given in Section 5. Finally, conclusions and future work are given in Section 6.

Problem statement

Nonlinear MIMO system

The class of nonlinear MIMO systems discussed in this paper is presented in Figure 1. Each input x _i (n) (i = 1,…,M) is nonlinearly transformed by a memoryless nonlinearity g _i (.). The outputs of these nonlinearities are then linearly combined by an L × M matrix H = [h _ji ] (assumed in this paper to be constant). Matrix H is defined by the unknown system to be identified. For example, in wireless MIMO communication systems, M is the propagation matrix representing the channel between M transmitting antennas and L receiving antennas.

The j^th output of the MIMO system is expressed as:

$$y_j\left(n\right)=\sum _{i=1}^M{h}_{\mathit{ji}}\left(n\right)g_i\left(x_i\left(n\right)\right)+N_j\left(n\right)$$

(1)

where N _j is a white Gaussian noise with variance σ₀². Let

$\begin{array}{l}X\left(n\right)=[x_1\left(n\right)x_2\left(n\right)\dots x_M\left(n\right){]}^t,g\left(X\left(n\right)\right)=[g_1\left(x_1\left(n\right)\right)\hfill \\ g_2\left(x_2\left(n\right)\right)\dots g_M\left(x_M\left(n\right)\right){]}^t,Y\left(n\right)=[y_1\left(n\right)y_2\left(n\right)\dots y_L\left(n\right){]}^t,\hfill \\ \mathit{\text{and}}N\left(n\right)={[N_1\left(n\right)N_2\left(n\right)\dots N_L\left(n\right)]}^t\text{.}\hfill \end{array}$

The system input–output relationship can be expressed in a matrix form as:

$$Y\left(n\right)=H\times g\left(X\left(n\right)\right)+N\left(n\right)\text{.}$$

(2)

Neural Network identification structure and algorithm

The neural network (Figure 2) is composed of M blocks. Each block k has a scalar input x _k (n) (k = 1,…,M), N neurons and a scalar output. The output of the k^th block is expressed as:

$$N{N}_k\left(n\right)=\sum _{i=1}^N{c}_{\mathit{ki}}f\left(a_{\mathit{ki}}{x}_k\left(n\right)+b_{\mathit{ki}}\right),k=1,....,M$$

(3)

Figure 2

NN identification structure.

Where f is the NN activation function. a _ki , c _ki , b _ki are, respectively, the input weight, bias term, and output weight of the i^th neuron in the k^th block. The output NN _k of the k^th block is connected to the j^th output of the system through weight w_jk. The system j^th output is then expressed as:

$$s_j\left(n\right)=\sum _{k=1}^M{w}_{\mathit{jk}}N{N}_k\left(n\right),j=1,,L$$

(4)

Weights w _jk will be represented by an LxM matrix: W = [w _jk ]. Let

$\begin{array}{l}S\left(n\right)={[s_1\left(n\right)s_2\left(n\right)\dots s_L\left(n\right)]}^{t}andNN\left(n\right)=[N{N}_1\left(n\right)\hfill \\ {N{N}_2\left(n\right)\dots N{N}_M\left(n\right)]}^t\text{.}\hfill \end{array}$

Equations (4) can then be expressed in a matrix form as:

$$S\left(n\right)=W\times NN\left(n\right)\text{.}$$

(5)

For the learning process, the NN parameters are updated so that to minimize the sum of the squared errors between the unknown system outputs and the corresponding outputs of the model (Figure 3):

$${‖e\left(n\right)‖}^2=\sum _{j=1}^L{e}_j^2\left(n\right)\text{.}$$

(6)

Figure 3

Adaptive learning scheme.

$\begin{array}{l}\text{Here}{e}_j\left(n\right)=y_j\left(n\right)-s_j\left(n\right)ande\left(n\right)=[e_1\left(n\right)e_2\left(n\right)\dots \hfill \\ e_L\left(n\right)]{}^t\text{.}\hfill \end{array}$

The gradient descent recursions for weight adaptation are:

$$W\left(n+1\right)=W\left(n\right)+2\mu e\left(n\right)N{N}^t\left(n\right)$$

(7)

$$c_{\mathit{ki}}\left(n+1\right)=c_{\mathit{ki}}\left(n\right)+2\mu f\left(a_{\mathit{ki}}{x}_k\left(n\right)+b_{\mathit{ki}}\right)\sum _{l=1}^L{w}_{\mathit{lk}}{e}_l\left(n\right)$$

(8)

$$\begin{array}{l}{a}_{\mathit{ki}}\left(n+1\right)=a_{\mathit{ki}}\left(n\right)+2\mu c_{\mathit{ki}}{x}_k\left(n\right)f^{\text{'}}(a_{\mathit{ki}}{x}_k\left(n\right)\hfill \\ +b_{\mathit{ki}}){\sum }_{l=1}^L{w}_{\mathit{lk}}{e}_l\left(n\right)\hfill \end{array}$$

(9)

$$\begin{array}{l}{b}_{\mathit{ki}}\left(n+1\right)=b_{\mathit{ki}}\left(n\right)+2\mu c_{\mathit{ki}}{f}^{\text{'}}(a_{\mathit{ki}}{x}_k\left(n\right)+b_{\mathit{\text{ki}}})\hfill \\ x{\sum }_{l=1}^L{w}_{\mathit{lk}}{e}_l\left(n\right)\hfill \end{array}$$

(10)

where μ is a small positive constant and$f^{\text{'}}\left(\right)$ represents the derivative:$f^{\text{'}}\left(x\right)=\frac{\partial f\left(x\right)}{\partial x}\text{.}$

Case study

After the derivation of the general formulas, it is important that we apply them to special cases in order to get closed-form expressions of the different recursions that can be illustrated to the reader. We have chosen here a case study that we think is good to illustrate our results. In this case study, the inputs x _i (n) will be assumed uncorrelated Zero-mean Gaussian variables with variance${\sigma }_{x_i}^2$. The NN activation function will be taken as the erf function. The unknown nonlinear transfer functions are taken from a family of nonlinear functions of the form$g_i\left(x\right)={\alpha }_{i}xexp\left(\frac{-{\beta }_i{x}^2}{2}\right)$, where α _i and β _i are positive constants. These nonlinear functions are reasonable models for amplitude conversions of nonlinear high power amplifiers (HPA) used in digital communications[12, 25, 26]. Note that other nonlinear functions may be considered, however, explicit closed-form solutions of the different derivations may not be possible.

Study of simplified structures: Linear adaptation

Before analyzing the full structure, we will analyze the following simplified schemes which will help us understand the complete structure:

• 1. The adaptive system is composed of an adaptive linear combiner W (Section 3.1).

• 2. The adaptive system is composed of W and scaled versions of the unknown nonlinearities (Section 3.2).

Linear adaptive system

This section studies the linear adaptive system that tries to model the nonlinear MIMO system (Figure 4):

Figure 4

Linear adaptive system.

Mean weight behavior and Wiener solution

Since matrix W is linear, it will not be able to identify the nonlinear blocks. However, we will see that it is able to identify matrix H to within a diagonal scaling matrix if the inputs are Zero-mean and independent.

The gradient descent update of matrix W is expressed as:

$$\begin{array}{cc}W\left(n+1\right)\hfill & =W\left(n\right)+2\mu e\left(n\right)X^t\left(n\right)\hfill \\ =W\left(n\right)+2\mu \left(\right(HgX\left(n\right)+N\left(n\right)\hfill \\ -W\left(n\right)X\left(n\right)(X^t\left(n\right)\hfill \end{array}$$

(11)

Averaging both sides of (11) and using the standard LMS assumption of small μ[10], we obtain:

$$\begin{array}{cc}E\left(W\left(n+1\right)\right)\hfill & \approx E\left(W\left(n\right)\right)+2\mu (H{R}_{g\left(X\right)X}\hfill \\ -E\left(W\left(n\right)\right)R_{\mathit{XX}})\hfill \\ =E\left(W\left(n\right)\right)(I-2\mu R_{\mathit{XX}})+2\mu H{R}_{g\left(X\right)X}\hfill \end{array}$$

(12)

Where$R_{\mathit{XX}}=E\left(X{X}^t\right),R_{g\left(X\right)X}=E\left(g\left(X\right)X^t\right)\text{.}$

By setting the updating gradient term to Zero, it can be shown that this equation has a single stationary point (Wiener solution[10]) which is expressed by:

$$W=W_0=HUwhereU=R_{g\left(X\right)X}{R}_{\mathit{XX}}^{-1}$$

(13)

Following Equation (12), the mean weights can be expressed as a function of the initial condition as:

$$\begin{array}{cc}E\left(W\left(n\right)\right)\hfill & =W\left(0\right){\left(I-2\mu R_{\mathit{XX}}\right)}^n\hfill \\ +2\mu H{R}_{g\left(X\right)X}\sum _{p=0}^{n-1}{\left(I-2\mu R_{\mathit{XX}}\right)}^p\hfill \end{array}$$

(14)

If μ is sufficiently small, the first term converges to 0 and the second term converges to$H{R}_{g\left(X\right)X}{R}_{\mathit{XX}}^{-1}$.

Hence, the mean weights converge to the Wiener solution:

$$W\left(\infty \right)=W_0=H{R}_{g\left(X\right)X}{R}_{\mathit{XX}}^{-1}$$

(15)

It can be easily shown that the stability condition on μ is[10]:

$$0<\mu <\frac{1}{{\lambda }_{max}}$$

(16)

where λ_max is the largest eigenvalue of the covariance matrix R _XX .

Note that for Zero-mean independent inputs, U is a diagonal matrix:

$$\begin{array}{l}U=R_{g\left(X\right)X}{R}_{\mathit{XX}}^{-1}\hfill \\ =\left[\begin{array}{ccc}\hfill \frac{E\left[g_1\left(x_1\right)x_1\right]}{{\sigma }_{x_1}^2}\hfill & \hfill \text{0…}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{E\left[g_2\left(x_2\right)x_2\right]}{{\sigma }_{x_2}^2}\hfill & \hfill \dots \hfill \\ \hfill 0\hfill & \hfill \text{0…}\hfill & \hfill \frac{E\left[g_M\left(x_M\right)x_M\right]}{{\sigma }_{x_M}^2}\hfill \end{array}\right]\hfill \end{array}$$

(17)

In this case, the linear adaptation allows the identification of matrix W to within a scaling matrix, which depends on the nonlinearities and the input signals. As expected, the scaling matrix reduces to the identity matrix if g _k (x _k ) = x _k .

Application to the case study:

For the particular nonlinear functions given in the case study (see Section 2.3), it is easy to show:

$$\begin{array}{l}E\left(x_i\left(n\right)g_i\left(x_i\left(n\right)\right)\right)=\frac{{\alpha }_i{{\sigma }_{x_i}}^2}{{\left(1+{\beta }_i{{\sigma }_{x_i}}^2\right)}^{\frac{3}{2}}}\mathit{\text{and}}\hfill \\ E\left(g_i^2\left(x_i\left(n\right)\right)\right)=\frac{{\alpha }_i^2{{\sigma }_{x_i}}^2}{{\left(1+2{\beta }_i{{\sigma }_{x_i}}^2\right)}^{\frac{3}{2}}}\hfill \end{array}$$

(18)

The mean weight transient recursions are expressed as:

$$\begin{array}{l}E\left(w_{\mathit{jk}}\left(n+1\right)\right)=E\left(w_{\mathit{jk}}\left(n\right)\right)\left(1-2\mu {\sigma }_{x_k}^2\right)\hfill \\ +2\mu h_{\mathit{jk}}\frac{{\alpha }_k{\sigma }_{x_k}^2}{{\left(1+{\beta }_k{\sigma }_{x_k}^2\right)}^{\frac{3}{2}}}\hfill \end{array}$$

(19)

Matrix U reduces to the following diagonal matrix:

$$U=\left[\begin{array}{ccc}\hfill \frac{{\alpha }_1}{{\left(1+{\beta }_1{\sigma }_{x_1}^2\right)}^{\frac{3}{2}}}\hfill & \hfill \text{0…}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{{\alpha }_2}{{\left(1+{\beta }_2{\sigma }_{x_2}^2\right)}^{\frac{3}{2}}}\hfill & \hfill \dots \hfill \\ \hfill 0\hfill & \hfill \text{0…}\hfill & \hfill \frac{{\alpha }_M}{{\left(1+{\beta }_M{\sigma }_{x_M}^2\right)}^{\frac{3}{2}}}\hfill \end{array}\right]$$

(20)

Transient MSE and Wiener MSE

The transient MSE is determined by:

$$\begin{array}{ll}E\left({‖e\left(n\right)‖}^2\right)\hfill & =E\left({‖Hg\left(X\left(n\right)\right)+N\left(n\right)-W\left(n\right)X\left(n\right)‖}^2\right)\hfill \\ =\sum _{j=1}^{L}E\left[e_j^2\left(n\right)\right]\hfill \end{array}$$

(21)

where:

$$E\left(e_j^2\left(n\right)\right)=E\left({‖H_{j}g\left(X\left(n\right)\right)+N_j\left(n\right)-W_j\left(n\right)X\left(n\right)‖}^2\right)$$

(22)

$\begin{array}{l}\text{where}{W}_j\left(n\right)={[w_{j1}\left(n\right)w_{j2}\left(n\right)\dots w_{\mathit{jM}}\left(n\right)]}^{t}and{H}_j=\hfill \\ {[h_{j1}{h}_{j2}\dots h_{\mathit{jM}}]}^t\text{.}\hfill \end{array}$

Using the independence of noise and weights at time n, we get:

$$\begin{array}{l}E\left(e_j^2\left(n\right)\right)={\sigma }_0^2+E\left({‖\left(H_{j}g\left(X\left(n\right)\right)-W_j\left(n\right)X\left(n\right)\right.‖}^2\right)\hfill \\ ={\sigma }_0^2+H_j^t{R}_{g\left(X\right)g\left(X\right)}{H}_j-2H_j^t{R}_{g\left(X\right)X}E\left(W_j\left(n\right)\right)\hfill \\ +E\left(W_j\left(n\right)R_{\mathit{XX}}{W}_j^t\left(n\right)\right)\hfill \end{array}$$

(23)

The total MSE is therefore expressed as:

$$\begin{array}{l}E\left({‖e\left(n\right)‖}^2\right)=L{\sigma }_0^2+\sum _{j=1}^L{H}_j^t{R}_{g\left(X\right)g\left(X\right)}{H}_j\hfill \\ -2H_j^t{R}_{g\left(X\right)X}E\left(W_j\left(n\right)\right)\hfill \\ +E\left(W_j^t\left(n\right)R_{\mathit{XX}}{W}_j\left(n\right)\right)\hfill \end{array}\text{.}$$

(24)

Wiener MSE:

The Wiener MSE,${\zeta }_0=E\left({‖e_{W_0}\left(n\right)‖}^2\right)$, is the minimum MSE that can be reached by the system if W is equal to the Wiener solution W₀ = HU. It can be easily shown that:

$$\begin{array}{l}{\zeta }_0=E\left({‖e_{W_0}\left(n\right)‖}^2\right)\hfill \\ =L{\sigma }_0^2+E\left({‖Hg\left(X\left(n\right)\right)-W_{0}X\left(n\right)‖}^2\right)\hfill \\ =L{\sigma }_0^2+E\left({‖H\left(g\left(X\left(n\right)\right)-UX\left(n\right)\right)‖}^2\right)\hfill \end{array}$$

(25)

It is clear from this equation that if the unknown functions are linear, then the Wiener MSE reduces to the noise power. The MSE is always larger than ζ₀ because of the misadjustment error introduced by the weight fluctuations.

Now we can write the MSE as a function of the Wiener MSE:

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & =E\left({‖H\left(g\left(X\left(n\right)\right)+N\left(n\right)-W\left(n\right)X\left(n\right)\right)‖}^2\right)\hfill \\ =E\left({‖e_{W_0}\left(n\right)-\left(W\left(n\right)-W_0\right)X\left(n\right)]‖}^2\right)\hfill \end{array}$$

(26)

Let the instantaneous deviation of the matrix weights with respect to the Wiener solution be denoted by:

$$V\left(n\right)=\left[v_{\mathit{jk}}\left(n\right)\right]=W\left(n\right)-W_0\text{.}$$

(27)

We have:

$$E\left({‖e\left(n\right)‖}^2\right)=E\left({‖e_{W_0}\left(n\right)-V\left(n\right)X\left(n\right)‖}^2\right)\text{.}$$

(28)

This expression is similar to that of the well-known LMS algorithm[10], and can be evaluated as the sum of the minimum error and excess error (or misadjustment) as:

$$E\left({‖e\left(n\right)‖}^2\right)={\zeta }_0+\sum _{j=1}^{L}tr\left(R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\right)$$

(29)

where$V_j\left(n\right)={\left[v_{j1}{v}_{j2}\dots v_{\mathit{jM}}\right]}^{t}and{K}_{V_j{V}_j}\left(n\right)=E\left(V_j\left(n\right)V_j^t\left(n\right)\right)$.

The misadjustment is expressed as:

$$\Delta \left(n\right)=tr\left(R_{\mathit{XX}}\sum _{j=1}^L{R}_{V_j{V}_j}\left(n\right)\right)\text{.}$$

(30)

At the convergence, we have:

$$E\left({‖e\left(\infty \right)‖}^2\right)={\zeta }_0+\Delta \left(\infty \right)\text{.}$$

(31)

Derivation of the misadjustment:

From Equation (11) it is easy to show that the weight fluctuations follow the recursion:

$$V\left(n+1\right)=V\left(n\right)+2\mu \left(e_{W_0}\left(n\right)-V\left(n\right)X\left(n\right)\right)X^t\left(n\right)$$

(32)

Taking the mean of this equation and applying the orthogonality principle between the input vector and the Wiener error, we get:

$$E\left(V\left(n+1\right)\right)=E\left(V\left(n\right)\right)\left(1-2\mu R_{\mathit{XX}}\right)$$

(33)

Thus, as expected, if μ is sufficiently small E( V (n)) converges to 0.

Similarly, for each vector V _j we can obtain the following recursion:

$$V_j\left(n+1\right)=V_j\left(n\right)+2\mu \left(e_{W_{0}j}\left(n\right)-X^t\left(n\right)V_j\left(n\right)\right)X\left(n\right))$$

(34)

The evaluation of the covariance matrix of the weight fluctuations is obtained by multiplying both sides of Equation (34) by$V_j^t\left(n+1\right)$ and averaging:

$$\begin{array}{cc}{K}_{V_j{V}_j^t}\left(n+1\right)\hfill & =K_{V_j^t{V}_j^t}\left(n\right)-2\mu R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\hfill \\ -2\mu K_{V_j{V}_j}\left(n\right)R_{\mathit{XX}}\hfill \\ +2\mu E\left[e_{W_{0}j}\left(n\right)X{V}_j^t\left(I-2\mu X{X}^t\right)\right]\hfill \\ +2\mu E{\left[e_{W_{0}j}\left(n\right)X{V}_j^t\left(I-2\mu X{X}^t\right)\right]}^t\hfill \\ +4{\mu }^{2}E\left[X{X}^t{K}_{V_j{V}_j}X{X}^t\right]\hfill \\ +4{\mu }^{2}E\left[e_{W_{0}j}^2\left(n\right)X{X}^t\right]\hfill \end{array}$$

(35)

These expectations are derived in Appendix III, which yields:

$$\begin{array}{cc}{K}_{V_j{V}_j^t}\left(n+1\right)\hfill & =K_{V_j^t{V}_j^t}\left(n\right)-2\mu R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\hfill \\ -2\mu K_{V_j{V}_j}\left(n\right)R_{\mathit{XX}}\hfill \\ +4{\mu }^2\left(-E\left[H_j^{t}g\left(X\right)XE\left(V_j^t\left(n\right)\right)X{X}^t\right]\right.\hfill \\ +tr\left(R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)\right)R_{\mathit{XX}}\hfill \\ \left.\left.+R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)R_{\mathit{XX}}\right)\right)\hfill \\ +4{\mu }^2\left(-E\left[H_j^{t}g\left(X\right)XE\left(V_j^t\left(n\right)\right)X{X}^t\right]\right.\hfill \\ +tr\left(R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)\right)R_{\mathit{XX}}\hfill \\ {\left.+R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)R_{\mathit{XX}}\right))}^t\hfill \\ +4{\mu }^2\left(tr\left(R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\right)R_{\mathit{XX}}\right.\hfill \\ \left.+2R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)R_{\mathit{XX}}\right)\hfill \\ +4{\mu }^2\left(E\left[g\left(X\right)g^t\left(X\right)H_j{H}_j^{t}X{X}^t\right]\right.\hfill \\ +{\sigma }_0^2{R}_{\mathit{XX}}-tr\left(R_{\mathit{XX}}{W}_{0j}^t{W}_{0j}^t\right)R_{\mathit{XX}}\hfill \\ \left.-2R_{\mathit{XX}}{W}_{0j}^t{W}_{0j}^t{R}_{\mathit{XX}}\right)\hfill \end{array}$$

(36)

Taking into account that E(V_j(∞)) = 0, K _VjVj can be obtained by solving the following equation:

$$\begin{array}{cc}{R}_{\mathit{XX}}\hfill & K_{V_j{V}_j}\left(\infty \right)+K_{V_j{V}_j}\left(\infty \right)R_{\mathit{XX}}-2\mu tr\left(R_{\mathit{XX}}{K}_{V_j{V}_j}\left(\infty \right)\right)R_{\mathit{XX}}\hfill \\ -4\mu R_{\mathit{XX}}{K}_{V_j{V}_j}\left(\infty \right)R_{\mathit{XX}}\hfill \\ =2\mu \left[E\left[g\left(X\right)g^t\left(X\right)H_j{H}_j^{t}X{X}^t\right]+2\mu {\sigma }_0^2{R}_{\mathit{XX}}\right.\hfill \\ \left.-2\mu tr\left(R_{\mathit{XX}}{W}_{0j}^t{W}_{0j}^t\right)R_{\mathit{XX}}-4\mu R_{\mathit{XX}}{W}_{0j}^t{W}_{0j}^t{R}_{\mathit{XX}}\right]\hfill \end{array}$$

(37)

This expression holds for any input signal. It can be simplified if$R_{\mathit{XX}}={\sigma }_x^{2}I$ . In this case we have:

$$tr\left(R_{\mathit{XX}}{K}_{V_j{V}_j}\left(\infty \right)\right)=\mu \frac{{\sigma }_0^2{\sigma }_x^{2}M+tr\left(E\left(g\left(X\right)g^t\left(X\right)H_j{H}_j^{t}X{X}^t\right)\right)-{\sigma }_x^4\left(M+2\right)tr\left(W_{0j}{W}_{0j}^t\right)}{1-\mu {\sigma }_x^2\left(M+2\right)}$$

(38)

It is now easy to determine the total misadjustment:

$$\Delta \left(\infty \right)=\sum _{j=1}^{L}tr\left(R_{\mathit{XX}}{K}_{V_j{V}_j}\left(\infty \right)\right)=\mu \frac{{\sigma }_0^2{\sigma }_x^{2}ML+tr\left[E\left(g\left(X\right)g^t\left(X\right)\sum _{j=1}^L{H}_j{H}_j^{t}X{X}^t\right)\right]-{\sigma }_x^4\left(M+2\right)tr\left(W_0{W}_0^t\right)}{1-\mu {\sigma }_x^2\left(M+2\right)}$$

(39)

Note that, as expected, in the case of linear functions Δ(∞) reduces to:

$${\Delta \left(\infty \right)|}_{g\left(X\right)=X}=\frac{\mu {\sigma }_0^2{\sigma }_x^{2}ML}{1-\mu {\sigma }_x^2\left(M+2\right)}\text{.}$$

(40)

The additional terms are due to the nonlinearities and they should be calculated specifically for each nonlinearity.

Application to the case study:

The MSE is expressed as:

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & =L{{\sigma }_0}^2+\sum _{j=1}^L\sum _{k=1}^M{\alpha }_k{\sigma }_{x_k}^2\left[\frac{{\alpha }_k}{{\left(1+2{\beta }_i{{\sigma }_{x_k}}^2\right)}^{\frac{3}{2}}}{h}_{\mathit{jk}}^2\right.\hfill \\ \left.-\frac{2}{{\left(1+{\beta }_k{{\sigma }_{x_k}}^2\right)}^{\frac{3}{2}}}{h}_{\mathit{jk}}{w}_{\mathit{jk}}\left(n\right)+w_{\mathit{jk}}^2\left(n\right)\right]\hfill \end{array}$$

(41)

The Wiener MSE is expressed in this case as:

$${\zeta }_0=L{{\sigma }_0}^2+\sum _{i=1}^M{\alpha }_i^2{\sigma }_{x_i}^2\frac{{\left(1+2{\beta }_i{{\sigma }_{x_i}}^2\right)}^{\frac{3}{2}}-1}{{\left(1+{\beta }_i{{\sigma }_{x_i}}^2\right)}^3}\sum _{j=1}^L{h}_{\mathit{ji}}^2$$

(42)

Adaptive W, the nonlinearities are frozen and known with scale factors

In this section, matrix W is adaptive, the nonlinearities are frozen and known with scale factors (Figure 5).

Figure 5

Simplified scheme, the nonlinearities are frozen and assumed known with a scaling factor.

Mean weight behavior and stationary points

The gradient descent update of matrix W is expressed as:

$$\begin{array}{cc}W\left(n+1\right)\hfill & =W\left(n\right)+2\mu e\left(n\right)\Omega g{\left(X\left(n\right)\right)}^t\hfill \\ =W\left(n\right)+2\mu \left[\right(Hg\left(X\left(n\right)\right)+N\left(n\right)\hfill \\ -W\left(n\right)\Omega g\left(X\left(n\right)\right)]\Omega g{\left(X\left(n\right)\right)}^t\hfill \end{array}$$

(43)

where$\Omega =\left[\begin{array}{ccc}\hfill {\eta }_1\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\eta }_M\hfill \end{array}\right]$.

Averaging both sides of (43) and using the standard LMS assumption of small μ, we obtain:

$$\begin{array}{cc}E\left(W\left(n+1\right)\right)\hfill & \approx E\left(W\left(n\right)\right)+2\mu \left(H\Omega R_{g\left(X\right)g\left(X\right)}\right.\hfill \\ \left.-E\left(W\left(n\right)\right){\Omega }^2{R}_{g\left(X\right)g\left(X\right)}\right)\hfill \\ =E\left(W\left(n\right)\right)\left(I-2\mu {\Omega }^2{R}_{g\left(X\right)g\left(X\right)}\right)\hfill \\ +2\mu H\Omega R_{g\left(X\right)g\left(X\right)}\hfill \end{array}$$

(44)

These recursions have a single stationary point (Wiener solution) which is:

$$W=W_0=H\times {\Omega }^{-1}$$

(45)

Following Equation (44), the mean weight behavior can be expressed as function of the initial condition as:

$$\begin{array}{cc}E\left(W\left(n\right)\right)\hfill & =W\left(0\right){\left(I-2\mu {\Omega }^2{R}_{g\left(X\right)g\left(X\right)}\right)}^n\hfill \\ +2\mu H\Omega R_{g\left(X\right)g\left(X\right)}\sum _{p=0}^{n-1}{\left(I-2\mu {\Omega }^2{R}_{g\left(X\right)g\left(X\right)}\right)}^p\hfill \end{array}$$

(46)

Hence, if μ is sufficiently small, it can be shown that the mean weights converge to the Wiener solution:

$$W\left(\infty \right)=W_0=H{\Omega }^{-1}\text{.}$$

(47)

The stability condition on μ is:$0<\mu <\frac{1}{{\lambda }_{max}}$

Where λ_max is the largest eigenvalue of the covariance matrix Ω²R_{g (X)g(X)}.

Thus, if each nonlinear function g _k (.) is known with a scaling factor η _k , then weights h _jk will be identified by w _jk (to the inverse of the scaling factor).

MSE

We have:

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & =E\left(\Vert Hg\left(X\left(n\right)\right)+N\left(n\right)\right.\hfill \\ \left.-W\left(n\right)\Omega g\left(X\left(n\right)\right){\Vert }^2\right)\hfill \\ =\sum _{j=1}^{L}E\left[e_j^2\left(n\right)\right]\hfill \end{array}$$

(48)

where:

$$\begin{array}{cc}E\left(e_j^2\left(n\right)\right)\hfill & =E\left(\Vert H_{j}g\left(X\left(n\right)\right)+N_j\left(n\right)\right.\hfill \\ \left.-W_j\left(n\right)\Omega g\left(X\left(n\right)\right){\Vert }^2\right)\hfill \end{array}$$

(49)

Using the independence of noise and weights at time n, we obtain:

$$\begin{array}{cc}E\left(e_j^2\left(n\right)\right)\hfill & ={\sigma }_0^2+E\left({‖\left(H_j-\Omega W_j\left(n\right)\left)g\right(X\left(n\right)\right)‖}^2\right)\hfill \\ ={\sigma }_0^2+H_j^t{R}_{g\left(X\right)g\left(X\right)}{H}_j\hfill \\ -2H_j^t\Omega R_{g\left(X\right)g\left(X\right)}E\left(W_j\left(n\right)\right)\hfill \\ +E\left(W_j^t\left(n\right){\Omega }^2{R}_{g\left(X\right)g\left(X\right)}{W}_j\left(n\right)\right)\hfill \end{array}$$

The MSE is therefore expressed as:

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & =L{\sigma }_0^2+\sum _{j=1}^L{H_j}^t{R}_{g\left(X\right)g\left(X\right)}{H}_j\hfill \\ -2{H_j}^t\Omega R_{g\left(X\right)g\left(X\right)}E\left(W_j\left(n\right)\right)\hfill \\ +E\left(W_j^t\left(n\right){\Omega }^2{R}_{g\left(X\right)g\left(X\right)}{W}_j\left(n\right)\right)\hfill \end{array}$$

(50)

Wiener MSE:

The Wiener MSE can be easily expressed as:

$$\begin{array}{cc}{\zeta }_0\hfill & =E\left(e_{W_0}^2\left(n\right)\right)\hfill \\ =L{\sigma }_0^2+E\left({‖\left(H-W_0\Omega \right)g\left(X\left(n\right)\right)‖}^2\right)=L{\sigma }_0^2\hfill \end{array}$$

(51)

Therefore the Wiener MSE is equal to the noise floor: There are no terms due to the nonlinearities. This is expected since the nonlinearities are known with a scaling matrix Ω (we have seen that the scaling matrix is canceled by W₀ since W₀ = HΩ^-1).

Let Z (n) = Ω g(X(n)), we can then express the MSE as a function of ζ₀, the weight fluctuation vector V(n) = W(n)–W₀, and Z(n):

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & =E\left({‖e_{W_0}\left(n\right)-\left(W\left(n\right)-W_0\right)Z\left(n\right)‖}^2\right)\hfill \\ ={\zeta }_0+E\left({‖V\left(n\right)Z\left(n\right)‖}^2\right)\hfill \\ ={\zeta }_0+\sum _{j=1}^{L}tr\left(R_{\mathit{ZZ}}{K}_{V_j{V}_j}\left(n\right)\right)\hfill \end{array}$$

(52)

Similarly to Equation (29), the misadjustment is expressed as:

$$\Delta \left(n\right)=tr\left(R_{\mathit{ZZ}}\sum _{j=1}^L{K}_{V_j{V}_j}\left(n\right)\right)\text{.}$$

(53)

The steady state MSE is then expressed as:

$$E\left({‖e\left(\infty \right)‖}^2\right)={\zeta }_0+\Delta \left(\infty \right)\text{.}$$

(54)

Derivation of the misadjustment:

It is easy to show that the weight fluctuations follow the recursion:

$$V\left(n+1\right)=V\left(n\right)+2\mu \left(e_{W_0}\left(n\right)-V\left(n\right)Z\left(n\right)\right)Z^t\left(n\right))\text{.}$$

(55)

Taking the mean of this equation and applying the orthogonality principle between the input vector and the Wiener error, we obtain:

$$E\left(V\left(n+1\right)\right)=E\left(V\left(n\right)\right)\times \left(1-2\mu R_{\mathit{ZZ}}\right)\text{.}$$

(56)

Thus, as expected, if μ is sufficiently small, E(V(n)) converges to 0.

For each vector V _j we have similar recursions:

$$V_j\left(n+1\right)=V_j\left(n\right)+2\mu \left(e_{W_{0}j}\left(n\right)-Z{\left(n\right)}^t{V}_j\left(n\right)\right)Z\left(n\right))\text{.}$$

(57)

The evaluation of the covariance matrix of the weight fluctuations is obtained by multiplying both sides of Equation (57) by$V_j^t\left(n+1\right)$ and averaging:

$$\begin{array}{c}{K}_{V_j{V}_j^t}\left(n+1\right)\hfill \\ =K_{V_j^t{V}_j^t}\left(n\right)-2\mu R_{\mathit{ZZ}}{K}_{V_j{V}_j}\left(n\right)-2\mu K_{V_j{V}_j}\left(n\right)R_{\mathit{ZZ}}\hfill \\ +2\mu E\left[e_{W_{0}j}\left(n\right)Z\left(n\right)V_j^t\left(I-2\mu Z\left(n\right)Z{\left(n\right)}^t\right)\right]\hfill \\ +2\mu E{\left[e_{W_{0}j}\left(n\right)Z\left(n\right)V_j^t\left(I-2\mu Z\left(n\right)Z{\left(n\right)}^t\right)\right]}^t\hfill \\ +4{\mu }^{2}E\left[Z\left(n\right)Z{\left(n\right)}^t{K}_{V_j{V}_j}Z\left(n\right)Z{\left(n\right)}^t\right]\hfill \\ +4{\mu }^{2}E\left[e_{W_{0}j}^2\left(n\right)Z\left(n\right)Z{\left(n\right)}^t\right]\hfill \end{array}$$

(58)

In a similar way as in Appendix III, K _VjVj (∞) can be obtained by solving the following equation:

$$\begin{array}{cc}{R}_{\mathit{ZZ}}\hfill & K_{V_j{V}_j}\left(\infty \right)+K_{V_j{V}_j}\left(\infty \right)R_{\mathit{ZZ}}-2\mu tr\left(R_{\mathit{ZZ}}{K}_{V_j{V}_j}\left(\infty \right)\right)R_{\mathit{ZZ}}\hfill \\ -4\mu R_{\mathit{ZZ}}{K}_{V_j{V}_j}\left(\infty \right)R_{\mathit{ZZ}}\hfill \\ =2\mu \left[E\left[g\left(X\right)g^t\left(X\right)H_j{H}_j^{t}Z{Z}^t\right]+2\mu {\sigma }_0^2{R}_{\mathit{ZZ}}\right.\hfill \\ \left.-2\mu tr\left(R_{\mathit{ZZ}}{W}_{0j}^t{W}_{0j}^t\right)R_{\mathit{ZZ}}-4\mu R_{\mathit{ZZ}}{W}_{0j}^t{W}_{0j}^t{R}_{\mathit{ZZ}}\right]\hfill \end{array}$$

(59)

This expression can not be further simplified because R _ZZ is not necessarily of the form${\sigma }_z^{2}I$.

Therefore, tr(R _ZZ K _VjVj (∞)) should be calculated for each nonlinearity and for each Ω.

It is interesting to study the case where nonlinearities are known with the same scaling factor, i.e., Ω = ηI. In this case, and if the input vectors are independent and the outputs of the nonlinearities are Zero-mean and of equal variance${\sigma }_g^2$, we have:

$$tr\left(R_{\mathit{ZZ}}{K}_{V_j{V}_j}\left(\infty \right)\right)=\mu \frac{{\sigma }_0^2{\eta }^2{\sigma }_g^{2}M}{1-\mu {\eta }^2{\sigma }_g^2\left(M+2\right)}$$

(60)

As expected, the total misadjustment reduces to:

$$\Delta \left(\infty \right)=\sum _{j=1}^{L}tr\left(R_{\mathit{ZZ}}{K}_{V_j{V}_j}\left(\infty \right)\right)=\mu \frac{{\sigma }_0^2{\eta }^2{\sigma }_g^{2}ML}{1-\mu {\eta }^2{\sigma }_g^2\left(M+2\right)}\text{.}$$

(61)

Here the value of the misadjustment is similar to that of linear identification of a linear system (LMS algorithm). This is expected since in this case there are no errors due to the approximation of the nonlinearities.

Case study

For the case study, it is easy to show that$G_{\mathit{ii}}=E\left(g\left(x_i^2\left(n\right)\right)\right)=\frac{{{\alpha }_i}^2{{\sigma }_x}^2}{{\left(1+2{\beta }_i{{\sigma }_{x_i}}^2\right)}^{\frac{3}{2}}}\text{.}$ This yields:

$$E\left(e_j^2\left(n\right)\right)={{\sigma }_0}^2+\sum _i^M\frac{{{\alpha }_i}^2{{\sigma }_{x_i}}^2}{{\left(1+2{\beta }_i{{\sigma }_{x_i}}^2\right)}^{\frac{3}{2}}}{\left(h_{\mathit{ji}}-w_{\mathit{ji}}\left(n\right){\eta }_i\right)}^2$$

(62)

Study of the full structure

This section deals with the full structure (Figures 2,3). All the NN and matrix W weights are updated.

Mean weight transient behavior

We take the following notations for the weights:$E\left(w_{\mathit{jk}}\left(n\right)\right)={\overline{w}}_{\mathit{jk}}\left(n\right),E\left(c_{\mathit{ki}}\left(n\right)\right)={\overline{c}}_{\mathit{ki}}\left(n\right),E\left(a_{\mathit{ki}}\left(n\right)\right)={\overline{a}}_{\mathit{ki}}\left(n\right),E\left(b_{\mathit{ki}}\left(n\right)\right)={\overline{b}}_{\mathit{ki}}\left(n\right)$.

The update of matrix W is expressed as:

$$\begin{array}{cc}W\left(n+1\right)\hfill & =W\left(n\right)+2\mu e\left(n\right)NN{\left(X\left(n\right)\right)}^t\hfill \\ =W\left(n\right)+2\mu \left[Hg\left(X\left(n\right)\right)+N\left(n\right)\right.\hfill \\ \left.-W\left(n\right)NN\left(X\left(n\right)\right)\right]NN{\left(X\left(n\right)\right)}^t\hfill \end{array}$$

(63)

Averaging both sides of (63) and using the standard LMS assumption of small μ, we obtain:

$$\begin{array}{cc}E\left(W\left(n+1\right)\right)\hfill & \approx E\left(W\left(n\right)\right)+2\mu \left(H{R}_{g\left(X\right)NN\left(X\right)}\left(n\right)\right.\hfill \\ \left.-E\left(W\left(n\right)\right)R_{NN\left(X\right)NN\left(X\right)}\left(n\right)\right)\hfill \\ =E\left(W\left(n\right)\right)\left(I-2\mu R_{NN\left(X\right)NN\left(X\right)}\left(n\right)\right)\hfill \\ +2\mu H{R}_{g\left(X\right)NN\left(X\right)}\left(n\right)\hfill \end{array}$$

(64)

Where$R_{NN\left(X\right)NN\left(X\right)}\left(n\right)=E\left[NN\left(X\left(n\right)\right)NN{\left(X\left(n\right)\right)}^t\right],R_{g\left(X\right)NN\left(X\right)}\left(n\right)=E\left[g\left(X\left(n\right)\right)NN{\left(X\left(n\right)\right)}^t\right]$.

These matrices are time-dependent since they depend on the NN block weights which are updated through time.

Using the scalar notation we have:

$$\begin{array}{c}{\overline{w}}_{\mathit{jk}}\left(n+1\right)={\overline{w}}_{\mathit{jk}}\left(n\right)\hfill \\ +2\mu E\left[\left(\sum _{i=1}^M{h}_{\mathit{ji}}{g}_i\left(x_i\left(n\right)\right)-\sum _{l=1}^M{w}_{\mathit{jl}}N{N}_l\left(n\right)\right))\right.\hfill \\ \left.\times \sum _{m=1}^N{c}_{\mathit{km}}f\left(a_{\mathit{km}}{x}_k\left(n\right)+b_{\mathit{km}}\right)\right]\hfill \\ \approx {\overline{w}}_{\mathit{jk}}\left(n\right)+2\mu \left[\sum _{i,m}^{M,N}{h}_{\mathit{ji}}{\overline{c}}_{\mathit{km}}E\left(g_i\left(x_i\left(n\right)\right)f\left({\overline{a}}_{\mathit{km}}{x}_k\left(n\right)+{\overline{b}}_{\mathit{km}}\right)\right)\right.\hfill \\ \left.-\sum _{l,m,i}^{M,N,N}{\overline{w}}_{\mathit{jl}}{\overline{c}}_{\mathit{li}}{\overline{c}}_{\mathit{km}}E\left(f\left({\overline{a}}_{\mathit{li}}{x}_l\left(n\right)+{\overline{b}}_{\mathit{li}}\right)f\left({\overline{a}}_{\mathit{km}}{x}_k\left(n\right)+{\overline{b}}_{\mathit{km}}\right)\right)\right]\hfill \end{array}$$

(65)

$\begin{array}{c}\text{Let:}{K}_i\left(a_{\mathit{km}},b_{\mathit{km}}\right)=E\left(g_i\left(x_i\left(n\right)\right)f\left(a_{\mathit{km}}{x}_k\left(n\right)+b_{\mathit{km}}\right)\right),and\hfill \\ F_{\mathit{lk}}\left(a_{\mathit{li}},b_{\mathit{li}},a_{\mathit{km}},b_{\mathit{km}}\right)=E\left(f\left(a_{\mathit{li}}{x}_l\left(n\right)+b_{\mathit{li}}\right)f\left(a_{\mathit{km}}{x}_k\left(n\right)+b_{\mathit{km}}\right)\right)\hfill \end{array}$

With these notations we have:

$$\begin{array}{cc}{\overline{w}}_{\mathit{jk}}\left(n+1\right)\hfill & ={\overline{w}}_{\mathit{jk}}\left(n\right)+2\mu \left[\sum _{i,m}^{M,N}{h}_{\mathit{ji}}{c}_{\mathit{km}}{K}_i\left({\overline{a}}_{\mathit{km}},{\overline{b}}_{\mathit{km}}\right)\right.\hfill \\ \left.-\sum _{l,m,i}^{M,N,N}{\overline{w}}_{\mathit{jl}}{\overline{c}}_{\mathit{li}}{\overline{c}}_{\mathit{km}}{F}_{\mathit{lk}}\left({\overline{a}}_{\mathit{li}},{\overline{b}}_{\mathit{li}},{\overline{a}}_{\mathit{km}},{\overline{b}}_{\mathit{km}}\right)\right]\hfill \end{array}$$

(66)

For the NN block weights we have:

$$\begin{array}{c}{\overline{c}}_{\mathit{ki}}\left(n+1\right)={\overline{c}}_{\mathit{ki}}\left(n\right)+2\mu E\left(\sum _{l=1}^L,w_{\mathit{lk}}{e}_l\left(n\right)f\left(a_{\mathit{ki}}{x}_k\left(n\right)+b_{\mathit{ki}}\right)\right)\hfill \\ \approx {\overline{c}}_{\mathit{ki}}\left(n\right)+2\mu \sum _{l=1}^L{\overline{w}}_{\mathit{lk}}E\left(\sum _{p=1}^M{h}_{\mathit{lp}}{g}_p\left(x\left(n\right)\right)f\left({\overline{a}}_{\mathit{ki}}x\left(n\right)+{\overline{b}}_{\mathit{ki}}\right)\right.\hfill \\ \left.-\sum _{m=1}^M{w}_{\mathit{lm}}\left(\sum _{q=1}^N,{\overline{c}}_{\mathit{mq}}f\left({\overline{a}}_{\mathit{mq}}x\left(n\right)+{\overline{b}}_{\mathit{mq}}\right)\right)f\left({\overline{a}}_{\mathit{ki}}x\left(n\right)+{\overline{b}}_{\mathit{ki}}\right)\right)\hfill \\ ={\overline{c}}_{\mathit{ki}}\left(n\right)+2\mu \sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(\sum _{p=1}^M{h}_{\mathit{lp}}{K}_p\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)\right.\hfill \\ \left.-\sum _{m=1}^M{\overline{w}}_{\mathit{lm}}\left(\sum _{q=1}^N,{\overline{c}}_{\mathit{mq}}{F}_{\mathit{mk}}\left({\overline{a}}_{\mathit{mq}},{\overline{b}}_{\mathit{mq}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)\right)\right)\hfill \end{array}$$

(67)

$$\begin{array}{c}{\overline{a}}_{\mathit{ki}}\left(n+1\right)={\overline{a}}_{\mathit{ki}}\left(n\right)\hfill \\ +2\mu E\left[c_{\mathit{ki}}\sum _{l=1}^L{w}_{\mathit{lk}}{e}_l\left(n\right)x\left(n\right)f^{\text{'}}\left(a_{\mathit{ki}}x\left(n\right)+b_{\mathit{ki}}\right)\right]\hfill \\ \approx {\overline{a}}_{\mathit{ki}}\left(n\right)+2\mu {\overline{c}}_{\mathit{ki}}\sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(\sum _{p=1}^M,h_{\mathit{lp}}E\left(g_p\left(x\left(n\right)\right)x\left(n\right)\right.\right.\hfill \\ \left.\times f^{\text{'}}\left({\overline{a}}_{\mathit{ki}}x\left(n\right)+{\overline{b}}_{\mathit{ki}}\right)\right)-\sum _{m=1}^M{\overline{w}}_{\mathit{lm}}\left(\sum _{q=1}^N{\overline{c}}_{\mathit{mq}}E\left(f\left({\overline{a}}_{\mathit{mq}}x\left(n\right)\right.\right.\right.\hfill \\ \left.\left.\left.\left.+{\overline{b}}_{\mathit{mq}}\right)x\left(n\right)f^{\text{'}}\left({\overline{a}}_{\mathit{ki}}x\left(n\right)+{\overline{b}}_{\mathit{ki}}\right)\right)\right)\right)\hfill \\ ={\overline{a}}_{\mathit{ki}}\left(n\right)+2\mu {\overline{c}}_{\mathit{ki}}\sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(\sum _{p=1}^M{h}_{\mathit{lp}}\frac{\partial K_p\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{a}}_{\mathit{ki}}}\right.\hfill \\ -\sum _{m,q,\left(m,q\right)\ne \left(k,i\right)}^{M,N}{\overline{w}}_{\mathit{lm}}{\overline{c}}_{\mathit{mq}}\frac{\partial F_{\mathit{mk}}\left({\overline{a}}_{\mathit{mq}},{\overline{b}}_{\mathit{mq}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{a}}_{\mathit{ki}}}\hfill \\ \left.-\frac{1}{2},{\overline{w}}_{\mathit{lk}},{\overline{c}}_{\mathit{ki}},\frac{\partial F_{\mathit{ki}}\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{a}}_{\mathit{ki}}}\right)\hfill \end{array}$$

(68)

$$\begin{array}{c}{\overline{b}}_{\mathit{ki}}(n+1)={\overline{b}}_{\mathit{ki}}\left(n\right)\hfill \\ +2\mu E\left[c_{\mathit{ki}}\sum _{l=1}^L{w}_{\mathit{lk}}{e}_l\left(n\right)f^{\text{'}}\left(a_{\mathit{ki}}x\left(n\right)+b_{\mathit{ki}}\right)\right]\hfill \\ \approx {\overline{b}}_{\mathit{ki}}\left(n\right)+2\mu {\overline{c}}_{\mathit{ki}}\sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(\sum _{p=1}^M,h_{\mathit{lp}}E\left(g_p\left(x\left(n\right)\right)f^{\text{'}}\left({\overline{a}}_{\mathit{ki}}x\left(n\right)\right.\right.\right.\hfill \\ \left.\left.+{\overline{b}}_{\mathit{ki}}\right)\right)-\sum _{m=1}^M{\overline{w}}_{\mathit{lm}}\left(\sum _{q=1}^N{\overline{c}}_{\mathit{mq}}E\left(f\left({\overline{a}}_{\mathit{mq}}x\left(n\right)\right.\right.\right.\hfill \\ \left.\left.\left.\left.+{\overline{b}}_{\mathit{mq}}\right)f^{\text{'}}\left({\overline{a}}_{\mathit{ki}}x\left(n\right)+{\overline{b}}_{\mathit{ki}}\right)\right)\right)\right)\hfill \\ ={\overline{b}}_{\mathit{ki}}\left(n\right)+2\mu {\overline{c}}_{\mathit{ki}}\sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(\sum _{p=1}^M{h}_{\mathit{lp}}\frac{\partial K_p\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{b}}_{\mathit{ki}}}\right.\hfill \\ \left.-\sum _{m,q}^{M,N}{\overline{w}}_{\mathit{lm}}{\overline{c}}_{\mathit{mq}}\frac{\partial F_{\mathit{mk}}\left({\overline{a}}_{\mathit{mq}},{\overline{b}}_{\mathit{mq}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{b}}_{\mathit{ki}}}\right)\hfill \end{array}$$

(69)

These equations hold for any nonlinearity. In the following, we will calculate them explicitly for the case study described in Section 2.3

Application to the case study:

Since the inputs are independent and Zero-mean, we have K _i (a _km ,b _km )=0, k≠i , and (see Appendix I)

$$\begin{array}{cc}{K}_i\left(a_{\mathit{im}},b_{\mathit{im}}\right)\hfill & =E\left(g_i\left(x_i\left(n\right)\right)f\left(a_{\mathit{im}}x\left(n\right)+b_{\mathit{im}}\right)\right)\hfill \\ =\sqrt{\frac{2}{\pi }}\frac{{\alpha }_i{\sigma }_x^2}{\left(1+{\beta }_i{\sigma }_x^2\right)}\frac{a_{\mathit{im}}}{\sqrt{{\sigma }_x^2\left(a_{\mathit{im}}^2+{\beta }_i\right)+1}}\hfill \\ -\frac{1}{2}\sqrt{\frac{2}{\pi }}{\alpha }_i{\sigma }_x^2{b}_{\mathit{im}}^2\frac{a_{\mathit{im}}}{{\left(1+{\sigma }_x^2\left({\beta }_i+a_{\mathit{im}}^2\right)\right)}^{\frac{3}{2}}}\hfill \end{array}$$

(70)

In the other hand we have:$F_{\mathit{lk}}\left(a_{\mathit{li}},b_{\mathit{li}},a_{\mathit{km}},b_{\mathit{km}}\right)=0,l\ne k$, and (see Appendix I)

$$\begin{array}{c}{F}_{\mathit{kk}}\left(a_{\mathit{ki}},b_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{km}}\right)\hfill \\ =E\left(f\left(a_{\mathit{ki}}x\left(n\right)+b_{\mathit{ki}}\right)f\left(a_{\mathit{km}}x\left(n\right)+b_{\mathit{km}}\right)\right)\hfill \\ =\frac{2}{\pi }{sin}^{-1}\left(\frac{a_{\mathit{ki}}{a}_{\mathit{km}}{\sigma }_x^2}{\sqrt{1+{\sigma }_x^2{a}_{\mathit{ki}}^2+{\sigma }_x^2{a}_{\mathit{km}}^2+{\sigma }_x^4{a}_{\mathit{ki}}^2{a}_{\mathit{km}}^2}}\right)\hfill \\ -\frac{1}{\pi }{b}_{\mathit{ki}}^2\frac{{\sigma }_x^2{a}_{\mathit{ki}}{a}_{\mathit{km}}}{\left(1+{\sigma }_x^2{a}_{\mathit{ki}}^2\right)\sqrt{1+{\sigma }_x^2\left(a_{\mathit{ki}}^2+a_{\mathit{km}}^2\right)}}\hfill \\ -\frac{1}{\pi }{b}_{\mathit{km}}^2\frac{{\sigma }_x^2{a}_{\mathit{ki}}{a}_{\mathit{km}}}{\left(1+{\sigma }_x^2{a}_{\mathit{km}}^2\right)\sqrt{1+{\sigma }_x^2\left(a_{\mathit{ki}}^2+a_{\mathit{km}}^2\right)}}\hfill \\ +b_{\mathit{ki}}{b}_{\mathit{km}}\frac{2}{\pi }\frac{1}{\sqrt{1+{\sigma }_x^2\left(a_{\mathit{ki}}^2+a_{\mathit{km}}^2\right)}}\hfill \end{array}$$

(71)

Inserting these expressions in equations (66)-(69), we obtain:

$$\begin{array}{cc}{\overline{w}}_{\mathit{jk}}\left(n+1\right)\hfill & ={\overline{w}}_{\mathit{jk}}\left(n\right)+2\mu \left[h_{\mathit{jk}}\sum _{m=1}^N{c}_{\mathit{km}}{K}_k\left({\overline{a}}_{\mathit{km}},{\overline{b}}_{\mathit{km}}\right)\right.\hfill \\ \left.-{\overline{w}}_{\mathit{jk}}\sum _{m,i}^N{\overline{c}}_{\mathit{ki}}{\overline{c}}_{\mathit{km}}{F}_{\mathit{kk}}\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}},{\overline{a}}_{\mathit{km}},{\overline{b}}_{\mathit{km}}\right)\right]\hfill \end{array}$$

(72)

$$\begin{array}{cc}{\overline{c}}_{\mathit{ki}}\left(n+1\right)\hfill & ={\overline{c}}_{\mathit{ki}}\left(n\right)+2\mu \sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(h_{\mathit{lk}}{K}_k\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)\right.\hfill \\ \left.-{\overline{w}}_{\mathit{lk}}\left(\sum _{q=1}^N,{\overline{c}}_{\mathit{kq}}{F}_{\mathit{kk}}\left({\overline{a}}_{\mathit{kq}},{\overline{b}}_{\mathit{kq}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)\right)\right)\hfill \end{array}$$

(73)

$$\begin{array}{cc}{\overline{a}}_{\mathit{ki}}\left(n+1\right)\hfill & ={\overline{a}}_{\mathit{ki}}\left(n\right)+2\mu {\overline{c}}_{\mathit{ki}}\sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(h_{\mathit{lk}}\frac{\partial K_k\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{a}}_{\mathit{ki}}}\right.\hfill \\ -{\overline{w}}_{\mathit{lk}}\sum _{q\ne k}^N{\overline{c}}_{\mathit{kq}}\frac{\partial F_{\mathit{kk}}\left({\overline{a}}_{\mathit{kq}},{\overline{b}}_{\mathit{kq}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{a}}_{\mathit{ki}}}\hfill \\ \left.-\frac{1}{2}{\overline{w}}_{\mathit{lk}}{\overline{c}}_{\mathit{kk}}\frac{\partial F_{\mathit{kk}}\left({\overline{a}}_{\mathit{kk}},{\overline{b}}_{\mathit{kk}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{a}}_{\mathit{kk}}}\right)\hfill \end{array}$$

(74)

$$\begin{array}{cc}{\overline{b}}_{\mathit{ki}}\left(n+1\right)\hfill & ={\overline{b}}_{\mathit{ki}}\left(n\right)+2\mu {\overline{c}}_{\mathit{ki}}\sum _{l=1}^L{\overline{w}}_{\mathit{lk}}\left(h_{\mathit{lk}}\frac{\partial K_k\left({\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{b}}_{\mathit{ki}}}\right.\hfill \\ \left.-\sum _q^N{\overline{w}}_{\mathit{lk}}{\overline{c}}_{\mathit{kq}}\frac{\partial F_{\mathit{kk}}\left({\overline{a}}_{\mathit{kq}},{\overline{b}}_{\mathit{kq}},{\overline{a}}_{\mathit{ki}},{\overline{b}}_{\mathit{ki}}\right)}{\partial {\overline{b}}_{\mathit{ki}}}\right)\hfill \end{array}$$

(75)

The explicit expressions of the different derivatives are detailed in Appendix II.

Stationary points

We obtain the stationary points by setting to 0 the expectations of the updating gradient terms in (64) and(4.5-7).

For W, we obtain:

$$\begin{array}{c}{W}_0=H\times R_{g\left(X\right)NN\left(X\right)}{R}_{NN\left(X\right)NN\left(X\right)}^{-1}=H\times U,where\hfill \\ U=R_{g\left(X\right)NN\left(X\right)}{R}_{NN\left(X\right)NN\left(X\right)}^{-1}\text{.}\hfill \end{array}$$

(76)

For c _ki we obtain the equations:

$$\begin{array}{c}\sum _{l=1}^L{w}_{\mathit{lk}}\left(h_{\mathit{lk}}{K}_k\left(a_{\mathit{ki}},b_{\mathit{ki}}\right)\right.\hfill \\ \left.-{\overline{w}}_{\mathit{lk}}\left(\sum _{q=1}^N,c_{\mathit{kq}}{F}_{\mathit{kk}}\left(a_{\mathit{kq}},b_{\mathit{kq}},a_{\mathit{ki}},b_{\mathit{ki}}\right)\right)\right)=0\hfill \end{array}$$

(77)

For a _ki we obtain the equations:

$$\begin{array}{c}\sum _{l=1}^L{w}_{\mathit{lk}}\left(h_{\mathit{lk}}\frac{\partial K_k\left(a_{\mathit{ki}},b_{\mathit{ki}}\right)}{\partial a_{\mathit{ki}}}\right.\hfill \\ -{\overline{w}}_{\mathit{lk}}\sum _{q\ne k}^N{\overline{c}}_{\mathit{kq}}\frac{\partial F_{\mathit{kk}}\left(a_{\mathit{kq}},b_{\mathit{kq}},a_{\mathit{ki}},b_{\mathit{ki}}\right)}{\partial a_{\mathit{ki}}}\hfill \\ \left.-\frac{1}{2}{w}_{\mathit{lk}}{c}_{\mathit{kk}}\frac{\partial F_{\mathit{kk}}\left(a_{\mathit{kk}},b_{\mathit{kk}},a_{\mathit{ki}},b_{\mathit{ki}}\right)}{\partial a_{\mathit{kk}}}\right)=0\hfill \end{array}$$

(78)

For b _ki we obtain:

$$\begin{array}{c}\sum _{l=1}^L{w}_{\mathit{lk}}\left(h_{\mathit{lk}}\frac{\partial K_k\left(a_{\mathit{ki}},b_{\mathit{ki}}\right)}{\partial b_{\mathit{ki}}}\right.\hfill \\ \left.-\sum _q^N{w}_{\mathit{lk}}{c}_{\mathit{kq}}\frac{\partial F_{\mathit{kk}}\left(a_{\mathit{kq}},b_{\mathit{kq}},a_{\mathit{ki}},b_{\mathit{ki}}\right)}{\partial b_{\mathit{ki}}}\right)=0\hfill \end{array}$$

(79)

The above equations are nonlinear in the NN variables. They can be solved numerically, but they are very difficult to solve analytically.

Convergence of the algorithm to the stationary points:

It is always interesting to show whether an algorithm is capable of converging to its stationary points or not. In our case it is difficult to establish this, since the updating equations of the weights are nonlinear, except for W.

In the case where the NN weights are frozen we can establish the convergence condition for W.

In this case we have:

$$\begin{array}{cc}E\left(W\left(n+1\right)\right)\hfill & =E\left(W\left(n\right)\right)\left(I-2\mu R_{NN\left(X\right)NN\left(X\right)}\right)\hfill \\ +2\mu H{R}_{g\left(X\right)NN\left(X\right)}\hfill \end{array}$$

(80)

The covariance matrices are fixed, since in this case the NN weights are frozen.

E(W(n)) can be expressed as a function of the initial condition as:

$$\begin{array}{cc}E\left(W\left(n\right)\right)\hfill & =W\left(0\right){\left(I-2\mu R_{NN\left(X\right)NN\left(X\right)}\right)}^n+2\mu H{R}_{g\left(X\right)NN\left(X\right)}\hfill \\ \times \sum _{p=0}^{n-1}{\left(I-2\mu R_{NN\left(X\right)NN\left(X\right)}\right)}^p\hfill \end{array}$$

(81)

If μ is sufficiently small, the steady state solution to (81) is:

$$W\left(\infty \right)=W_0=H{R}_{g\left(X\right)NN\left(X\right)}{R}_{NN\left(X\right)NN\left(X\right)}^{-1}\text{.}$$

(82)

Hence, the mean weights converge to the stationary point W₀, and the stability condition on μ is:

$$0<\mu <\frac{1}{{\lambda }_{max}}$$

(83)

Where λ_max is the largest eigenvalue of the correlation matrix R _NN(X)NN(X) .

Application to the case study:

For the case study it can be shown that U reduces to a diagonal matrix:

$$\begin{array}{cc}U\hfill & =R_{g\left(X\right)NN\left(X\right)}{R}_{NN\left(X\right)NN\left(X\right)}^{-1}\hfill \\ =\left[\begin{array}{ccc}\hfill {\gamma }_1\hfill & \hfill 0\dots \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\gamma }_2\dots \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\dots \hfill & {\gamma }_M\hfill \end{array}\right]\text{,}\hfill \end{array}$$

(84)

where:

$${\gamma }_k=\frac{{\sum }_m^N{K}_k\left(a_{\mathit{km}},b_{\mathit{km}}\right)}{{\sum }_{m,i}^{N,M}{c}_{\mathit{ki}}{c}_{\mathit{km}}{F}_{\mathit{kk}}\left(a_{\mathit{ki}},b_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{km}}\right)}$$

(85)

This indicates that weights w _jk are scaled versions of the unknown weights h _jk , the scale factor γ _k is the same for all the weights connecting the k^th NN block to the outputs and it depends only on block k weights. If the error is sufficiently small, the k^th block NN will approximate the k^th nonlinearity to the inverse of the scale factor.

MSE expression

The transient MSE is determined by:

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & ={\sum }_{j=1}^{L}E\left[e_j^2\left(n\right)\right]\hfill \\ =E\left(\Vert Hg\left(X\left(n\right)\right)+N\left(n\right)\right.\hfill \\ \left.-W\left(n\right)NN\left(X\left(n\right)\right){\Vert }^2\right)\hfill \end{array}$$

(86)

where:

$$\begin{array}{cc}E\left(e_j^2\left(n\right)\right)\hfill & =E\left(\Vert H_{j}g\left(X\left(n\right)\right)+N_j\left(n\right)\right.\hfill \\ \left.-W_j\left(n\right)NN\left(X\left(n\right)\right){\Vert }^2\right)\hfill \\ =E\left((\sum _{i=1}^M{h}_{\mathit{ji}}{g}_i\left(x_i\left(n\right)\right)+N_j\left(n\right)\right.\hfill \\ \left.-\sum _{k-1}^M{w}_{\mathit{jk}}\times \sum _{i=1}^N{c_{\mathit{ki}}f\left(a_{\mathit{ki}}{x}_k\left(n\right)+b_{\mathit{ki}}\right))}^2\right)\hfill \end{array}$$

(87)

Which can be expressed as:

$$\begin{array}{c}E\left(e_j^2\left(n\right)\right)={{\sigma }_0}^2+\sum _{i,l}^M{h}_{\mathit{ji}}{h}_{\mathit{jl}}E\left(g_i\left(x_i\left(n\right)\right)g_l\left(x_l\left(n\right)\right)\right)\hfill \\ -2\sum _{i,k}^M{h}_{\mathit{ji}}\left(n\right)w_{\mathit{jk}}\left(\sum _{m=1}^N,c_{\mathit{km}}E\left(g_i\left(x_i\left(n\right)\right)f\left(a_{\mathit{km}}{x}_k\left(n\right)+b_{\mathit{km}}\right)\right)\right)\hfill \\ +\sum _{k,l}^M\sum _{i,m}^N{w}_{\mathit{jl}}{w}_{\mathit{jk}}{c}_{\mathit{li}}{c}_{\mathit{ki}}E\left(f\left(a_{\mathit{li}}{x}_l\left(n\right)+b_{\mathit{li}}\right)f\left(a_{\mathit{km}}{x}_k\left(n\right)+b_{\mathit{km}}\right)\right)\hfill \end{array}$$

(88)

Let G_il = E(g _i (x _i (n))gl(x _i (n))). Using the notations of Section 4.1, we have:

$$\begin{array}{cc}E\left(e_j^2\left(n\right)\right)\hfill & ={{\sigma }_0}^2+\sum _{i,l}^M{h}_{\mathit{ji}}{h}_{\mathit{jl}}{G}_{\mathit{il}}\hfill \\ -2\sum _{i,k}^M{h}_{\mathit{ji}}{w}_{\mathit{jk}}\left(\sum _{m=1}^N,c_{\mathit{km}}{K}_i\left(a_{\mathit{km}},b_{\mathit{km}}\right)\right)\hfill \\ +\sum _{k,l}^M\sum _{i,m}^N{w}_{\mathit{jl}}{w}_{\mathit{jk}}{c}_{\mathit{li}}{c}_{\mathit{km}}{F}_{\mathit{lk}}\left(a_{\mathit{li}},b_{\mathit{li}},a_{\mathit{km}},b_{\mathit{km}}\right)\hfill \end{array}$$

(89)

Application to the case study:

It can be easily shown that:

$$\begin{array}{cc}E\left(e_j^2\left(n\right)\right)\hfill & ={{\sigma }_0}^2+\sum _i^M{h}_{\mathit{ji}}^2{G}_{\mathit{ii}}\hfill \\ -2\sum _k^M{h}_{\mathit{jk}}{w}_{\mathit{jk}}\left(\sum _{m=1}^N,c_{\mathit{km}}{K}_k\left(a_{\mathit{km}},b_{\mathit{km}}\right)\right)\hfill \\ +\sum _k^M{w}_{\mathit{jk}}^2\sum _{i,m}^N{c}_{\mathit{ki}}^2{F}_{\mathit{kk}}\left(a_{\mathit{ki}},b_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{km}}\right)\text{.}\hfill \end{array}$$

(90)

The 1^st term of$E\left(e_j^2\left(n\right)\right)$represents the noise power, the 2^nd term is the signal power of the j^th MIMO output, the 3^rd term is the sum of the individual contributions of the neurons weighed by W and H weights, the 4^th term represents the sum of the coupling terms between neurons inside the same block weighed by W. Note that since the inputs are Zero-mean and independent, there are no coupling terms between neurons in different blocks (as in Eq. (89)).

The total MSE is then expressed as:

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & =L{{\sigma }_0}^2+\sum _{j,i}^M{h}_{\mathit{ji}}^2{G}_{\mathit{ii}}\hfill \\ -2\sum _{j,k}^M{h}_{\mathit{jk}}{w}_{\mathit{jk}}\left(\sum _{m=1}^N,c_{\mathit{km}}{K}_k\left(a_{\mathit{km}},b_{\mathit{km}}\right)\right)\hfill \\ +\sum _{j,k}^M{w}_{\mathit{jk}}^2\sum _{i,m}^N{c}_{\mathit{ki}}^2{F}_{\mathit{kk}}\left(a_{\mathit{ki}},b_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{km}}\right)\text{.}\hfill \end{array}$$

(91)

Case of frozen NN weights:

It is interesting to see the behavior of the MSE in the case where the NN weights are frozen.

In this case we have:

$$\begin{array}{cc}{\zeta }_0\hfill & =E\left({‖e_{W_0}\left(n\right)‖}^2\right)\hfill \\ =L{\sigma }_0^2+E\left({‖Hg\left(X\left(n\right)\right)-W_{0}NN\left(X\left(n\right)\right)‖}^2\right)\text{.}\hfill \end{array}$$

(92)

Here the minimum MSE depends on the noise floor and on the NN approximation error of the nonlinearities. It is clear from this equation and from Section 3.2 that, if the NN blocks ideally identify the nonlinearities (to within scale factors), then ζ₀ reduces to the noise floor.

The MSE can be written as a function of ζ₀ as:

$$\begin{array}{cc}E\left({‖e\left(n\right)‖}^2\right)\hfill & =E\left({‖e_{W_0}\left(n\right)-\left(W\left(n\right)-W_0\right)NN\left(X\left(n\right)\right)]‖}^2\right)\hfill \\ =E\left({‖e_{W_0}\left(n\right)-V\left(n\right)X\left(n\right)‖}^2\right)\hfill \end{array}$$

(93)

The steady state MSE is in this case:

$$E\left({‖e\left(\infty \right)‖}^2\right)={\zeta }_0+\sum _{j=1}^{L}tr\left(R_{NN\left(X\right)NN\left(X\right)}{K}_{V_j{V}_j}\left(\infty \right)\right)\text{.}$$

(94)

The misadjustment can be derived similarly as in Sections 3.1 and 3.2. We obtain a similar equation as (53), by replacing R _ZZ by R _NN(X)NN(X) . The equation can not be simplified further.

It is interesting to notice, however, that if the NN blocks perfectly identify the nonlinearities and if the conditions above equation (60) are fulfilled, then:

$$\begin{array}{cc}\Delta \left(\infty \right)\hfill & =\sum _{j=1}^{L}tr\left(R_{NN\left(X\right)NN\left(X\right)}{K}_{V_j{V}_j}\left(\infty \right)\right)\hfill \\ =\mu \frac{{\sigma }_0^2{\sigma }_g^{2}ML}{1-\mu {\sigma }_g^2\left(M+2\right)}\hfill \end{array}$$

(95)

Simulation examples

In this section we present some simulation results which are applied to the case study described in Section 2.3. In these simulations, we have considered a 2 × 2 MIMO system (i.e., M = L = 2). For the parameterized nonlinearities we have chosen α₁=α₂=1, β₁=1, β₂=2. Unless otherwise specified, the inputs are uncorrelated Zero-mean white Gaussian processes with σ _xi = 1. In the simulations, the unknown combining matrix was fixed and was taken as$H=\left[\begin{array}{cc}1\hfill & 0.3\hfill \\ 0.3\hfill & 1\hfill \end{array}\right]$. For example, in a MIMO communication system, H can be seen as the propagation matrix between 2 transmitting antennas and 2 receiving antennas.

Linear adaptation

For the linear adaptation case (Section 3.1), the adaptive system is composed of a 2x2 matrix W. For the noise we have taken σ₀ = 0.001. The mean weight recursions and the MSE transient behaviors (Figures 6,7) have been estimated over 20 Monte Carlo (MC) simulations and compared to the theoretical derivations (Equations (19) and (41)). This chosen number of MC simulations shows excellent fit between the Theory and MC estimations which confirms the validity of the different assumptions made. A larger number of MC simulations allows a better smoothing of the curves, but the conclusions remain the same.

Figure 6

Smoothed MSE vs. iteration number - Linear adaptation case.

Figure 7

Mean weight transient behavior - Linear adaptation case.

Matrix W converges to a scaled version of H:$W_0=\left[\begin{array}{cc}0.3536\hfill & 0.0577\hfill \\ 0.1061\hfill & 0.1925\hfill \end{array}\right]=\text{H\hspace{0.17em}U,}U=\left[\begin{array}{cc}0.3536\hfill & 0.0000\hfill \\ 0.0000\hfill & 0.1925\hfill \end{array}\right].$ Note the typical behavior of the LMS algorithm: A time constant controls the transient part of the learning curve and the mean weight curve. This is fundamentally different from the full NN system learning which is governed by several time constants and presents plateau regions (Section 5.2). It should be noted that the steady state MSE is high because of the error caused by the fact that the nonlinearities are not approximated (actually they are modeled by the identity function) (Equation (25)).

MSE surface for the full NN algorithm

We move now to the study of the full NN algorithm. In this simulation, we have taken N = 3 neurons in each of the two NN blocks. The learning rate was fixed to μ =0.0045. Figure 8 shows the MSE surface (i.e., Eq. (90), with no time dependence) as a function of w₁₁ and w₁₂ (the other parameters were fixed). It is clear that the MSE is quadratic in w₁₁ and w₂₁. It presents a single global minimum (as shown in Equations (76 and 84)).

Figure 8

MSE surface, MSE ( w ₁₁ , w ₂₁ ).

Figure 9 (resp. Figure 10) shows the MSE surface as a function of a₁₁ and c₁₁ (resp. a₁₁ and a₁₂) (the other parameters were fixed). It can be noted the flat areas (plateau regions) around the minima of the MSE surface. This explains the slow evolution of the NN weights when the algorithm gets close to its convergence point.

Figure 9

MSE surface, MSE ( a ₁₁ , c ₂₁ ).

Figure 10

MSE surface, MSE ( a ₁₁ , a _{1 2} ).

The MSE evolution during the learning process (Equation (90)) has been compared to 20 MC estimations (Figure 11a): The theory shows very good fit with the simulation results. In the Figure, we notice that the MSE presents several phases (each phase is controlled by a time constant) which end by a plateau phase where the MSE decreases very slowly. This is a typical behavior of the backpropagation algorithm[1] which is fundamentally different from that of the linear adaptation scheme (Figure 6). Here the MSE error is much smaller. This is expected, since here the additional MSE error due to the nonlinearities (Eq. 92) is highly reduced because our NN blocks have correctly identified the unknown nonlinearities (Figure 12,13). Here we are in a situation close to that of Section 3.2 (Equations 51-52).

Figure 11

a: MSE during the learning process (theory and simulation), σ ₀ =0.001 and μ =0.0045. b: MSE for different values of noise variance, the learning rate was μ=0.0045 (theoretical results). c : MSE for different values of learning rate μ (the noise variance was set to σ₀=0.001), theoretical results.

Figure 12

Identification of g ₁ ( x ), by the 1 ^st NN block.

Figure 13

Identification of g ₂ ( x ) by the 2 ^nd NN block.

Figure 11b shows the MSE evolution during the learning process for different values of the noise variance σ₀. It can be seen that, as the noise variance decreases, the MSE decreases. However, below a certain value of σ₀ (here σ₀=0.0005), the MSE curves are almost identical. This is because in this case, the weight misadjustment error (for the linear part) and the nonlinear approximation error (of the nonlinear memoryless part) are much higher than the error caused by the presence of noise (see Eqs. 92-93).

Figure 11c investigates the influence of the learning rate μ. It can be seen that as μ increases (up to μ=0.002), the algorithm is faster and the MSE is lower at the end of the simulation time. However, for μ>0.002, as μ increases, the algorithm is faster at the beginning of the learning process, but the MSE is higher at the end of the simulation time. This is due to the misadjustment error which is higher for higher μ (see, e.g., Eq. 95).

Mean weight transient behavior for the full NN algorithm

Here we keep the system described in Section 5.2. The mean weight recursions for the linear combiner W and the two NN blocks are shown in Figure 14,15,16 for both theory and MC estimations. The theoretical and estimated curves are indistinguishable. This confirms the validity of the different assumptions made in Sections 4.1 and 4.2.

Figure 14

Mean weight behavior: Matrix W (theory and simulation).

Figure 15

Mean weight behavior: First NN block (theory and simulation).

Figure 16

Mean weight behavior: Second NN block (theory and simulation vs. iterations).

Notice that, in Figure 14, W weights have a fast evolution at the beginning of the learning process (with values approaching H×U(n) where U is a diagonal matrix). They then evolve slowly till the end of the learning process. The slow evolution is justified by the plateau regions presented by the MSE surface. At the end of this simulation, matrix U was close to a diagonal matrix:$U_{\mathit{Sim}}=\left[\begin{array}{cc}1.2702\hfill & 0.0001\hfill \\ 0.0003\hfill & 1.0946\hfill \end{array}\right],$ (and$U_{\mathit{Theory}}=\left[\begin{array}{cc}\hfill 1.270\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1.095\hfill \end{array}\right]$). This result is expected since the inputs are uncorrelated (Equations 84-85).

Figure 12,13 show that functions g₁ (x) and g₂ (x) have been correctly identified by the corresponding NN blocks (the NN functions are normalized by the scaling factors γ₁=1/1.2702, γ₁=1/1.0946, respectively).

Impact of correlated inputs

In the simulations below we study the impact of correlated inputs. The input signal vector is chosen here as a 2D Gaussian process with covariance matrix of the form$R_{\mathit{XX}}=\left[\begin{array}{cc}1\hfill & \rho \hfill \\ \rho \hfill & 1\hfill \end{array}\right]\text{.}$ The number of neurons in each NN block was taken as N = 5 neurons. The learning rate μ=0.0075. We have run several simulations for different values of the cross correlation. Note that (ρ=0) corresponds to independent inputs, and (ρ=1) corresponds to the same input (i.e., x₁=x₂). The values of matrix U and the MSE after n=210⁵ iterations are shown in Table 1. It can be seen from Table 1 that, in practice, matrix U remains very close to a diagonal matrix, even for high correlation between inputs. This indicates that the system is capable of correctly identifying the nonlinearities even when the inputs are highly correlated. The identification performances for the cases (ρ=0.6) and ( ρ=0.99) are illustrated in Figures 17,18 and19,20, respectively. As expected, the MSE increases as the correlation between inputs increases (Table 1). When ρ=1 (i.e., the two inputs are the same) the system is capable of correctly identifying the overall MIMO input–output transfer function. However, in this case, it is not capable of separating the nonlinearities (as U is not diagonal). The reason is that in this case, the system is seen by the learning algorithm as a 1x2 SIMO system which has several equivalent structures. Figure 21 shows an example of two equivalent structures. Therefore, the adaptive system is structurally not able to separate the nonlinearities. It is worth to note that for the case (ρ=0.999), the inputs look like noisy versions of each other (i.e., this is equivalent to a 1x2 system identification problem with noisy inputs). Thus, the MSE for the case (ρ=0.999 ) is larger than the MSE for the case (ρ=1).

Table 1 Effect of correlated inputs

Figure 17

Identification of g ₁ ( x ), case ρ =0.6.

Figure 18

Identification of g ₂ ( x ), case ρ =0.6.

Figure 19

Identification of g ₁ ( x ), case ρ =0.99.

Figure 20

Identification of g ₂ ( x ), case ρ =0.99.

Figure 21

Example of two equivalent structures when the inputs are the same.

Conclusion and future work

The paper provides a statistical analysis of NN modeling and identification of a class of nonlinear MIMO systems. The study investigates the MSE error, mean weight behavior, stationary points, misadjustment error, and stability conditions. The unknown system is composed of a set of single-input memoryless nonlinearities followed by a combining matrix. The NN model is composed of a set of single-input memoryless NN blocks followed by an adaptive linear combiner. The paper is supported with simulation results which show good agreement between the theoretical recursions and MC simulations. Future work will focus on 3 research directions. The first will explore the theoretical findings in order to express the effect of the number of neurons on the transient and steady state behavior of the algorithm. The second research axis will investigate the case where matrix H is time-varying and/or with memory (this may have applications, for example, in adaptive control of nonlinear dynamical MIMO systems). Finally, we will study the algorithm behavior and performance for specific inputs (such as space-time coded signals used in wireless communications and their impact on the system capacity).

Appendix I

• 1) Calculation of F _kk Let x ₁ and x ₂ be two zero-mean Gaussian variables such that ${\sigma }_{x_1}^2={\sigma }_{x_2}^2={\sigma }_x^{2}andE\left(x_1{x}_2\right)=\rho$Therefore, $F_{\mathit{kk}}\left(a_{\mathit{ki}},b_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{km}}\right)=E{\left(f\left(a_{\mathit{ki}}{x}_k\left(n\right)+b_{\mathit{ki}}\right)f\left(a_{\mathit{km}}{x}_k\left(n\right)+b_{\mathit{km}}\right)\right)}_{\rho ={\sigma }_x^2}\text{.}$Using Price’s theorem we have: $E\left[\frac{{\partial }^{2}f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)f\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)}{\partial x_1\partial x_2}\right]=\frac{\partial E\left[f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)f\left(w_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)\right]}{\partial \rho }\text{.}$Let $U\left(\rho \right)=E\left[\frac{{\partial }^{2}f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)f\left(w_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)}{\partial x_1\partial x_2}\right]\text{,}$Then: $E{\left[f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)f\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)\right]}_{\rho ={\sigma }_x^2}-E{\left[f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)f\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)\right]}_{\rho =0}={\int }_0^{{\sigma }_x^2}U\left(\rho \right)d\rho \text{.}$

  Thus, using the un-correlation criteria between x₁ and x₂ for ρ=0 , we have:$E{\left[f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)f\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)\right]}_{\rho =0}=E\left[f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)\right]E\left[f\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)\right]\text{.}$ Thus:$F\left(a_{\mathit{ki}},b_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{km}}\right)=E\left[f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)\right]E\left[f\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)\right]+{\int }_0^{{\sigma }_x^2}U\left(\rho \right)d\rho \text{.}$ We have:

  $\begin{array}{c}U\left(\rho \right)=E\left[\frac{{\partial }^{2}f\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)f\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)}{\partial x_1\partial x_2}\right]\hfill \\ =\frac{2}{\pi }{a}_{\mathit{ki}}{a}_{\mathit{km}}\frac{1}{2\pi {\left|R\right|}^{\frac{1}{2}}}{\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }{e}^{-\frac{1}{2}{\left(a_{\mathit{ki}}{x}_1+b_{\mathit{ki}}\right)}^2}{e}^{-\frac{1}{2}{\left(a_{\mathit{km}}{x}_2+b_{\mathit{km}}\right)}^2}\hfill \\ e^{-\frac{1}{2}{X}^t{R}^{-1}X}d{x}_{1}d{x}_2\hfill \end{array}$

  where$X={\left[x_1{x}_2\right]}^{t}andR=\left[\begin{array}{cc}\hfill {\sigma }_x^2\hfill & \hfill \rho \hfill \\ \hfill \rho \hfill & \hfill {\sigma }_x^2\hfill \end{array}\right]\text{.}$Combining the terms in the exponentials and completing the squares, the integrals can be calculated:

  $\begin{array}{c}U\left(\rho \right)=\frac{2}{\pi }\frac{a_{\mathit{ki}}{a}_{\mathit{km}}}{\sqrt{1+{\sigma }_x^2{a}_{\mathit{ki}}^2+{\sigma }_x^2{a}_{\mathit{km}}^2+\left({\sigma }_x^4-{\rho }^2\right)a_{\mathit{ki}}^2{a}_{\mathit{km}}^2}}\times \hfill \\ exp\left[\frac{1}{2}\left[-b_{\mathit{ki}}^2-b_{\mathit{km}}^2+\frac{1}{a_{\mathit{ki}}^2+\frac{{\sigma }_x^2}{{\sigma }_x^2-{\rho }^2}}\left[b_{\mathit{ki}}^2{a}_{\mathit{ki}}^2+\right.\right.\right.\hfill \\ \left.\left.\left.\frac{{\left(b_{\mathit{ki}}{a}_{\mathit{ki}}\left(a_{\mathit{ki}}^2+\frac{{\sigma }_x^2}{{\sigma }_x^2-{\rho }^2}\right)+b_{\mathit{km}}{a}_{\mathit{km}}\frac{\rho }{{\sigma }_x^2-{\rho }^2}\right)}^2}{\left(a_{\mathit{ki}}^2+\frac{{\sigma }_x^2}{{\sigma }_x^2-{\rho }^2}\right)\left(a_{\mathit{km}}^2+\frac{{\sigma }_x^2}{{\sigma }_x^2-{\rho }^2}\right)a_{\mathit{km}}^2-\frac{{\rho }^2}{{\sigma }_x^2-{\rho }^2}}\right]\right]\right]\text{.}\hfill \end{array}$

  Note that in the biasless case (i.e. all the bias terms are set to 0) this expression reduces to:

  $$U\left(\rho \right)=\frac{2}{\pi }\frac{a_{\mathit{ki}}{a}_{\mathit{km}}}{\sqrt{1+{\sigma }_x^2{a}_{\mathit{ki}}^2+{\sigma }_x^2{a}_{\mathit{km}}^2+\left({\sigma }_x^4-{\rho }^2\right)a_{\mathit{ki}}^2{a}_{\mathit{km}}^2}}\text{.}$$

  The integral is then simple to calculate:

  $${\int }_0^{{\sigma }_x^2}U\left(\rho \right)d\rho =\frac{2}{\pi }{sin}^{-1}\left(\frac{a_{\mathit{ki}}{a}_{\mathit{km}}{\sigma }_x^2}{\sqrt{1+{\sigma }_x^2{a}_{\mathit{ki}}^2+{\sigma }_x^2{a}_{\mathit{km}}^2+{\sigma }_x^4{a}_{\mathit{ki}}^2{a}_{\mathit{km}}^2}}\right)$$

  In the other hand, since$E\left[f\left(w_k{x}_1\right)\right]=E\left[f\left(w_i{x}_2\right)\right]=0$, then:$F\left(a_{\mathit{ki}},a_{\mathit{km}},0,0\right)=\frac{2}{\pi }{sin}^{-1}\left(\frac{a_{\mathit{ki}}{a}_{\mathit{km}}{\sigma }_x^2}{\sqrt{1+{\sigma }_x^2{a}_{\mathit{ki}}^2+{\sigma }_x^2{a}_{\mathit{km}}^2+{\sigma }_x^4{a}_{\mathit{ki}}^2{a}_{\mathit{km}}^2}}\right)\text{.}$When the bias terms are not set to 0, a Taylor series expansion on the bias terms can be used in order to avoid the calculation of the integral.

• 2) Calculation of K

  $$\begin{array}{cc}{K}_k\hfill & \left(a_{\mathit{km}},b_{\mathit{km}}\right)=E\left[g_k\left(x_k\right)f\left(a_{\mathit{km}}{x}_k+a_{\mathit{km}}\right)\right]\hfill \\ =\frac{1}{\sqrt{2\pi }}\frac{1}{{\sigma }_x}{\int }_{-\infty }^{+\infty }{\alpha }_{k}x{e}^{-\frac{{\beta }_k{x}^2}{2}}{e}^{-\frac{x^2}{2{\sigma }_x^2}}{\int }_0^{a_{\mathit{km}}x+b_{\mathit{km}}}{e}^{-\frac{u^2}{2}}dudx\text{.}\hfill \end{array}$$

  The inside integral can be eliminated by integrating by parts on variable x.The integral is then evaluated by combining the terms in the exponentials and completing the squares. This yields:

  $$\begin{array}{cc}{K}_k\hfill & \left(a_{\mathit{km}},b_{\mathit{km}}\right)=\sqrt{\frac{2}{\pi }}\frac{{\alpha }_k}{\frac{1}{{\sigma }_x^2}+{\beta }_k}\frac{a_{\mathit{km}}}{\sqrt{{\sigma }_x^2\left(a_{\mathit{km}}^2+{\beta }_k\right)+1}}\hfill \\ \times exp\left(\frac{-b_{\mathit{km}}^2}{2}\left(1-\frac{{\sigma }_x^2}{1+{\sigma }_x^2\left({\beta }_k+a_{\mathit{km}}^2\right)}\right)\right)\text{.}\hfill \end{array}$$

  Again, a Taylor series expansion can be used to simplify this expression.Note that in the biasless case we have:

  $$K_k\left(a_{\mathit{km}},0\right)=\sqrt{\frac{2}{\pi }}\frac{\alpha }{\frac{1}{{\sigma }_x^2}+{\beta }_k}\frac{a_{\mathit{km}}}{\sqrt{{\sigma }_x^2\left(a_{\mathit{km}}^2+{\beta }_k\right)+1}}$$

Appendix II

The derivatives needed to compute the different recursions are expressed as follows:

$$\begin{array}{c}\frac{\partial K_k\left(a_{\mathit{km}},0\right)}{\partial a_{\mathit{km}}}=\sqrt{\frac{2}{\pi }}\frac{\alpha {\sigma }_x^2}{{\left({\sigma }_x^2\left(a_{\mathit{km}}^2+{\beta }_k\right)+1\right)}^{\frac{3}{2}}},\frac{\partial K_k\left(a_{\mathit{km}},b_{\mathit{km}}\right)}{\partial a_{\mathit{km}}}=\frac{\partial K_k\left(a_{\mathit{km}},0\right)}{\partial a_{\mathit{km}}}-\sqrt{\frac{2}{\pi }}{\alpha }_k{\sigma }_x^2\frac{b_{\mathit{km}}^2}{2}\frac{1+{\sigma }_x^2{\beta }_k-2{\sigma }_x^2{a}_{\mathit{km}}^2}{{\left(1+{\sigma }_x^2\left({\beta }_k+a_{\mathit{km}}^2\right)\right)}^{\frac{5}{2}}}\hfill \\ \hfill \frac{\partial F_{\mathit{kk}}\left(a_{\mathit{ki}},0,a_{\mathit{ki}},0\right)}{\partial a_{\mathit{ki}}}=\frac{4}{\pi }\frac{{\sigma }_x^2{a}_{\mathit{ki}}}{\left({\sigma }_x^2{a}_{\mathit{ki}}^2+1\right)\sqrt{1+2{\sigma }_x^2{a}_{\mathit{ki}}^2}},\hfill \\ \hfill \frac{\partial F_{\mathit{kk}}\left(a_{\mathit{ki}},a_{\mathit{ki}},b_{\mathit{ki}},b_{\mathit{ki}}\right)}{\partial a_{\mathit{ki}}}=\frac{\partial F\left(a_{\mathit{ki}},0,a_{\mathit{ki}},0\right)}{\partial a_{\mathit{ki}}}-\frac{2}{\pi }{b}_{\mathit{ki}}^2{\sigma }_x^2{a}_{\mathit{ki}}\frac{5{\sigma }_x^2{a}_{\mathit{ki}}^2+3}{{\left(1+{\sigma }_x^2{a}_{\mathit{ki}}^2\right)}^2{\left(1+2{\sigma }_x^2{a}_{\mathit{ki}}^2\right)}^{\frac{3}{2}}}\hfill \\ \hfill \frac{\partial F\left(a_{\mathit{ki}},0,a_{\mathit{km}},0\right)}{\partial a_{\mathit{ki}}}=\frac{2}{\pi }\frac{{\sigma }_x^2{a}_{\mathit{km}}}{\left({\sigma }_x^2{a}_{\mathit{ki}}^2+1\right)\sqrt{1+{\sigma }_x^2\left(a_{\mathit{ki}}^2+a_{\mathit{km}}^2\right)}}\hfill \\ \frac{\partial F\left(a_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{ki}},b_{\mathit{km}}\right)}{\partial a_{\mathit{ki}}}=\frac{\partial F\left(a_{\mathit{ki}},a_{\mathit{ki}},0,0\right)}{\partial a_{\mathit{ki}}}-\frac{1}{\pi }{b}_{\mathit{ki}}^2\frac{{\sigma }_x^2{a}_{\mathit{km}}}{\left(1+{\sigma }_x^2{a}_{\mathit{ki}}^2\right)\sqrt{1+{\sigma }_x^2\left(a_{\mathit{km}}^2+a_{\mathit{ki}}^2\right)}}\left(\frac{1+{\sigma }_x^2{a}_{\mathit{km}}^2}{1+{\sigma }_x^2\left(a_{\mathit{km}}^2+a_{\mathit{ki}}^2\right)}-\frac{2{\sigma }_x^2{a}_{\mathit{ki}}^2}{1+{\sigma }_x^2{a}_{\mathit{ki}}^2}\right)\hfill \\ \hfill -\frac{1}{\pi }{a}_{\mathit{km}}^2\frac{{\sigma }_x^2{a}_{\mathit{km}}}{{\left(1+{\sigma }_x^2\left(a_{\mathit{km}}^2+a_{\mathit{ki}}^2\right)\right)}^{\frac{3}{2}}}-b_{\mathit{ki}}{b}_{\mathit{km}}\frac{2}{\pi }\frac{{\sigma }_x^2{a}_{\mathit{ki}}}{{\left(1+{\sigma }_x^2\left(a_{\mathit{km}}^2+a_{\mathit{ki}}^2\right)\right)}^{\frac{3}{2}}}\hfill \\ \hfill \frac{\partial F\left(a_{\mathit{ki}},a_{\mathit{km}},b_{\mathit{ki}},b_{\mathit{km}}\right)}{\partial b_{\mathit{km}}}=-\frac{2}{\pi }{b}_{\mathit{ki}}\frac{{\sigma }_x^2{a}_{\mathit{ki}}{a}_{\mathit{km}}}{\left(1+{\sigma }_x^2{a}_{\mathit{ki}}^2\right)\sqrt{1+{\sigma }_x^2\left(a_{\mathit{km}}^2+a_{\mathit{ki}}^2\right)}}-b_{\mathit{km}}\frac{2}{\pi }\frac{1}{\sqrt{1+{\sigma }_x^2\left(a_{\mathit{km}}^2+a_{\mathit{ki}}^2\right)}}\hfill \end{array}$$

Appendix III

$$\begin{array}{cc}{K}_{V_j{V}_j^t}\left(n+1\right)\hfill & =K_{V_j^t{V}_j^t}\left(n\right)-2\mu R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\hfill \\ -2\mu K_{V_j{V}_j}\left(n\right)R_{\mathit{XX}}\hfill \\ +2\mu E\left[e_{W_{0}j}\left(n\right)X{V}_j^t\left(I-2\mu X{X}^t\right)\right]\hfill \\ +2\mu E{\left[e_{W_{0}j}\left(n\right)X{V}_j^t\left(I-2\mu X{X}^t\right)\right]}^t\hfill \\ +4{\mu }^{2}E\left[X{X}^t{K}_{V_j{V}_j}X{X}^t\right]\hfill \\ +4{\mu }^{2}E\left[e_{W_{0}j}^2\left(n\right)X{X}^t\right]\hfill \end{array}$$

(96)

The calculations are similar to[9] Appendix, the main difference is that here we deal with a multi-dimensional input. Therefore, we will follow the same methodology as in[9]. Following[10] the expectation before the last one can be calculated as:

$$\begin{array}{cc}E\left[X{X}^t{K}_{V_j{V}_j}\left(n\right)X{X}^t\right]\hfill & =tr\left(R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\right)R_{\mathit{XX}}\hfill \\ +2R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)R_{\mathit{XX}}\text{.}\hfill \end{array}$$

(97)

The first expectation is expressed as:

$$\begin{array}{cc}E\hfill & \left[e_{W_{0j}}\left(n\right)X{V}_j^t\left(I-\mu X{X}^t\right)\right]\hfill \\ =E\left[e_{W_{0j}}\left(n\right)X{V}_j^t\right]-\mu E\left[e_{W_{0j}}\left(n\right)X{V}_j^{t}X{X}^t\right]\hfill \end{array}$$

(98)

The first term is Zero (orthogonality principle). The second term is:

$$\begin{array}{cc}E\left[e_{W_{0j}}\left(n\right)X{V}_j^{t}X{X}^t\right]\hfill & =E\left[\left(H_j^t{g}_j\left(x_j\right)+N_j\left(n\right)\right.\right.\hfill \\ \left.\left.-W_{0j}^{t}X\left(n\right)\right)X{V}_j^{t}X{X}^t\right]\hfill \end{array}$$

(99)

The middle term is Zero (Zero-mean white noise), the last expectation is:

$$\begin{array}{cc}E\left[W_{0j}^{t}X\left(n\right)X{V}_j^{t}X{X}^t\right]\hfill & =E\left[X\left(n\right)X{W}_0{V}_j^t\left(n\right)X{X}^t\right]\hfill \\ \approx tr\left(R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)\right)R_{\mathit{XX}}\hfill \\ +2R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)R_{\mathit{XX}}\hfill \end{array}$$

(100)

The first expectation in Eq. (99)$E\left[H_j^{t}g\left(X\right)X{V}_j^t\left(n\right)X{X}^t\right]\approx E\left[H_j^{t}g\left(X\right)XE\left(V_j^t\left(n\right)\right)X{X}^t\right]$ involves the nonlinearity g_j (x _j ) and should be evaluated explicitly.The remaining expectation in (96) is:$E\left[e_{\mathit{oj}}^2\left(n\right)X{X}^t\right]\text{.}$

$$E\left[e_{\mathit{oj}}^2\left(n\right)X{X}^t\right]=E\left[{\left(H_j^{t}g\left(X\right)-W_{0j}^{t}X\right)}^{2}X{X}^t\right]+{\sigma }_0^2{R}_{\mathit{XX}}$$

(101)

We have

$$\begin{array}{cc}E\left[{\left(W_{0j}^{t}X\right)}^{2}X{X}^t\right]\hfill & =E\left[X{X}^t{W}_{0j}^t{W}_{0j}^{t}X{X}^t\right]\hfill \\ =tr\left(R_{\mathit{XX}}{W}_{0j}^t{W}_{0j}^t\right)R_{\mathit{XX}}\hfill \\ +2R_{\mathit{XX}}{W}_{0j}^t{W}_{0j}^t{R}_{\mathit{XX}}\hfill \end{array}$$

(102)

The first term in (102) is:

$$\begin{array}{cc}E\left[{\left(H_j^{t}g\left(X\right)\right)}^{2}X{X}^t\right]\hfill & =E\left[H_j{H}_j^{t}g\left(X\right)g^t\left(X\right)X{X}^t\right]\hfill \\ =E\left[g\left(X\right)g^t\left(X\right)H_j{H}_j^{t}X{X}^t\right]\hfill \end{array}$$

• (96) is then expressed as:

  $$\begin{array}{c}{K}_{V_j{V}_j^t}\left(n+1\right)=K_{V_j^t{V}_j^t}\left(n\right)-2\mu R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\hfill \\ -2\mu K_{V_j{V}_j}\left(n\right)R_{\mathit{XX}}+4{\mu }^2\left(-E\left[H_j^{t}g\left(X\right)XE\left(V_j^t\left(n\right)\right)X{X}^t\right]\right.\hfill \\ \left.+tr\left(R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right))R_{\mathit{XX}}+R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)R_{\mathit{XX}}\right)\right)\hfill \\ +4{\mu }^2\left(-E\left[H_j^{t}g\left(X\right)XE\left(V_j^t\left(n\right)\right)X{X}^t\right]\right.\hfill \\ {\left.+tr\left(R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right))R_{\mathit{XX}}+R_{\mathit{XX}}{W}_{0}E\left(V_j^t\left(n\right)\right)R_{\mathit{XX}}\right)\right)}^t\hfill \\ +4{\mu }^2\left(tr\left(R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)\right)R_{\mathit{XX}}+2R_{\mathit{XX}}{K}_{V_j{V}_j}\left(n\right)R_{\mathit{XX}}\right)\hfill \\ +4{\mu }^2\left(E\left[g\left(X\right)g^t\left(X\right)H_j{H}_j^{t}X{X}^t\right]+{\sigma }_0^2{R}_{\mathit{XX}}\right.\hfill \\ \left.-tr\left(R_{\mathit{XX}}{W}_{0}t{W}_0^t\right)R_{\mathit{XX}}-2R_{\mathit{XX}}{W}_{0j}^t{W}_{0j}^t{R}_{\mathit{XX}}\right)\hfill \end{array}$$

Endnotes

This work has been supported by The Natural Sciences and Engineering Research Council of Canada (NSERC).

The time index of the weights has been omitted from the right hand side of the equations to make them easier to read.