Optimal MSE solution for a decision feedback equalizer

Abstract

Due to the inherent feedback in a decision feedback equalizer (DFE) the minimum mean square error (MMSE) or Wiener solution is not known exactly. The main difficulty in such analysis is due to the propagation of the decision errors, which occur because of the feedback. Thus in literature, these errors are neglected while designing and/or analyzing the DFEs. Then a closed form expression is obtained for Wiener solution and we refer this as ideal DFE (IDFE). DFE has also been designed using an iterative and computationally efficient alternative called least mean square (LMS) algorithm. However, again due to the feedback involved, the analysis of an LMS-DFE is not known so far. In this paper we theoretically analyze a DFE taking into account the decision errors. We study its performance at steady state. We then study an LMS-DFE and show the proximity of LMS-DFE attractors to that of the optimal DFE Wiener filter (obtained after considering the decision errors) at high signal to noise ratios (SNR). Further, via simulations we demonstrate that, even at moderate SNRs, an LMS-DFE is close to the MSE optimal DFE. Finally, we compare the LMS DFE attractors with IDFE via simulations. We show that an LMS equalizer outperforms the IDFE. In fact, the performance improvement is very significant even at high SNRs (up to 33%), where an IDFE is believed to be closer to the optimal one. Towards the end, we briefly discuss the tracking properties of the LMS-DFE.

Introduction

A channel equalizer is an important component of a communication system and is used to mitigate the inter symbol interference (ISI) introduced by the channel. The equalizer depends upon the channel characteristics. A variety of equalizers have been proposed and utilized in communication systems [1–3] Usually simple linear equalizers (LE) would suffice (see for e.g., [1–3]) but for a channel with deep spectral nulls one would require a more complex, non LE like a decision feed back equalizer (DFE).

A LE is a linear filter that is used to mitigate ISI while a Wiener filter (WF) equalizer is an optimal filter that minimizes the mean square error (MSE) between the input symbols and the decoded symbols (decoded after the equalizer). Closed form expression for WF LE is available ([4, 5] etc). This closed form expression involves a matrix inverse which can be computationally intensive if the filter has a large dimension. Alternatively, least mean square linear equalizer (LMS-LE), a computationally efficient iterative algorithm, is used extensively (see [4–6]) to obtain the WF equalizer. It can also track the time variations in the WF, if required, as in the case of Wireless channels. For a fixed channel its convergence to the WF has been studied in [6, 7] (see also the references therein). Its performance on a wireless (time varying) channel has been studied theoretically in [8, 9] (see also [4, 5, 10] and the references there in, where the performance has been studied via simulations, approximations and upper bounds on probability of error).

Decision feedback are nonlinear equalizers (a pair of linear filters one in the forward path and another in the feedback path), which can provide significantly better performance than LE [3, 11, 12], especially for ‘bad’ channels. A DFE feeds back the previous decisions of the transmitted symbols, to nullify the ISI due to them (which can now happen without amplifying the thermal noise) and makes a better decision about the current symbol. Although these equalizers have also been used for quite sometime, due to feedback their behavior is much more complex than that of the LEs. Hence their performance is not well understood. Existence of a hard decoder inside the feedback loop, due to its nonlinearity, makes the study all the more difficult. A DFE mainly exploits the finite alphabet structure of the hard decoder output [2, 13] and hence the hard decoder cannot be ignored (i.e., its performance is better than a system with a soft decoder).

Since the statistics of the previous decisions in a DFE are not known, there is no known technique available that provides an minimum MSE (MMSE) DFE (we will call it as DFE-WF in the rest of the article) even for a fixed channel [2, 3, 14]. Thus an MMSE DFE is commonly designed by assuming perfect decisions (see, e.g., [2, 15]). For convenience, for the rest of the article, we will call such a DFE as ideal DFE (IDFE). In this article IDFE is also computed using perfect channel estimates. The IDFE often outperforms the Linear WF significantly [3, 11, 12]. But it is generally believed that DFE-WF, the true MSE optimal DFE (designed considering the decision errors), can outperform even this.

Another way to obtain an optimal filter is to replace the feedback filter at the receiver by a precoder at the transmitter [3, 14]. This way one can indeed obtain the optimal filter but this requires the knowledge of the channel at the transmitter. For wireless channels, which are time varying, this is often not an attractive solution [2, 3].

Some research has been done to deal with the decision errors in a DFE. Sternad et al. [16] approximated the errors in decisions with an additive white Gaussian noise (AWGN) independent of the input sequence and obtained a DFE WF. But as is stated in the article this approximation is not realistic. Erdogan et al. [13] obtain an H^∞ optimal DFE taking into account the decision errors. However no comparison to DFE-WF was provided.

Ideal DFE also contains a matrix inverse for which LMS is again used as a computationally efficient alternative in practical communications systems. However, convergence of LMS-DFE is not well understood even for a fixed channel, again due to the complexity introduced by the feedback. Trajectory of the LMS-DFE algorithm, on a fixed channel, with a soft decoder in the feedback loop has been approximated by an ODE in [17]. But this ODE does not approximate the LMS-DFE with a hard decoder. Beneveniste et al. [6] have shown the ODE approximation of an LMS-DFE with a hard decoder. But the ODE obtained by them is not explicit enough. Furthermore, they do not relate the attractors of this ODE to the DFE-WF.

Our conjuncture is that LMS can actually converge to the true DFE WF (obtained considering the decision errors) and one of the main goals of this article is to prove the same. In this article, we study an LMS-DFE on a fixed channel using an ODE approximation. Towards this, we first obtain the stationary performance of the system with DFE and prove the existence of DFE-WF (the minimum MSE solution) for every channel state (whenever the domain of optimization is compact). We then show that the DFE-WF and an LMS-DFE attractor are close to each other at high signal to noise ratios (SNRs). We show the same is true for nominal values of SNRs via simulations.

Further we demonstrate via simulations, that the LMS-DFE can outperform the commonly used IDFE, at all practical SNRs. An interesting observation is that, the improvement is significant even at high SNRs where an IDFE does not suffer from error propagation and is believed to be close to the true DFE-WF.

The article is organized as follows. Our system model, notations and assumptions are discussed in Section “The model and notation”. In Section “The issues and our approach” we discuss our approach. Section “Analysis of LMS-DFE and DFE-WF” obtains an ODE approximation and then the analysis of the attractors of LMS-DFE. Section “Numerical examples” provides some examples. Section “Tracking analysis” briefs the tracking behavior while Section “Conclusions” concludes the article. The sections Appendices 1 to 5 provide proofs for our theorems.

The model and notation

We consider a communication system with a DFE (see Figure 1). Inputs {s _k } enter a finite impulse response channel ${\left\{z_l\right\}}_{l=0}^{L-1}$, and are corrupted by an additive zero mean white Gaussian noise {n _k } with variance σ². The channel output, u _k , at time k, is given by,

$$u_k=\sum _{l=0}^{L-1}{s}_{k-l}{z}_l+n_k.$$

Figure 1

Block diagram of a wireless channel followed by a DFE.

The channel output passes through a DFE given by a linear forward filter θ _f and a linear feedback filter θ _b . In addition, there is a hard decoder Q(.). The output of the decoder at time k is,

$${ŝ}_k=Q\left(\sum _{l=0}^{N_f-1}{{\theta }_f}_l{u}_{k-l}+\sum _{l=1}^{N_b}{{\theta }_b}_l{ŝ}_{k-l}\right).$$

(1)

We provide below the assumptions made and the notations used in this article. Most of these assumptions can be generalized as discussed at the end of this section.

• Sequences {s _k }and {n _k }are independent and identically distributed (i.i.d) sequences and are independent of each other. The inputs {s _k }are uniformly distributed over { + 1,−1}(BPSK modulation).

• $f_{\mathcal{N}}\left(y\right)$ is the N dimensional standard i.i.d Gaussian density, where N is the dimension of the vector y, i.e., $f_{\mathcal{N}}\left(y\right)={\left(2\Pi \right)}^{-N/2}\mathit{ex}{p}^{-\frac{{\left|y\right|}^2}{2}}$. Whenever not mentioned, integrability is with respect to $f_{\mathcal{N}}\left(y\right)\mathit{dy}$.

• The equalizer forward, feedback filters are given by ${\left\{{\theta }_{f_l}\right\}}_{l=0}^{N_f-1}$, ${\left\{{\theta }_{b_l}\right\}}_{l=1}^{N_b}$ respectively. Also, let $N_L\triangleq N_f+L-1$.

• We assume that the symbols are modulated using BPSK and so the hard decoder equals, Q(x):=1_{x≥0}−1_{x<0}in (1).

• For any vector, x, x _l represents its l^thcomponent and $x_l^k$, l≤k, represents the vector [ x _k x_k−1 … x _l ]^T.

  The following vector notations are used:

  $$\begin{array}{ll}{S}_k& \triangleq s_{k-N_L+1}^k,N_k\triangleq n_{k-N_f+1}^k,\\ U_k& \triangleq u_{k-N_f+1}^k,{Ŝ}_k\triangleq {ŝ}_{k-N_b+1}^k,\\ X_k& \triangleq {\left[U_k^T{Ŝ}_{k-1}^T\right]}^T,G_k\triangleq {\left[S_k^T{X}_k^T\right]}^T,\\ {\theta }_f& \triangleq {{\theta }_f}_0^{N_f-1},{\theta }_b\triangleq {{\theta }_b}_1^{N_b},\\ J_k& \triangleq {\left[S_k^T{Ŝ}_{k-1}^T{N}_k^T\right]}^T,\Theta \triangleq {\left[{{\theta }_f}^T{{\theta }_b}^T\right]}^T,\\ Z& \triangleq \left[z_0,z_1\dots z_{L-1}\right].\end{array}$$

  In the above, S _k ,U _k ,N _k and ${Ŝ}_{k-1}$, respectively represent the vector of input symbols, channel outputs, noise samples and the decoder decisions that influence the equalizer output at time k. Vector X _k forms input to the equalizer at time k while G _k , J _k are the two alternate representations of the system state at time k. Vector Z is the vector form of the channel while θ _f , θ _b are that of the equalizer feed-forward and feedback filters.

• Θ _k represent the time varying equalizer at time k.

• Let $\mathcal{S}:=\{+1,-1\}$. Under the above assumptions, {G _k } and {J _k } are Markov chains for a fixed channel, equalizer pair at (Z,Θ). These two Markov chains take values in ${\mathcal{S}}^{N_L}\times {\mathcal{S}}^{N_b}\times {\mathcal{R}}^{N_f}$, where $\mathcal{R}$ is the set of real numbers. The current and the previous states of both these Markov chain are represented by the ordered pairs (i,y), (j,y^′) respectively. Here i,j take values from the discrete part of the state part of the state space, ${\mathcal{S}}^{N_L}\times {\mathcal{S}}^{N_b}$, while y,y^′ take values in ${\mathcal{R}}^{N_f}$.

• Ψ={ψ _l }l=0N _L −1 represents the convolution of the channel {z _l } and the forward filter θ _f .

• The input to the hard decoder for a given state of the Markov chain is represented by,

  $$e_{\Theta }(i,y):=\sum _{l=0}^{N_L-1}{\psi }_l{s}_{k-l}+\sum _{l=0}^{N_f}{{\theta }_f}_l{n}_{k-l}+\sum _{l=1}^{N_b}{{\theta }_b}_l{ŝ}_{k-l}.$$

  Note that ${ŝ}_{k-1}=Q\left(e_{\Theta }\right(j,y^{\prime }\left)\right)$.

• B(Θ,δ), $\bar{B}(\Theta ,\delta )$ are the open and closed balls respectively with center Θ and radius δ.

• The equalizer output without noise, e _Θ (i,0)≠0for all values of i at the LMS attractor. Without this assumption the LMS algorithm makes more errors than the correct decisions.

Thus, the channel outputs {u _k }pass through a DFE Θ with a hard decoder. The performance of this system will depend upon the DFE filters Θ. We are interested in a filter Θ that minimizes the commonly used criterion, the mean of the squared error between the input symbol s _k and their corresponding decisions Θ^tX _k (MSE):

$$\text{MSE}=E\left[{\left|s_k-{\Theta }^t{X}_k\right|}^2\right].$$

(2)

The LMS algorithm,

$${\Theta }_{k+1}={\Theta }_k-{\mu }_k{H}_{{\Theta }_k}\left(G_k\right);H_{\Theta }\left(G\right)\triangleq X\left(X^t\Theta -s\right),$$

(3)

a computationally efficient iterative algorithm, is expected to provide the MMSE solution. However, with a feedback structure inserted, the convergence behavior of LMS is not understood properly. In fact, it is not even clear if the minimum mean square problem is well posed neither is it clear if an MMSE solution exists. Even prior to these questions, one first needs to define the expectation in (2) appropriately. One is often interested in optimizing a stationary performance, i.e., expectation in (2) is with respect to the stationary distribution of the system. However the stationary distribution depends upon the parameter Θ. The existence of the stationary distribution for any given Θ is not known. We take up these issues one by one and our final goal is to show that the above iterative algorithm (3) indeed converges close to the MMSE solution.

One can easily extend the theory of this article to any finite alphabet (complex) input source with any arbitrary distribution and to a complex channel. However we stick to BPSK modulation and to a real channel to keep the explanations simple. Also, the theory to follow, considers an optimal equalizer for delay 0. The entire theory will go through for any arbitrary delay. Indeed in Section “Numerical examples”, an example with an optimal equalizer for delay 1, is presented. This is once again done to simplify the explanations.

The issues and our approach

A DFE-WF on a fixed channel (if it exists) is given by,

$${\Theta }^{\ast }=\text{arg}\underset{\Theta }{min}E{\left[{\Theta }^t{X}_k-s_k\right]}^2,$$

(4)

where the expectation on the right hand side is defined under stationarity for a given Θ. Vector $X_k={\left[U_k^T,{Ŝ}_{k-1}^T\right]}^T$, includes previous decisions ${Ŝ}_{k-1}$ and hence its stationary distribution depends upon the parameter Θ. Thus this is a complex case of optimization in which, the stationary distribution defining the average cost also depends upon the parameter to be optimized. There is no known technique to compute a WF, Θ^∗ of (4), even for a fixed channel.

In practical systems, a DFE WF is commonly designed assuming perfect decisions (i.e., ${Ŝ}_k=S_{k-N_b+1}^k$), which we have called IDFE. It is easy to see that the IDFE for a fixed channel is given by,

$$\begin{array}{ll}{\Theta }_{\text{IDFE}}& ={\left(E\left[{\mathcal{X}}_k{\mathcal{X}}_k^t\right]\right)}^{-1}E\left[{\mathcal{X}}_k{s}_k\right],\\ \text{where}{\mathcal{X}}_k^t& :=\left[\begin{array}{c}{U}_k^t{S_{k-N_b+1}^{k-1}}^t\end{array}\right].\end{array}$$

This computation may be expensive because of matrix inversion and LMS (3) is actually used as an alternative [4, 5]. Our claim is that in case of a DFE, apart from being computationally efficient the LMS algorithm also outperforms the IDFE, Θ_IDFE. This is because we will see briefly that the LMS attractors are close to DFE-WF while IDFE is away from DFE-WF. We achieve this goal by showing that the LMS-DFE attractors are close to that of the DFE-WF at high SNRs (later in Section “Numerical examples” we show that this covers the practically used SNRs). Further, LMS can also be used to track the channel variations. We first study an LMS-DFE on a fixed channel and later on briefly discuss its tracking behavior.

Another issue related to (4) is that we should take the expectation in the right hand side under stationarity. However, it appears that the existence of stationary distribution of {X _k } for a given Θ is not known. Thus, first, in Theorem 2, we show the existence of a unique stationary distribution (and stationary density w.r.t. $f_{\mathcal{N}}\left(y\right)\mathit{dy}$) for {X _k } for any Θ.

As is usually done in adaptive algorithm analysis, we study the LMS-DFE using an ODE approximating it. Using the stationary distribution of {X _k } we convert the ODE in [6] to the following more tractable ODE,

$$\stackrel{\blacksquare }{\Theta }\left(t\right)=-\frac{1}{2}{E}_{\Theta }\left[{\nabla }_{\Theta }{\left[{\Theta }^{t}X-s\right]}^2\right]=-E_{\Theta }\left[X\left({\Theta }^{t}X-s\right)\right].$$

(5)

The attractors of the LMS-DFE will be the zeros of the RHS of the above ODE, while the DFE-WF will be a zero of the gradient (if it exists) of the cost in the RHS of (4). Under certain conditions (with ∇representing the gradient),

$$\begin{array}{ll}{\nabla }_{\Theta }{E}_{\Theta }\left[{\left[{\Theta }^{t}X-s\right]}^2\right]=& E_{\Theta }\left[{\nabla }_{\Theta }{\left[{\Theta }^{t}X-s\right]}^2\right]\\ +E\left[{\left[{\Theta }^{t}X-s\right]}^2{\nabla }_{\Theta }{\Pi }_{\Theta }\right],\end{array}$$

(6)

where Π _Θ is the stationary density of the Markov chain, {J _k }, w.r.t. the Lebesgue measure, when the DFE Θ is used. One can expect the LMS-DFE attractors to be close to the DFE-WF, if the second term in the RHS of (6) is close to zero. However, we could not even get differentiability of Π _Θ . Nevertheless, we achieve the required differentiability (Theorem 3) by considering a hard decoder that is a slightly perturbed version of the original hard decoder. We also show that the DFE-WF and an LMS-DFE attractor of this perturbed decoder converge to that of the original hard decoder as the level of perturbation tends to zero (Theorem 4). We then analyze this perturbed decoder and show that the LMS-DFE attractors of this decoder are close to its DFE-WFs at high SNR (Theorem 5). This suggests that at high SNR an LMS attractor for the original decoder is close to its DFE-WF.

Analysis of LMS-DFE and DFE-WF

We provide a step by step analysis of LMS-DFE and its connection to DFE-WF in this section, while addressing the issues raised in Section “The issues and our approach” one after the other.

Previous ODE approximation result

We start with an ODE approximation for LMS-DFE, which will be used in the subsequent sections for performance analysis. DFE with a hard decoder has been approximated by an ODE in [6]. We start our LMS-DFE analysis with this ODE. We reproduce the ODE approximation result of [6] here in our notations. Towards this goal, as a first step, we write down the ODE approximating LMS-DFE (3): let Θ(t,a) denote the solution of the ODE,

$$\stackrel{\blacksquare }{\Theta }\left(t\right)=-h\left(\Theta \right)\text{with}h\left(\Theta \right):=\underset{n\to \infty }{lim}{P}_{\Theta }^n{H}_{\Theta }(j,y^{\prime }),$$

(7)

with initial condition Θ(0)=a, where $P_{\Theta }^n$ is the n-step transition function of the Markov chain J _k with DFE Θ, and $P_{\Theta }^n{H}_{\Theta }(j,y^{\prime })$ is the expectation of the function H _Θ (G) (defined in (3)) using the conditional measure P Θ n(.|j,y^′)(Note G _k is a fixed function of J _k ). The limit in (7) will be independent of the initial condition (j,y^′) ([6], p. 252).

It is easy to see that the LMS algorithm satisfies all the required hypothesis of ([6], Theorem 13, p. 278) and hence one can approximate its trajectory on any finite time scale with the solution of the ODE (7), the precise result is:

Theorem 1

For any initial condition Θ₀, finite time T, with $t\left(r\right):=\sum _{k=0}^r{\mu }_k$,

$$\begin{array}{l}\underset{\left\{r:t\left(r\right)\le T\right\}}{sup}\left|{\Theta }_r-\Theta \left(t\left(r\right),{\Theta }_0\right)\right|{\to }^{p}0\\ \text{as}\sum _k{\mu }_k^{1+\delta }\to 0\text{for some}\delta <0.5,\end{array}$$

whenever μ _k ≤1 for all k and if $lim\underset{k}{inf}\frac{{\mu }_{k+r}}{k}>0$for every integer r. ▀ 

Stationary distribution and a simplified ODE

We will show below that the RHS of the ODE (7) is same as that of the ODE (5) and hence equate the ODE (7) with a more tractable ODE (5). As a first step, we prove that the Markov chain {J _k } has a stationary distribution for any given DFE, channel pair (Z,Θ). In the following, at many places we do not include channel value Z for notation, as this article mainly works with fixed channel behavior. However the proofs are applicable for any pair (Z,Θ) and the notation includes Z, when required to be specific.

Theorem 2

The following results hold:

• (i) For every fixed (Z,Θ), Markov chain {J _k }has a unique stationary distribution π _Z,Θ.

• (ii) Starting from any initial condition (i,y), the n-step transition measure ($P_{\Theta }^n\left(.\right|i,y)$) of the Markov chain converges geometrically to the stationary distribution, Π _Θ , in total variation norm.

• (iii) The continuous part of the stationary distribution has a density, Π _Θ that is continuous with respect to (Z,Θ)in L ₁norm.

• (iv) The MSE under stationarity is continuous in (Z,Θ).

Proof: Please see Appendix 1. ▀ 

For each Θ, {J _k } is a Markov chain taking values in ${\mathcal{S}}^{N_L}\times {\mathcal{S}}^{N_b}\times {\mathcal{R}}^{N_f}$. Its transition function

$$\begin{array}{ll}{P}_{\Theta }^1(i,y\in B|j,y^{\prime })=& \tilde{\delta }(i,j)\bar{\delta }(y,y^{\prime })P\left(i_1\right)P(y_1\in B_{y^{\prime }})\\ \times P_{\Theta }\left(i_{N_L+1}\right|j,y^{\prime }),\end{array}$$

(8)

where $\bar{\delta }(y,y^{\prime })$ equals 1 when the vector formed from all but the last component of the vector y^′equals the vector formed from all but the first component of the vector y and otherwise zero and $\tilde{\delta }(i,j)=\bar{\delta }\left(i_1^{N_L},j_1^{N_L}\right)\bar{\delta }\left(i_{N_L+1}^{N_b+N_L},j_{N_L+1}^{N_b+N_L}\right)$ (note that the first component, $i_1^{N_L}$, represents the sample value of S _k , while the second one, $i_{N_L+1}^{N_b+N_L}$, represents the sample value of ${Ŝ}_{k-1}$). The only component of the transition function (8) that depends upon Θ is $P_{\Theta }(i_{N_L+1}=1|j,y^{\prime })=1_{\left\{e_{\Theta }(j,y^{\prime })>0\right\}}$.

By i.i.d nature of the input s _k and noise n _k one can choose n₀large enough such that the continuous part of the n step transition function $P_{\Theta }^n(i,y\in B|j,y^{\prime })$ is absolutely continuous with respect to $f_{\mathcal{N}}\left(y\right)\mathit{dy}$ for all n≥n₀. Further, n₀ is chosen larger than N _L to ensure ensure s _k , $S_{k-n_0}$ are independent. Fix one such n. The corresponding density (Radon–Nikodym derivative)

$$\begin{array}{l}{p}_{\Theta }^n\left(i,y|j,y^{\prime }\right)=\sum _l{\int }_{v}P\left(S_{k+1}^{k+n}\right){\pi }_{q=1}^n\\ \times P_{\Theta }\left({ŝ}_{k+n-q}\right|x\left(q\right)\left)f_{\mathcal{N}}\right(v)\mathit{dv},\end{array}$$

(9)

where

$$\begin{array}{ll}l& =\left(S_{k+1}^{k+n-N_L},{Ŝ}_k^{k+n-1-N_b}\right),v:=N_{k+1}^{k+n-N_f}/\sigma \text{and}\\ x\left(q\right)& :=\left(S_{k+n-q-N_L+1}^{k+n-q},{Ŝ}_{k+n-q-N_b}^{k+n-q-1},N_{k+n-q-N_f+1}^{k+n-q}\right).\end{array}$$

From (9) it is easy to see that the density of the n-step transition function $p_{\Theta }^n(i,y|j,y^{\prime })\le 1$, for all values of i,y, j,y^′ and n≥n₀. Also, we have by Theorem 2.ii, $\left|p_{\Theta }^n\left(.\right|j,y^{\prime })-{\Pi }_{\Theta }\right|\to 0$ for every value of j,y^′as n→∞ in L₁ norm. Further, the function H _Θ () (given in (3)) can be bounded uniformly by, |H _Θ (G _k )|≤C₁|X _k |² + C₂|X _k | for all Θ in a small neighborhood, for some appropriate constants C₁, C₂. The above bound is square integrable and depends only on {J _k }. Hence Lemma 1 of Appendix 5 is applicable and we have,

$$\begin{array}{ll}\underset{n\to \infty }{lim}\left(P_{\Theta }^n{H}_{\Theta }\right)(j,y^{\prime })& ={\pi }_{\Theta }{H}_{\Theta }\left(G\right)\text{which}\Rightarrow h\left(\Theta \right)\\ =\frac{1}{2}{E}_{\Theta }\left[{\nabla }_{\Theta }{\left[{\Theta }^{t}X-s\right]}^2\right].\end{array}$$

Thus ODE (7) simplifies to ODE (5).

By Theorem 2, MSE is a continuous function of Θ and so by confining our search in (4) to a compact region, we obtain the existence of the WF, DFE-WF. Next we consider the LMS attractors which are now the attractors of ODE (5). The ODE attractors will be zeros of the RHS of (5), while the DFE-WF will be a zero of the gradient (if it exists) of the MSE (the cost in the RHS of (4)). As discussed in Section “The issues and our approach”, these two can be related as in (6) and for comparison of the two zeros, one needs to study, ∇ _Θ Π _Θ , the gradient of the stationary density. That is, to get the connection between an LMS-DFE attractor and the DFE-WF one needs to consider the differentiability of the stationary density.

Differentiability of the stationary density

One can see from Equation (9) that it is difficult to comment on differentiability of the n-step transition density itself. Thus, it is even more difficult to discuss the differentiability of the stationary density. To proceed further with the analysis, we perturb the hard decoder Q such that the n-step transition density and the stationary density become differentiable. Next we show that the LMS attractors and the DFE-WF of this perturbed decoder converge to that of the original decoder as the level of perturbation tends to zero. Finally we study the DFE using these perturbed decoders in Section “LMS attractors versus WF at high SNRs”.

We alter the decoder function Q(x)(of Equation (1)) to,

$$\begin{array}{l}{Q}_{{\epsilon }_0}\left(x\right)=\left\{\begin{array}{cccccc}1,& \text{with}& \text{prob}& 1,& \text{if}& x>{\epsilon }_0,\\ -1,& \text{with}& \text{prob}& 1,& \text{if}& x<-{\epsilon }_0,\\ 1,& \text{with}& \text{prob}& \frac{1}{2}\left[\mathit{cos}\left(\frac{(x-{\epsilon }_0)\Pi }{2{\epsilon }_0}\right)+1\right],& \text{if}& \left|x\right|\le {\epsilon }_0,\end{array}\right.\end{array}$$

(10)

where ε₀ is a small constant. Also, in (10) when |x|≤ε₀, $Q_{{\epsilon }_0}\left(x\right)$ will be taken as −1 when it is not 1. Observe that the perturbed decoder is also a hard decoder. With the perturbed decoder $Q_{{\epsilon }_0}\left(x\right)$, the Θ dependent component of the transition function is,

$$\begin{array}{ll}{P}_{\Theta }^{\left({\epsilon }_0\right)}(i_{N_L+1}=1|j,y^{\prime })=& \frac{1_{\left\{\left|e_{\Theta }(j,y^{\prime })\right|\le {\epsilon }_0\right\}}}{2}\\ \times \left[cos\left(\frac{\left(e_{\Theta }\right(j,y^{\prime })-{\epsilon }_0)\Pi }{2{\epsilon }_0}\right)+1\right]\\ +1_{\left\{e_{\Theta }(j,y^{\prime })\ge {\epsilon }_0\right\}}.\end{array}$$

The partial derivative, $\frac{\partial P_{\Theta }^{\left({\epsilon }_0\right)}(i_{N_L+1}=1|j,y^{\prime })}{\mathrm{\partial \Theta }}$ exists everywhere and equals,

$$\frac{-1_{\left\{\left|e_{\Theta }(j,y^{\prime })\right|\le {\epsilon }_0\right\}}\Pi }{4{\epsilon }_0}sin\left(\frac{\left(e_{\Theta }\right(j,y^{\prime })-{\epsilon }_0)\Pi }{2{\epsilon }_0}\right)\frac{\partial e_{\Theta }(j,y^{\prime })}{\mathrm{\partial \Theta }}.$$

(11)

By the uniform upper bound on the derivative (11) and by the bounded convergence theorem one can see that the n-step transition density (9) (with n≥n₀) becomes differentiable (details are in Appendix 2, Lemma 2) and equals (using the notations of Equation (9)),

$$\begin{array}{ll}\frac{\partial p_{\Theta }^{\left({\epsilon }_0\right),n}}{\mathrm{\partial \Theta }}\left(i,y|j,y^{\prime }\right)=& \sum _l{\int }_v\sum _{m=1}^n{\pi }_{\stackrel{q=1}{q\ne m}}^n{P}_{\Theta }^{\left({\epsilon }_0\right)}\left({ŝ}_{k+n-q}\right|x\left(q\right))\\ \times \frac{\partial P_{\Theta }^{\left({\epsilon }_0\right)}\left({ŝ}_{k+n-m}\right|x\left(m\right))}{\mathrm{\partial \Theta }}\\ P\left(S_{k+1}^{k+n}\right)f_{\mathcal{N}}\left(v\right)\mathrm{dv.}\end{array}$$

(12)

For these perturbed decoders, we show that the stationary density (with respect to $f_{\mathcal{N}}\left(y\right)\mathit{dy}$) also becomes differentiable. Furthermore, using an Implicit function theorem, we get a bound on the norm of this gradient.

Theorem 3

For every ε₀>0, for every Θ₀, the Markov chain {J _k } has a unique stationary distribution, ${\Pi }_{\Theta }^{\left({\epsilon }_0\right)}$. It’s continuous part has a density, ${\Pi }_{\Theta }^{\left({\epsilon }_0\right)}$, that is continuously differentiable with respect to Θ in L₂ norm. Further, for every δ>0 and ${\sigma }_0^2>0$ there exists a constant C<∞ such that for all Θ∈B(Θ₀,δ), ${\sigma }^2\le {\sigma }_0^2$,

$$\begin{array}{ll}{\left|{\nabla }_{\Theta }{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\right|}^2\le & C\left(\sum _{i}P\left(\left|S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}\right.\right.\right.\\ \left.\left.\left.+{\theta }_f^t{N}_k\right|\le {\epsilon }_0\right)+{\sigma }^2\right).\end{array}$$

(13)

Proof: The proof is provided in Appendix 2. ▀ 

We conclude this section by showing that the DFE-WFs and the LMS-DFE attractors of the perturbed decoder converge to that of the original decoder. In the following, let ${\Theta }_n^{\ast }$ and ${\Theta }_n^{\text{LMS}}$ denote the DFE-WF and an LMS-DFE attractor (whose existence at high SNRs with small ε₀ is established at the end of Appendix 4 and hence is assumed in the proof of the following theorem) for perturbation ${{\epsilon }_0}_n$.

Theorem 4

For any σ², for any sequence ${{\epsilon }_0}_n\to 0$, there exists a subsequence ${{\epsilon }_0}_{\mathit{\text{nk}}}\to 0$, a DFE-WF Θ^∗ of the original decoder and an LMS-DFE attractor Θ^LMS of the original decoder, such that,

$${\Theta }_{\mathit{\text{nk}}}^{\ast }\to {\Theta }^{\ast }\text{and}{\Theta }_{\mathit{\text{nk}}}^{\text{LMS}}\to {\Theta }^{\text{LMS}}.$$

Proof: Please see Appendix 3. ▀ 

Thus we can always take the perturbation ε₀in (10) small enough such that the LMS attractors and the DFE-WFs for the perturbed decoder are close enough to the corresponding equalizers for the original decoder. Henceforth, we analyze these perturbed decoders to draw important conclusions.

LMS attractors versus WF at high SNRs

In this section we would like to understand the connection between an LMS attractor and a DFE-WF for a perturbed decoder. Since the former is a zero of the RHS of Equation (5) and the later is the zero of the gradient of the MSE (the cost in the RHS of (4)), we study the connection between the two.

Fix an ε₀>0. With the error defined by, err _Θ (J _k ):=(s _k −e _Θ (J _k ))(note that i defined in the notations in Section “The model and notation” represents, $(S_k,{Ŝ}_{k-1})$, the discrete part of the Markov chain, {J _k }),

$$\begin{array}{l}{\nabla }_{\Theta }{E}_{J_k\left(\Theta \right)}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2\right]\\ {=}^a\sum _i{\nabla }_{\Theta }{E}_{f_{\mathcal{N}}}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\left(J_k\right)\right]\\ {=}^b\sum _i{E}_{f_{\mathcal{N}}}{\nabla }_{\Theta }\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\left(J_k\right)\right]\\ =\sum _i{E}_{f_{\mathcal{N}}}\left[{\nabla }_{\Theta }\left({\text{err}}_{\Theta }{\left(J_k\right)}^2\right){\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\left(J_k\right)\right]\\ +\sum _i{E}_{f_{\mathcal{N}}}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2{\nabla }_{\Theta }{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\left(J_k\right)\right]\\ =E_{J_k\left(\Theta \right)}\left[{\nabla }_{\Theta }\left({\text{err}}_{\Theta }{\left(J_k\right)}^2\right)\right]\\ +\sum _i{E}_{f_{\mathcal{N}}}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2{\nabla }_{\Theta }{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\left(J_k\right)\right].\end{array}$$

(14)

Here equality a follows by the existence of the stationary density ${\Pi }_{\Theta }^{\left({\epsilon }_0\right)}$ with respect to the Gaussian measure $f_{\mathcal{N}}\left(y\right)\mathit{dy}$. Equality b is given by Lemma 2 of Appendix 5. The above equality above equality (14) is true for any ε₀>0 and for any σ². We will show below that the DFE-WF will be close to the limiting LMS-DFE if the second term on the right hand side of (14) is small.

We have assumed that $S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}\ne 0$ at an LMS attractor. By continuity, $S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}\ne 0$ for all Θ in a small neighborhood of the LMS attractor. We can further choose an ε₁ small enough such that

$$0\notin [S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}-{\epsilon }_1,S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}+{\epsilon }_1],$$

for all $(S_k,{Ŝ}_{k-1})$ and for all Θ in a small neighborhood of an LMS attractor. Choose ε₀≤ε₁. By Chebyshev’s inequality, if 0∉[c−ε₀,c + ε₀](for some c) and if n is a Gaussian random variable with mean zero and variance σ², then

$$\begin{array}{l}P(\left|c+n\right|\le {\epsilon }_0)\le P(\left|n\right|\ge min\{\left|c-{\epsilon }_0\right|,\left|c+{\epsilon }_0\right|\left\}\right)\to 0\\ \text{as}{\sigma }^2\to 0.\end{array}$$

Thus, from the upper bound (13) of Theorem 3, for any fixed ε₀≤ε₁,

$$\left|{\nabla }_{\Theta }{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\right|\to 0\text{as}{\sigma }^2\to 0.$$

Thus by Cauchy Schwartz inequality as σ²→0(note err _Θ has all moments),

$$\begin{array}{l}\left|{\nabla }_{\Theta }{E}_{\Theta }\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2\right]-E_{\Theta }\left[{\nabla }_{\Theta }\left({\text{err}}_{\Theta }{\left(J_k\right)}^2\right)\right]\right|\\ =\left|\sum _i{E}_{f_{\mathcal{N}}}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2{\nabla }_{\Theta }{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\left(J_k\right)\right]\right|\\ \le \sum _i{\left(E_{f_{\mathcal{N}}}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^4\right]\right)}^{1/2}\left|{\nabla }_{\Theta }{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\right|\\ \to 0.\end{array}$$

(15)

Next we show the following: (15) implies the LMS-DFE attractors will be close to the DFE-WFs. In general two functions f₁, f₂can be close to each other at every point, but their zeros may be far apart, i.e., if x₁ is a zero of f₁ then f₂(x₁) can be close to zero but the zero of f₂closest to x₁may still be far away. It is useful to rule out this possibility in our scenario. We show this using the following theorem. Define,

$$\begin{array}{l}s(\Theta ,{\sigma }^2):=E_{J_k\left(\Theta \right)}\left[{\nabla }_{\Theta }\left({\text{err}}_{\Theta }{\left(J_k\right)}^2\right)\right]\text{and}\\ w(\Theta ,{\sigma }^2,\eta ):=s(\Theta ,{\sigma }^2)+\mathrm{\eta .}\end{array}$$

Theorem 5

There exists an ε₂with 0<ε₂≤ε₁ such that for any ε₀≤ε₂, there exists a continuous function $q:B(0,\delta )\subset \mathcal{R}\times \mathcal{R}\mapsto {\mathcal{R}}^{N_f}$, with

$$w\left(q\left({\sigma }^2,\eta \right),{\sigma }^2,\eta \right)=0.$$

Proof: Please see Appendix 4. ▀ 

Using the above theorem, we obtain the proximity of LMS attractors and the WFs in the following.

For any fixed ε₀≤ε₂, $\left|{\nabla }_{\Theta }{\Pi }_{\Theta }^{\left({\epsilon }_0\right)}\right|$ near an LMS attractor, tends to zero as σ²→0. Thus by (15), there exists a small enough ${\sigma }_0^2$ such that for all ${\sigma }^2\le {\sigma }_0^2$,

$$\begin{array}{ll}\left|\left({\sigma }^2,{\eta }_w\right)\right|\le \delta ,\text{where}{\eta }_w:=& {\nabla }_{\Theta }{E}_{\Theta }\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2\right]\\ -E_{\Theta }\left[{\nabla }_{\Theta }{\text{err}}_{\Theta }{\left(J_k\right)}^2\right].\end{array}$$

Note that q(σ²,0) is a zero of s(Θ,σ²) (note w(q(σ²,0),σ²,0)=s(Θ,σ)) and hence is an LMS attractor at σ². Similarly from (14), q(σ²,η _w )is a zero of the gradient of MSE cost and hence is a DFE-WF. Thus, for all ${\sigma }^2\le {\sigma }_0^2$, the LMS attractors, q(σ²,0), by continuity arguments of Theorem 5 will be close to that of the WFs, q(σ²,η _w ).

It is clear from the above Theorem that at high SNRs, for very small ε₀(close to the practical decoder), the LMS attractor is close to the DFE-WF. Since, IDFE Θ_IDFE, is designed with an improper assumption (like perfect decisions), there is a good chance of these filters to be inefficient in comparison to the LMS attractors. We will see this in the examples provided in Section “Numerical examples”.

We conclude this section by pointing out another useful consequence of the Theorem 5. This theorem also establishes the existence of the LMS attractors at high SNRs for perturbed decoders with perturbation level ε₀ small. A Remark at the end of Appendix 4 establishes this point.

One of the uses of the above ODE approximation is that, one can approximately obtain the performance (e.g., Bit error rate, MSE) of LMS-DFE at any time by using the trajectory of this ODE. Of course, obtaining bit error rate (BER) theoretically is still a problem because the BER of a system with a fixed known channel and a fixed DFE is still not available. But our ODE approximation is still useful because one can obtain the performance (transient as well as stationary) of the LMS-DFE with only one simulation, which would not be possible otherwise. This is because by Theorem 1, the ODE solution approximates the LMS-DFE trajectory in probability.

Numerical examples

In this section we reinforce the theory developed so far using some examples. We take a few examples of channels obtained from previous studies and show the proximity of the DFE-WF and the LMS attractor for practical values of SNRs. We also show that in many cases, the IDFE performs much worse than the DFE-WF but an LMS attractor performs close to the DFE-WF. BER and the MSE are used to compare the various equalizers. For every sample of the channel, we have used Monte-Carlo simulations to estimate the corresponding BER and MSE using one million samples of data.

DFE-WF, Θ^∗, for every sample of the channel is obtained by running a gradient descent type of algorithm on the cost function (4) itself, where the gradient was approximated at each point by finite difference approximation, [E_{Θ + Δ}(X(Θ + Δ)^t(Θ + Δ)−s]²−E _Θ (X(Θ)^t(Θ)−s)²) /|Δ|. Here the expectation E _Θ (X(Θ)^t(Θ)−s)² is estimated by the sample path averages,

$$\frac{1}{N}\sum _{i=1}^N{\left(X_i^t\left(\Theta \right)\Theta -s_i\right)}^2$$

using a large number of samples, N. Vector sequence ${\left\{X_i\left(\Theta \right)\right\}}_{i=1}^N$ is obtained by running the DFE with fixed coefficients Θ(and on a channel that is fixed at its current sample). Thus Θ^∗ is estimated as the limit of the steepest descent algorithm:

$$\begin{array}{ll}{\Theta }_{k+1}={\Theta }_k& -\frac{{\mu }_k}{N\left|{\Delta }_k\right|}\left[\sum _{i=1}^N\left[{\left(X_{k,i}^t\left({\Theta }_k\right){\Theta }_k-s_{k,i}\right)}^2\right.\right.\\ \left.\left.-{\left(X_{k,i}^t({\Theta }_k+{\Delta }_k)\left({\Theta }_k+{\Delta }_k\right)-s_{k,i}\right)}^2\right]\right].\end{array}$$

Here s_k,i are i.i.d with the distribution of the inputs, s _k . Sequences {Δ _k }and {μ _k }are chosen appropriately to reduce to zero. In our simulations we used ${\mu }_k=\frac{0.07}{k^{0.6}}$, Δ _k =5μ _k and N=4×10⁵.

Least mean square attractors are obtained as the time limit of the LMS algorithm (3), with similar settings as with DFE-WF estimation.

We consider two examples in Tables 1 and 2. In Table 1, we have used an interesting example (significant part of the raised cosine channel of ([1], p. 199)) to show that the LMS attractors are close to the WFs at practical SNRs. Its coefficients are provided in the table. We also provide BER in this table. We further show the euclidean distance between the equalizer and the corresponding DFE-WF in first sub-columns. One can see that the distance between the LMS-DFE and DFE-WF is small while that between the IDFE and DFE-WF is large. One can also see an improvement up to 18% in BER in LMS-DFE in comparison to the IDFE. In fact this improvement is more at high SNRs (where the IDFE is assumed to have lesser problem because of error propagation). Further, the BER of the DFE-WF is close to that of the LMS attractor.

Table 1 Comparison of DFEs for raised cosine channel with N _f = 5, N _b = 10 and channel fixed at [0.45 0.59 0.43 0.11 −0.22 −0.32 −0.27 0 0.11 0.11]

Table 2 Comparison of DFEs with N _f = N _b = 2, and channel fixed at [ 0.41 .82 0.41]

We have developed the theory for an equalizer with delay zero. One can easily extend these results to the equalizer with any arbitrary delay. In fact, the channel in Table2is one such example. Here the equalizer with delay 1 will be the best one. The channel of Table 2 is very widely used (see [1], p. 165 and [4], p. 414). We can see once again a huge improvement (up to 30%) in BER for the LMS-DFE with respect to Θ_IDFE. We also see that the LMS attractors are close to the DFE-WF, Θ^∗for all practical SNRs.

In this section, we are comparing directly the time limits of LMS algorithm (3) with that of the true DFE-WF iteration mentioned at the beginning of this section. These two limits are further compared with IDFE, closed form expression. That the LMS trajectory approximates the solution of the ODE (5) is established theoretically (Theorem 1) in this article. In [9, 18] etc., we have demonstrated the same even via numerical simulations, for time varying channels. In Figures two, three and four of [9], it is shown numerically that the LMS trajectory approximates the appropriate ODE solution when the underlying channel is a time varying AR (2) process.

Tracking analysis

LMS being an iterative algorithm can track the channel variations if the update co-efficients μ _k , in (3), converge to a non zero value. In [9, 18], we study the tracking behavior of an LMS-DFE, while it is operating on a wireless channel characterized by an AR(2) process. We demonstrate that an LMS-DFE can also track the time varying DFE-WF, whose variations result from the variations in a wireless channel. We also show that LMS-DFE can outperform the IDFE, on a time varying channel.

Conclusions

Obtaining MSE optimal filter for DFE is a long-standing problem. Precoding provides one practical solution but may not be feasible with wireless channels. The difficulty in the design and or analysis is because, the analysis of the past decisions (with feedback) is not known so far. To circumvent this, one commonly uses the optimal WF obtained assuming perfect past decisions. LMS, a computationally efficient alternative, is an iterative algorithm designed to converge to the WF. However, once again because of the feedback involved, complete analysis of an LMS-DFE is not available.

We show via ODE analysis, that LMS itself can provide/track the optimal WF. This article concentrates on fixed channel behavior and proves that the attractors of the LMS are close to that of the optimal DFE at high SNRs. Proofs become nontrivial partly because of the non-differentiability of the hard decoder. We circumvent this problem, by studying another hard decoder which is a slightly perturbed version of the original one. We first show that the LMS attractors and the DFE-WFs of the perturbed decoder converge to that of the original decoder and then show that the two themselves are close to each other at high SNRs. Next, we show by examples that the SNRs need not be very high, i.e., in fact practically used SNRs (upto 1.5 dB) can be sufficient. We also show that the BER (probability of error) of the commonly used WF, designed assuming perfect past decisions (also using perfect channel estimates), can be up to 33% higher than the optimal WF even at high SNRs (where the former is believed to be closer to the later).

In [18], we show that the LMS-DFE converges and then moves close to the instantaneous DFE-WF after the initial transience, while it is tracking a DFE-WF of a wireless channel modeled by an AR(2) process. We also show in [18] that the performance measures BER and MSE of the LMS-DFE are close to that of the DFE-WF after the transient period, while that of an IDFE are substantially inferior to that of the DFE-WF and the LMS-DFE.

Thus we conclude: (1) in case of a DFE, an LMS algorithm (originally designed for computational efficiency) converges and/or tracks a filter close to the Wiener solution; (2) the closed form expression for DFE WF (obtained after approximating the decision errors to zero) is far away from the Wiener solution and its performance can be significantly inferior.

Appendices

Appendix 1

Proof of Theorem 2: Using the results of [19], we prove the existence and continuity of the stationary distribution of the Markov chain, {J _k }. For any (Z₀,Θ₀) and for any ε>0, ε₀>0,

$$M_1\triangleq \underset{S_k,{Ŝ}_{k-1},\Theta \in \bar{B}({\Theta }_0,\epsilon ),Z\in \bar{B}(Z_0,{\epsilon }_0)}{min}\left({\Psi }^t{S}_k+{\theta }_b^t{Ŝ}_{k-1}\right).$$

(16)

Continuity of the map considered in (16) and compactness of the closed balls $\bar{B}({\Theta }_0,\epsilon )$, $\bar{B}(Z_0,{\epsilon }_0)$ ensures |M₁|<∞.

The map (Θ,N _k )↦θ _f ^tN _k is continuous and hence the inverse image of the open set {x>−M₁}under this map is open. Thus it is possible to get a open set C and a δ≤ε such that,

$$\left\{(N_k,\Theta ):{\theta }_f^t{N}_k>-M_1\right\}\supset C\times \bar{B}({\Theta }_0,\delta ).$$

(17)

Thus whenever $\Theta \in \bar{B}({\Theta }_0,\epsilon )$ and $Z\in \bar{B}(Z_0,{\epsilon }_0)$, the decoder (1) outputs 1 (irrespective of the inputs/past decisions) when the noise vector is in C. Hence,

$$\begin{array}{ll}P({Ŝ}_k=[1..1\left]\right)& =P\left({\cap }_{l=k-N_b-1}^k\right\{{ŝ}_l=1\left\}\right)\\ \ge P\left({\cap }_{l=k-N_b-1}^k\{N_l\in C\}\right)\\ \ge P(N_{k-N_b-N_f+1}^k\in C_1\times C_2),\end{array}$$

where sets $C_1\in {\mathcal{R}}^{N_b}$, $C_2\in {\mathcal{R}}^{N_f}$ are selected such that their respective Lebesgue measures are not equal to zero and ∩l=k−N _b −1k{N _l ∈C}⊃C₁×C₂.

Define $\mathcal{G}:=\left[1..1\right]\times \left[1..1\right]\times {\mathcal{R}}^{N_f}$. For any n₀>max{N _b + N _f + 1,N _L }, for any initial condition $J_{k-n_0}$, for any measurable set B _N , and for any $\Theta \in \bar{B}({\Theta }_0,\delta )$, $Z\in \bar{B}(Z_0,{\epsilon }_0)$

$$\begin{array}{l}P(J_k\in \{\left[1..1\right]\times \left[1..1\right]\times B_N\left\}\right|J_{k-n_0})\\ =P(S_k=[1..1],{Ŝ}_{k-1}=[1..1],\\ N_k\in B_N\left|J_{k-n_0}\right)\\ \ge P(S_k=[1..1],N_{k-N_b-N_f+1}^k\in C_1\\ \times C_2,N_k\in B_N)\\ \ge \mathrm{\alpha P}(N_k\in B_N\cap C_2)\end{array}$$

where α:=P(S _k =[ 1 .. 1 ])P(N k−N _b + N _f k−N _f ∈C₁). Thus for any $\Theta \in \bar{B}({\Theta }_0,\delta )$, $Z\in \bar{B}(Z_0,{\epsilon }_0)$ and for any initial condition $J_{k-n_0}$, the n₀-step conditional measure is majorized:

$$P_{Z,\Theta }(J_k\in E|J_{k-n_0})\ge {\nu }_{n_0}(E\cap \mathcal{G}),$$

where the measure ${\nu }_{n_0}\left(\right)$ is defined by, ${\nu }_{n_0}\left(\right[1..1]\times [1..1]\times B_E):=\mathrm{\alpha P}(N_k\in B_E\cap C_2)$. Thus the entire state space ${\mathcal{S}}^{N_L}\times {\mathcal{S}}^{N_b}\times {\mathcal{R}}^{N_f}$ is ${\nu }_{n_0}-$small (hence also a petite set) for all the Markov chains {J _k }, parameterized by $\Theta \in \bar{B}({\Theta }_0,\delta )$ and $Z\in \bar{B}(Z_0,{\epsilon }_0)$. Then using ([19], Proposition 9.1.7, p. 206 and Theorem 10.01, p. 230) one obtains the existence and uniqueness of the stationary distribution, π_Z,Θfor each Z,Θ.

Define $\rho =1-{\nu }_{n_0}\left(\mathcal{G}\right)$. Then, by ([19], Theorem 16.2.4 in page 392), for all $\Theta \in \bar{B}({\Theta }_0,\delta )$, $Z\in \bar{B}(Z_0,{\epsilon }_0)$ and for all initial conditions (j,y^′) we get:

$$\left|P_{Z,\Theta }^n\left(.\right|j,y^{\prime })-{\pi }_{Z,\Theta }\right|\le {\rho }^{\frac{n}{n_0}},$$

where |.| represents the total variation norm. This along with the continuity of the transition function, establishes the continuity of the stationary distribution π_Z,Θ under total variation norm at (Z₀,Θ₀). This is because, for any $\Theta \in \bar{B}({\Theta }_0,\delta )$ and $Z\in \bar{B}(Z_0,{\epsilon }_0)$

$$\begin{array}{ll}\underset{(Z,\Theta )\to (Z_0,{\Theta }_0)}{lim}& \left|{\pi }_{Z,\Theta }-{\pi }_{Z_0,{\Theta }_0}\right|\\ \le \underset{(Z,\Theta )\to (Z_0,{\Theta }_0)}{lim}\left[\left|{\pi }_{Z,\Theta }-P_{Z_0,{\Theta }_0}^n\left(.\right|j,y^{\prime })\right|\right.\\ +\left|{\pi }_{Z_0,{\Theta }_0}-P_{Z_0,{\Theta }_0}^n\left(.\right|j,y^{\prime })\right|\\ \left.+\left|P_{Z_0,{\Theta }_0}^n\left(.\right|j,y^{\prime })-P_{Z,\Theta }^n(.|j,y^{\prime })\right|\right]\\ \le 2{\rho }^{\frac{n}{n_0}}+\underset{\Theta \to {\Theta }_0}{lim}\left|P_{Z_0,{\Theta }_0}^n(x,.)-P_{Z,\Theta }^n(x,.)\right|\\ =2{\rho }^{\frac{n}{n_0}},\end{array}$$

for all n≥1, where the last equality follows by continuity of the transition function with respect to (Z,Θ). By letting n→∞

$$\underset{(Z,\Theta )\to (Z_0,{\Theta }_0)}{lim}\left|{\pi }_{Z,\Theta }-{\pi }_{Z_0,{\Theta }_0}\right|=0.$$

The stationary distribution, π_Z,Θ, has discrete and continuous components. The continuous component of π_Z,Θ, is absolutely continuous with respect to the measure $f_{\mathcal{N}}\left(y\right)\mathit{dy}$ for every (Z,Θ). Hence the stationary density, π_Z,Θfor {J _k }exists. Continuity in total variation norm of the stationary distribution implies the continuity of the stationary densities in L₁norm ([20], Theorem 8.2, p. 110). It is also easy to see that the stationary density Π_Z,Θ(i,y)≤1 for all (i,y).

Now by fixing the channel at some value Z, MSE, the cost in RHS of Equation (4), can be rewritten as, $E_{\Theta }{\left[{\Theta }^{t}X-s\right]}^2=\sum _{S,Ŝ}{E}_{f_{\mathcal{N}}}\left[{\left({\Theta }^{t}X-s\right)}^2{\Pi }_{\Theta }\right]$. Lemma 1 in Appendix 5, now gives the continuity of the MSE with respect to Θ for any fixed Z.

One can show the same conclusions for the Markov chain, {G _k }, as G _k =Γ(J _k ) for some fixed one-one, onto C^∞ function Γ, whenever the channel and equalizer values are fixed. ▀ 

Appendix 2

Proof of Theorem 3: The existence and continuity of the stationary density ${\Pi }_{\Theta }^{\left({\epsilon }_0\right)}$ for every ε₀is achieved in a similar way as in the proof of the Theorem 2. The only difference being, ε₀ must be added to −M₁ in the definition of the set (17). We leave superscript ε₀to simplify the notations in the rest of this proof.

We use Implicit function theorem to prove differentiability. For that, we will consider the Banach spaces:

• $X={\mathcal{R}}^{N_f+N_b}$ with Euclidean norm.

• $Y=\{g:{\mathcal{S}}^{N_L+N_b}\times X\to \mathcal{R};\left|g\right|<\infty \}$ with L₂ norm, |.|, defined by,

  $$\left|g\right|:=\frac{1}{\left|\mathcal{S}\right|}\sum _i{\left({\int }_y{\left|g(i,y)\right|}^2{f}_{\mathcal{N}}\left(y\right)\mathit{dy}\right)}^{1/2},$$

  where $\left|\mathcal{S}\right|$ represents the cardinality of set ${\mathcal{S}}^{N_L+N_b}$.

Fix n₀>max{N _f + N _b ,N _L }. We consider the following continuous map f:X×Y↦Y,

$$f(\Theta ,\Pi )=g(\Theta ,\Pi )-\Pi +\left(\sum _j{\int }_{y^{\prime }}\Pi (j,y^{\prime })f_{\mathcal{N}}\left(y^{\prime }\right){\mathit{dy}}^{\prime }-1\right),$$

where,

$$g(\Theta ,\Pi )(i,y):=\sum _j{\int }_{y^{\prime }}{p}_{\Theta }^{n_0}(i,y|j,y^{\prime }\left)\Pi \right(j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime })d{y}^{\prime }.$$

Observe that (Θ,Π _Θ ) is a zero of f.

By Lemmas 1 and 2 the function f is differentiable with respect to Π and Θ, respectively and further the derivative $\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}$ is a homeomorphism. Also, $\left|{\left(\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}\right)}^{-1}\right|$ and $\left|\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|$ are upper bounded locally by the RHS of (18) and (24) respectively.

Using similar logic one can easily show that both the partial derivatives of f are continuous in (Θ,Π). Hence by Implicit function theorem on Banach spaces, ([21], Theorem 3.1.10 and Corollary 3.1.11, p. 115), the map Θ↦Π _Θ is continuously differentiable and the derivative is given by,

$${\nabla }_{\Theta }{\Pi }_{\Theta }=-{\left[{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}\right|}_{(\Theta ,{\Pi }_{\Theta })}\right]}^{-1}{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{(\Theta ,{\Pi }_{\Theta })}.$$

Upper bound 13 is obtained by bounding the above gradient using the upper bounds (18) and (24). ▀ 

Lemma 1.

f is differentiable with respect to Π and the derivative is a homeomorphism. Also for any $\delta >0,{\sigma }_0^2>0$ there exists a constant C₀<∞ such that,

$$\left|{\left[{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}\right|}_{(\Theta ,{\Pi }_{\Theta })}\right]}^{-1}\right|\le C_0$$

(18)

for all Θ∈B(Θ₀,δ), ${\sigma }^2\le {\sigma }_0^2$.

Proof: The function f is affine linear in the second variable Π∈Y. Thus,

$${\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}\right|}_{(\Theta ,\widehat{\Pi })}\left(\Pi \right)=g(\Theta ,\Pi )-\Pi +\left(\sum _j{\int }_{y^{\prime }}{\Pi }_{(}j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime }){\mathit{dy}}^{\prime }\right).$$

(19)

We will show below that this map is one-one through contradiction. It is easy to see that g(Θ,Π)−Π is in the set,

$$\mathcal{H}:=\left\{\Pi :\sum _j{\int }_{y^{\prime }}{\Pi }_{(}j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime })d{y}^{\prime }=0\right\}\subset \mathrm{Y.}$$

Operator, Γ that maps $\Pi \mapsto \left(\sum _j{\int }_{y^{\prime }}{\Pi }_{(}j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime })d{y}^{\prime }\right)$, i.e.,

$$\begin{array}{rl}\Gamma :Y& \to Y\\ \Pi & \mapsto \Gamma \left(\Pi \right)\text{with}\\ \Gamma \left(\Pi \right)(i,y):& =\sum _j{\int }_{y^{\prime }}{\Pi }_{(}j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime }\left)d{y}^{\prime }\text{for all}\right(i,y).\end{array}$$

has one-dimensional range which lies inside ${\mathcal{H}}^c$. We can show that the partial derivative (19) is one-one, if we show that there is no common non-zero vector in the null space of both the operators. Say there exists a vector Π≠0 in the null space of both the operators. Let,

$$\begin{array}{rl}D& :=\left\{(i,y):\Pi (i,y)\ge 0\right\},\\ \alpha & :=\sum _j{\int }_{\{y^{\prime }:(j,y^{\prime })\in D\}}{\Pi }_{(}j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime })d{y}^{\prime },\\ {\left|\Pi \right|}_1& :=\sum _j{\int }_{y^{\prime }}\left|{\Pi }_{(}j,y^{\prime })\right|f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }.\end{array}$$

As Π is in the null space of the operator, Γ,

$$\sum _j{\int }_{\{y^{\prime }:(j,y^{\prime })\in D^c\}}{\Pi }_{(}j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime })d{y}^{\prime }=-\mathrm{\alpha .}$$

Hence |Π|₁=2α. Also, because g(Θ,Π)=Π,

$$\begin{array}{ll}g(\Theta ,\Pi )(i,y)& \ge 0\text{for all}i,y\in D\text{and}\\ g(\Theta ,\Pi )(i,y)& <0\text{for all}i,y\in D^c.\end{array}$$

Then,

$$\begin{array}{l}{\left|g(\Theta ,\Pi )\right|}_1\\ =\sum _i{\int }_{y:(i,y)\in D}g(\Theta ,\Pi )(i,y)f_{\mathcal{N}}\left(y\right)\mathit{dy}-\sum _i{\int }_{y:(i,y)\in D^c}g(\Theta ,\Pi )(i,y)f_{\mathcal{N}}\left(y\right)\mathit{dy}\\ =\sum _i{\int }_{y:(i,y)\in D}\sum _j{\int }_{y^{\prime }}{p}_{\Theta }^{n_0}(i,y|j,y^{\prime }\left)\Pi \right(j,y^{\prime }\left)f_N\right(y^{\prime }\left)d{y}^{\prime }{f}_{\mathcal{N}}\right(y)\mathit{dy}\\ -\sum _i{\int }_{y:(i,y)\in D^c}\sum _j{\int }_{y^{\prime }}{p}_{\Theta }^{n_0}(i,y|j,y^{\prime }\left)\Pi \right(j,y^{\prime }\left)f_N\right(y^{\prime }\left)d{y}^{\prime }{f}_N\right(y)\mathit{dy}\\ \stackrel{\text{Fubini}}{=}\sum _j{\int }_{y^{\prime }}\left(\sum _i{\int }_{y:(i,y)\in D}{p}_{\Theta }^{n_0}(i,y|j,y^{\prime }\left)f_{\mathcal{N}}\right(y)\mathit{dy}\right)\Pi (j,y^{\prime })f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }\\ -\sum _j{\int }_{y^{\prime }}\left(\sum _i{\int }_{y:(i,y)\in D^c}{p}_{\Theta }^{n_0}(i,y|j,y^{\prime }\left)f_{\mathcal{N}}\right(y)\mathit{dy}\right)\Pi (j,y^{\prime })f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }\\ =\sum _j{\int }_{y^{\prime }}{P}_{\Theta }^{n_0}(i,y\in D|j,y^{\prime }\left)\Pi \right(j,y^{\prime }\left)f_{\mathcal{N}}\right(y^{\prime })d{y}^{\prime }-\sum _j{\int }_{y^{\prime }}{P}_{\Theta }^{n_0}(i,y\in D^c|j,y^{\prime })\Pi (j,y^{\prime })f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }\\ =\sum _j{\int }_{y^{\prime }}\left(P_{\Theta }^{n_0}(i,y\in D|j,y^{\prime })-P_{\Theta }^{n_0}(i,y\in D^c|j,y^{\prime })\right)\Pi (j,y^{\prime })f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }\\ =\sum _j{\int }_{y^{\prime }:j,y^{\prime }\in D}\left(1-2P_{\Theta }^{n_0}(i,y\in D^c|j,y^{\prime })\right)\left|\Pi (j,y^{\prime })\right|f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }+\sum _j{\int }_{y^{\prime }:j,y^{\prime }\in D^c}\left(1-2P_{\Theta }^{n_0}(i,y\in D|j,y^{\prime })\right)\left|\Pi (j,y^{\prime })\right|f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }\\ \le \sum _j{\int }_{y^{\prime }:j,y^{\prime }\in D}\left(1-2{\nu }_{n_0}\left(D^c\right)\right)\left|\Pi (j,y^{\prime })\right|f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }+\sum _j{\int }_{y^{\prime }:j,y^{\prime }\in D^c}\left(1-2{\nu }_{n_0}\left(D\right)\right)\left|\Pi (j,y^{\prime })\right|f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }\\ =\frac{{\left|\Pi \right|}_1}{2}\left(2-2{\nu }_{n_0}\left(D^c\right)-2{\nu }_{n_0}\left(D\right)\right)\\ ={\left|\Pi \right|}_1(1-{\nu }_{n_0}(Y\left)\right)<{\left|\Pi \right|}_1.\end{array}$$

This provides a contradiction as $0<{\nu }_{n_0}\left(Y\right)<1$ and hence |Π|₁=|g(Θ,Π)|₁<|Π|₁. This proves that the partial derivative (19) is one-one. The inequality is obtained by using the majorizing measure, ${\nu }_{n_0}\left(.\right)$, defined in the proof of continuity of stationary distribution.

The map g(Θ,Π)is compact integral operator ([22], Example 2, p. 277). The last map of the partial derivative has one-dimensional range and hence is compact. Therefore, the partial derivative equals T−I, where T is a compact operator. Then by Riesz–Schauder Theory ([22], Theorem 1, p. 283), the fact that $\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}$ is one-one implies that it is onto and also further that the inverse is bounded. Hence $\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}$ is a linear homeomorphism.

Furthermore, the mapping $({\sigma }^2,\Theta )\mapsto \left|{\left[{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Pi }}\right|}_{(\Theta ,{\Pi }_{\Theta })}\right]}^{-1}\right|$ is continuous. This continuity follows by the joint continuity of the n₀-step transition function, $p_{\Theta }^{n_0}(i,y|j,y^{\prime })$ with respect to (σ²,Θ) and then by bounded convergence theorem (as $p_{\Theta }^{n_0}(i,y|j,y^{\prime })+1$ is uniformly bounded) and finally by the continuity of the map x↦x⁻¹ ([23], p. 135). Hence the lemma follows for some C₀<∞, $\delta >0,{\sigma }_0^2>0$. ▀ 

Lemma 2.

f is differentiable with respect to Θ. The partial derivative ${\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{(\Theta ,{\Pi }_{\Theta })}$ is upper bounded by bound (24).

Proof: We reintroduce the notations that will be used here (notation of Equation (9)).

• $i=\left[S_{k+n_0-(N_f+L-2)}^{k+n_0},{Ŝ}_{k+n_0-1-(N_b-1)}^{k+n_0-1}\right]$, $y=N_{k+n_0-(N_f-1)}^{k+n_0}$, represent the current state of the Markov Chain, at k + n₀.

• $j=\left[S_{k-(N_f+L-2)}^k,{Ŝ}_{k-(N_b-1)}^{k-1}\right],y^{\prime }=N_{k-(N_f-1)}^k$ represent the initial condition for n₀−step transition function, which transition function, which is the state of the Markov chain at k.

• $l=\left[S_{k+1}^{k+n_0-(N_f+L-3)},{Ŝ}_k^{k+n_0-1-(N_b-2)}\right]$, v=N k + 1k + n₀−(N _f −2)represent the intermediate input, decision and noise vectors.

• $x\left(q\right):=\left[S_{k+n_0-q-(N_f+L-1)}^{k+n_0-q},{Ŝ}_{k+n_0-q-N_b}^{k+n_0-q-1},N_{k+n_0-q-N_f}^{k+n_0-q}\right]$ represent the intermediate state of the Markov chain at k + n₀−q.

To begin with, we will show component wise differentiability of the function f, i.e., differentiability of f(Θ,Π)(i,y) for every (i,y). We will show the differentiability of the n₀−step transition function, $p_{\Theta }^{n_0}\left(i,y|j,y^{\prime }\right)$ along with that. Positive and finite constants (like c, c^′′etc) are introduced in the derivations as and when required. While obtaining upper bounds we have taken advantage of the finite alphabet nature of the set $\mathcal{S}$. By simple computations, one can see that the density with respect to the Gaussian measure is,

$$\begin{array}{ll}{p}_{\Theta }^{n_0}\left(i,y|j,y^{\prime }\right)=\sum _l& {\int }_v{\pi }_{q=1}^{n_0}{P}_{\Theta }\left({ŝ}_{k+n_0-q}\right|x\left(q\right))\\ \times P\left(S_{k+1}^{k+n_0}\right)f_{\mathcal{N}}\left(v\right)\mathrm{dv.}\end{array}$$

(20)

Hence,

$$\begin{array}{ll}f(\Theta ,\Pi )(i,y)=& \sum _{l,j}{\int }_{v,y^{\prime }}{\pi }_{q=1}^{n_0}{P}_{\Theta }\left({ŝ}_{k+n_0-q}\right|x\left(q\right)\left)P\right(S_{k+1}^{k+n_0})\\ \times \Pi (j,y^{\prime })f_{\mathcal{N}}\left(v\right)f_{\mathcal{N}}\left(y^{\prime }\right)\mathit{dvdy}\\ -\Pi (i,y)+\left(\sum _j{\int }_{y^{\prime }}\Pi (j,y^{\prime })f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }-1\right).\end{array}$$

The only component of the above functions, depending upon Θ is $P_{\Theta }\left({ŝ}_{k+n_0-q}\right|j,y^{\prime })$. By (11),

$$\begin{array}{l}\left|\frac{\partial P_{\Theta }}{\mathrm{\partial \Theta }}\left({ŝ}_{k+n_0-q}\left|x\right(q)\right)\right|\le c\left|N_{k+n_0-q-N_f}^{k+n_0-q}\right|,\end{array}$$

uniformly in (i,y), for every Θ, (j,y^′) and for every q. Thus for any Θ _h in a small neighborhood of 0 and for any i,y, j,y^′, Θ and q (by mean value ([24], Theorem X.4.5, p.312)),

$$\begin{array}{l}\left|P_{\Theta }\left({ŝ}_{k+n_0-q}\right|x\left(q\right))-P_{\Theta +{\Theta }_h}({ŝ}_{k+n_0-q}\left|x\right(q\left)\right)\right.\\ \left.-{\Theta }_h^t\frac{\partial P_{\Theta }}{\mathrm{\partial \Theta }}\left({ŝ}_{k+n_0-q}\left|x\right(q)\right)\right|\\ \le \left|P_{\Theta }\left({ŝ}_{k+n_0-q}\right|j,y^{\prime })-P_{\Theta +{\Theta }_h}({ŝ}_{k+n_0-q}|j,y^{\prime })\right|\\ +\left|{\Theta }_h^t\frac{\partial P_{\Theta }}{\mathrm{\partial \Theta }}\left({ŝ}_{k+n_0-q}|j,y^{\prime }\right)\right|\\ \le 2c\left|{\Theta }_h\right|\left|N_{k+n_0-q-N_f}^{k+n_0-q}\right|.\end{array}$$

(21)

For obtaining the above upper bound, the mean value theorem is used as explained below for a two dimensional function, which can easily be generalized to any n-dimensional function. Say f is any function of two variables. One can write, f(a + h,b + k)−f(a,b) as sum of terms f(a + h,b + k)−f(a + h,b)−f(a,b + k) + f(a,b), f(a + h,b)−f(a,b) and f(a,b + k)−f(a,b). The first term is bounded by mean value theorem for two variables, ([23], Theorem 9.40, p.235), while the remaining two terms can be bounded using mean value theorem for one variable, ([23], Theorem 5.10, p.108).

Finally by dominated convergence theorem, we will obtain the existence of the following partial derivatives, in the paragraphs that follow:

$$\begin{array}{ll}\frac{\partial p_{\Theta }^{n_0}}{\mathrm{\partial \Theta }}\left(i,y|j,y^{\prime }\right)& =\sum _l{\int }_v\frac{\partial \left({\pi }_{q=1}^n{P}_{\Theta }\left({ŝ}_{k+n_0-q}\right|x\left(q\right))\right)}{\mathrm{\partial \Theta }}\\ \times P\left(S_{k+1}^{k+n_0}\right)f_{\mathcal{N}}\left(v\right)\mathit{dv},\end{array}$$

(22)

$$\begin{array}{ll}{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{\Theta ,\Pi }(i,y)& =\sum _j{\int }_{y^{\prime }}\frac{\partial p_{\Theta }^{n_0}}{\mathrm{\partial \Theta }}\left(i,y|j,y^{\prime }\right)\Pi (j,y^{\prime })f_{\mathcal{N}}\left(y^{\prime }\right)d{y}^{\prime }.\end{array}$$

(23)

For obtaining the partial derivative (23), we will need to study the following set of functions one for each different value of i,j,l,r

$$\begin{array}{l}{\int }_{v,y^{\prime }}\left(P_{\Theta }\left({ŝ}_{k+n_0-r}\right|x\left(r\right))-P_{\Theta +{\Theta }_h}({ŝ}_{k+n_0-r}\left|x\right(r\left)\right)\right.\\ \left.-{\Theta }_h^t\frac{\partial P_{\Theta }}{\mathrm{\partial \Theta }}\left({ŝ}_{k+n_0-r}\left|x\right(r)\right)\right)\\ {\pi }_{q=1,q\ne r}^{n_0}{P}_{\Theta }\left({ŝ}_{k+n_0-q}\right|x\left(q\right))\\ \times \Pi (j,y^{\prime })f_{\mathcal{N}}\left(v\right)f_{\mathcal{N}}\left(y^{\prime }\right)\mathrm{dvdy.}\end{array}$$

One can easily see from (21) that the function inside each of the above integral is dominated by some constant multiple of the integrable function,

$$\begin{array}{l}\left|N_{k+n_0-r-N_f}^{k+n_0-r}\right|\left|\Pi (j,y^{\prime })\right|,\end{array}$$

and that the above bound is is integrable by Cauchy Schwartz inequality. So by dominated convergence theorem, the limit $\underset{\left|{\Theta }_h\right|\to 0}{lim}$ (which arises while defining the partial derivate) can be taken inside the integral for every i,j,l,r and this establishes the existence of component wise partial derivative, ${\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{\Theta ,\Pi }(i,y)$. Also, this component wise partial derivative, is uniformly upper bounded by,

$$\begin{array}{l}\left|{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{\Theta ,\Pi }(i,y)\right|\le \frac{c^{\mathrm{\prime \prime }}}{{\epsilon }_0}\left[\left|y\right|+1\right]\text{for all}(i,y),\text{all}\Theta ,\end{array}$$

where the constant c^′′depends on |Π|.

One can now prove the existence of the overall partial derivative $\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}$ at every (Θ,Π)using the above upper bound and the dominated convergence theorem (in L₂ norm). Consider the limit,

$$\begin{array}{ll}\underset{\left|{\Theta }_h\right|\to 0}{lim}& \frac{1}{\left|{\Theta }_h\right|}\sum _i{\int }_y\left|f(\Theta ,{\Pi }_{\Theta })(i,y)-f(\Theta +{\Theta }_h,{\Pi }_{\Theta })(i,y)\right.\\ {\left.-{\Theta }_h^t{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{\Theta ,{\Pi }_{\Theta }}\left(i,y\right)\right|}^2{f}_{\mathcal{N}}\left(y\right)\mathit{dy}\\ =\sum _i{\int }_y\underset{\left|{\Theta }_h\right|\to 0}{lim}\frac{1}{\left|{\Theta }_h\right|}\left|f(\Theta ,{\Pi }_{\Theta })(i,y)\right.\\ -f(\Theta +{\Theta }_h,{\Pi }_{\Theta })(i,y)\\ {\left.-{\Theta }_h^t{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{\Theta ,{\Pi }_{\Theta }}\left(i,y\right)\right|}^2{f}_{\mathcal{N}}\left(y\right)\mathit{dy}\\ =0.\end{array}$$

The first equality follows because the function inside the integral tends to zero at every point and is upper bounded by the following integrable function,

$$\begin{array}{l}\frac{c^{\prime }}{{\epsilon }_0}\left[\left|y\right|+1\right]1_{\left\{\left|{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{\Theta ,\Pi }(i,y)\right|\le 1\right\}}\\ +{\left(\frac{c^{\mathrm{\prime \prime }}}{{\epsilon }_0}\left[\left|y\right|+1\right]\right)}^2{1}_{\left\{\left|{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{\Theta ,\Pi }(i,y)\right|>1\right\}}.\end{array}$$

We will now upper bound this partial derivative for all (Θ,Π _Θ ). First observe that, because Π _Θ (i,y)≤1for all (i,y), from (11),

$$\begin{array}{l}\left|{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{(\Theta ,{\Pi }_{\Theta })}(i,y)\right|\\ =\left|\sum _{l,j}{\int }_{y^{\prime },v}\frac{\partial \left({\pi }_{q=1}^n{P}_{\Theta }\left({ŝ}_{k+n_0-q}\right|x\left(q\right))\right)}{\mathrm{\partial \Theta }}\right.\\ \left.\times P\left(S_{k+1}^{k+n_0}\right){\Pi }_{\Theta }(j,y^{\prime })f_{\mathcal{N}}\left(v\right)f_{\mathcal{N}}\left(y^{\prime }\right)\mathit{dvd}{y}^{\prime }\right|\\ \le \left(c_1\sum _{r=1}^n\sum _{l,j}{\int }_{v,y^{\prime }}{1}_{\left\{\left|e_{\Theta }\left(x\right(r\left)\right)\right|\le {\epsilon }_0\right\}}{f}_{\mathcal{N}}\left(v\right)f_{\mathcal{N}}\left(y^{\prime }\right)\mathit{dvd}{y}^{\prime }\right)\\ +c_2\left|y\right|+c_{3}E\left(\left|N_k\right|\right),\end{array}$$

for some appropriate constants c₁,c₂,c₃. Then using ${\left(\sum _{k=1}^n{a}_k\right)}^2\le n\sum _{k=1}^n{a}_k^2$, |x|²≤|x| (when |x|≤1), we get,

$$\begin{array}{l}{\left|{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{(\Theta ,{\Pi }_{\Theta })}\right|}^2={\left(\sum _i{\left({\int }_y{\left|{\left.\frac{\mathrm{\partial f}}{\mathrm{\partial \Theta }}\right|}_{(\Theta ,{\Pi }_{\Theta })}(i,y)\right|}^2\frac{f_{\mathcal{N}}}{\left|\mathcal{S}\right|}\left(y\right)\mathit{dy}\right)}^{1/2}\right)}^2\\ \le c_1^{\prime }\sum _{r=1}^n\sum _{l,j,i}{\int }_{v,y^{\prime },y}{1}_{\left\{\left|e_{\Theta }\left(x\right(r\left)\right)\right|\le {\epsilon }_0\right\}}{f}_{\mathcal{N}}\left(v\right)f_{\mathcal{N}}\left(y^{\prime }\right)f_{\mathcal{N}}\left(y\right)\mathit{dvd}{y}^{\prime }\mathit{dy}\\ +c_2^{\prime }\sum _i{\int }_y{\left|y\right|}^2{f}_{\mathcal{N}}\left(y\right)\mathit{dy}+c_3^{\prime }{\left[E\left(\left|N_k\right|\right)\right]}^2\\ =c_1^{\mathrm{\prime \prime }}\sum _{l}P\left(\left\{\left|b_l+{\theta }_f^t{N}_k\right|\le {\epsilon }_0\right\}\right)+c_2^{\mathrm{\prime \prime }}{\sigma }^2,\end{array}$$

(24)

where the constants b _l take values $S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}$. ▀ 

Appendix 3

Proof of Theorem 4: Let $f_1(\Theta ,{\epsilon }_0):=E_{J_k\left(\Theta \right)}^{\left({\epsilon }_0\right)}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2\right],$ and $f_2(\Theta ,{\epsilon }_0):=\left|E_{J_k\left(\Theta \right)}^{\left({\epsilon }_0\right)}{\nabla }_{\Theta }\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2\right]\right|.$ Note that for any fixed ε₀, LMS attractors will be the zeros, i.e., minima of f₂(.,ε₀) while the DFE-WFs are the minima of the MSE cost, f₁(.,ε₀).Also note that ε₀=0corresponds to the original decoder.

Let $\left\{{{\epsilon }_0}_n\right\}$ be any sequence converging to 0. Let $\Omega =\left\{{{\epsilon }_0}_n\right\}$. Take a compact set C large enough such that the WF is inside it (as Θ is increased to infinity, eventually MSE will start increasing and will tend to infinity). One can follow steps as in Theorem 2 and show that the stationary density ${\Pi }_{{\Theta }_n}^{{{\epsilon }_0}_n}$ converges to ${\Pi }_{\Theta }^0$ as $({{\epsilon }_0}_n,{\Theta }_n)\to (0,\Theta )$. Similarly, one can also show that both functions f₁,f₂are jointly continuous in (Θ,ε₀)∈C×Ω.

The domain of the parameter Θ for every ε₀, say D(ε₀), is the same compact set C and hence the correspondence ε₀↦D(ε₀)is compact and continuous [25]. Then by the maximum theorem ([25], p. 235), ${D_1}_n^{\ast }:=\text{arg}{\text{min}}_{\Theta \in C}{f}_1(\Theta ,{{\epsilon }_0}_n)$ and ${D_2}_n^{\ast }:=\text{arg}{\text{min}}_{\Theta \in C}{f}_2(\Theta ,{{\epsilon }_0}_n)$ are compact valued upper semi-continuous correspondences on Ω. Thus by ([25], Proposition 9.8, p. 231) there exists a subsequence of LMS attractors ${\Theta }_{n_k}^{\text{LMS}}$ converging to an LMS attractor of the original decoder, ${\Theta }_0^{\text{LMS}}$. Once again by the same proposition there exists a further subsequence such that the DFE-WFs ${\Theta }_{{n_k}_l}^{\ast }$ converge to a DFE-WF of the original decoder, ${\Theta }_0^{\ast }$. Thus there exists a sequence (after renaming) ${{\epsilon }_0}_n\to 0$ such that ${\Theta }_n^{\text{LMS}}\to {\Theta }_0^{\text{LMS}}$ and ${\Theta }_n^{\ast }\to {\Theta }_0^{\ast }$. ▀ 

Appendix 4

Proof of Theorem 5: We have assumed in Section “The model and notation”, that $S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}\ne 0$ for all values of $S_k,{Ŝ}_{k-1}$ at an LMS attractor. By continuity, this implies the same (in fact the sign of the term, $S_k^t\Psi +{\theta }_b^t{Ŝ}_{k-1}$, for each $S_k,{Ŝ}_{k-1}$ remains same) in a small neighborhood of the LMS attractor. Thus, when σ² = 0 (the noiseless case), for the original decoder at an LMS attractor (call it ${\Theta }_0^{\ast }$), we have,

$$\frac{\partial P_{\Theta }}{\mathrm{\partial \Theta }}(i,y|j,y^{\prime })=0\text{for all}(i,y\left)\right(j,y^{\prime }).$$

Thus following the steps as in the proof of Theorem 3 (whose proof is obtained in Appendix 2) one can show that the gradient of the stationary density, ∇ _Θ Π _Θ exists and equals zero at ${\Theta }_0^{\ast }$. Hence at ${\Theta }_0^{\ast }$ (LMS attractor of the original decoder at σ² = 0),

$${\nabla }_{\Theta }{E}_{J_k\left(\Theta \right)}\left[{\text{err}}_{\Theta }{\left(J_k\right)}^2\right]=E_{J_k\left(\Theta \right)}\left[{\nabla }_{\Theta }\left({\text{err}}_{\Theta }{\left(J_k\right)}^2\right)\right].$$

Therefore in this case, the DFE-WF coincides with the LMS-DFE attractor, ${\Theta }_0^{\ast }$.

Choose ${{\epsilon }_0}_2>0$ such that, ${\epsilon }_{0_2}<\left|(S_k^t{\Psi }_0^{\ast }+{Ŝ}_{k-1}^t{\Theta }_{0_b}^{\ast }\right|$ for all values of s _k and ${Ŝ}_{k-1}$. The DFE-WF (${\Theta }^{\ast ,{\epsilon }_0}$) and the LMS-DFE attractor (${\Theta }^{\text{LMS},{\epsilon }_0}$) coincide and equal ${\Theta }_0^{\ast }$ for a noiseless system having a perturbed decoder, with ${\epsilon }_0\le {{\epsilon }_0}_2$. This happens because when there is no noise the perturbed decoder coincides with the original decoder for ${\epsilon }_0\le {\epsilon }_{0_2}$.

Fix ${\epsilon }_0\le {{\epsilon }_0}_2$. Then, $w({\Theta }^{\ast ,{\epsilon }_0},0,0)=0$ and the partial derivative,

$$\begin{array}{ll}{\left.\frac{\mathrm{\partial w}}{\mathrm{\partial \Theta }}\right|}_{({\Theta }_{{\epsilon }_0}^{\ast },0,0)}& =\frac{\mathit{ds}}{\mathrm{d\Theta }}=E_{f_{\mathcal{N}}}\left[{\nabla }_{\Theta }\left({\nabla }_{\Theta }\left({\text{err}}_{\Theta }{\left(J_k\right)}^2\right){\Pi }_{\Theta }\left(J_k\right)\right)\right],\\ =E_{\Theta }\left[{\nabla }_{\Theta }{\nabla }_{\Theta }\left({\text{err}}_{\Theta }{\left(J_k\right)}^2\right)\right]=2R_{\mathit{\text{xx}}}\left({\Theta }^{\ast ,{\epsilon }_0}\right),\end{array}$$

where $R_{\mathit{\text{xx}}}\left({\Theta }^{\ast ,{\epsilon }_0}\right)$ is the autocorrelation matrix of the vector X _k (Θ), under stationarity, at ${\Theta }^{\ast ,{\epsilon }_0}$. As ${\Theta }^{\ast ,{\epsilon }_0}$ is a WF, the above partial derivative will be invertible (all the eigenvalues of the derivative should be negative for the equilibrium point to be an attractor).

Continuity of the above partial derivative with respect to σ²,η,Θ can be seen as before. Applying Implicit function theorem at $({\Theta }^{\ast ,{\epsilon }_0},0,0)$, one gets a δ>0, and a continuous function q(σ²,η)such that $q(0,0)={\Theta }^{\ast ,{\epsilon }_0}$ and w(q(σ²,η),σ²,η)=0, for all (σ²,η) with |(σ²,η)|≤δ. ▀ 

Remark on existence of LMS attractors

The above theorem also provides the following useful conclusion. For all σ²≤δ, the zeros of w(.,σ²,0) exist and equal q(σ²,0). These zeros are continuous in σ². One can see that these zeros will indeed be LMS attractors as invertibility of the derivative of the function f() at σ²=0 guarantees its invertibility in a small neighborhood of σ²=0.

Appendix 5

In this appendix we state and prove the lemmas, which are used in this article.

Lemma 3.

Let ${\Pi }_{{\Theta }_n}\stackrel{L_1}{\to }{\Pi }_{\Theta }$. Let |f(Θ _n ,x)|≤g₁(x), |f(Θ _n ,x)|²≤g₂(x) for all n, where g₁,g₂ are integrable functions (with respect to measure μ). Also let f be continuous and $\left|{\Pi }_{{\Theta }_n}\left(x\right)\right|\le C<\infty$ for all x. Then as n→∞,

$$\int f({\Theta }_n,x){\Pi }_{{\Theta }_n}\left(x\right)\mu \left(\mathit{dx}\right)\to \int f(\Theta ,x){\Pi }_{\Theta }\left(x\right)\mu \left(\mathit{dx}\right).$$

Proof: We have

$$\begin{array}{l}\left|\int \left(f({\Theta }_n,x){\Pi }_{{\Theta }_n}\left(x\right)-f(\Theta ,x){\Pi }_{\Theta }\left(x\right)\right)\mu \left(\mathit{dx}\right)\right|\\ \le \int \left|\left(f({\Theta }_n,x){\Pi }_{{\Theta }_n}\left(x\right)-f(\Theta ,x){\Pi }_{\Theta }\left(x\right)\right)\right|\mu \left(\mathit{dx}\right)\\ \le \int \left|f({\Theta }_n,x)\right|\left|{\Pi }_{{\Theta }_n}\left(x\right)-{\Pi }_{\Theta }\left(x\right)\right|\mu \left(\mathit{dx}\right)\\ +\int \left|f({\Theta }_n,x)-f(\Theta ,x)\right|{\Pi }_{\Theta }\left(x\right)\mu \left(\mathit{dx}\right)\\ \le {\left(\int {\left|f({\Theta }_n,x)\right|}^2\mu \left(\mathit{dx}\right)\right)}^{1/2}\\ \times {\left(\int {\left|{\Pi }_{{\Theta }_n}\left(x\right)-{\Pi }_{\Theta }\left(x\right)\right|}^2\mu \left(\mathit{dx}\right)\right)}^{1/2}\\ +\int \left|f({\Theta }_n,x)-f(\Theta ,x)\right|{\Pi }_{\Theta }\left(x\right)\mu \left(\mathit{dx}\right).\end{array}$$

The first term on the right converges to zero because,

$$\begin{array}{l}\left(\int {\left|{\Pi }_{{\Theta }_n}\left(x\right)-{\Pi }_{\Theta }\left(x\right)\right|}^2\mu \left(\mathit{dx}\right)\right)\\ \le 4C^2\left(\int \left|{\Pi }_{{\Theta }_n}\left(x\right)-{\Pi }_{\Theta }\left(x\right)\right|\mu \left(\mathit{dx}\right)\right).\end{array}$$

The second term converges to zero by continuity of the function f(.,.) in Θ and by the bounded convergence theorem, ▀ 

Lemma 4.

Let Π _Θ represent the Radon-Nikodym derivative of measure Π _Θ with respect to the common measure μ for all $\Theta \in {\mathcal{R}}^m$, for some m. Assume Π _Θ ≤1everywhere for all Θ. If ∇ _Θ Π _Θ exists in L₂ norm, then

$${\nabla }_{\Theta }{E}_{\Theta }\left(g\right(\Theta \left)\right)=E_{\mu }{\nabla }_{\Theta }\left(g\right(\Theta \left)\Pi \right(\Theta \left)\right),$$

where g(Θ,.) is square integrable, continuously differentiable in Θ and bounded by a square integrable function uniformly in a neighborhood of Θ.

Proof: Since,

$$\begin{array}{l}\frac{1}{\left|{\Theta }_h\right|}\int \left|g(\Theta +{\Theta }_h){\Pi }_{\Theta +{\Theta }_h}-g\left(\Theta \right){\Pi }_{\Theta }\right.\\ \left.-{\Theta }_h^t\left(g\left(\Theta \right){\nabla }_{\Theta }{\Pi }_{\Theta }-{\nabla }_{\Theta }g\left(\Theta \right){\Pi }_{\Theta }\right)\right|\mu \left(\mathit{dw}\right)\\ \le \frac{1}{\left|{\Theta }_h\right|}\int \left|g\left(\Theta \right)\left({\Pi }_{\Theta +{\Theta }_h}-{\Pi }_{\Theta }-{\Theta }_h^t{\nabla }_{\Theta }{\Pi }_{\Theta }\right)\right|\mu \left(\mathit{dw}\right)\\ +\frac{1}{\left|{\Theta }_h\right|}\int \left|{\Pi }_{\Theta +{\Theta }_h}\left(g(\Theta +{\Theta }_h)-g\left(\Theta \right)\right.\right.\\ \left.\left.-{\Theta }_h^t{\nabla }_{\Theta }g\left(\Theta \right)\right)\right|\mu \left(\mathit{dw}\right)\\ +\frac{1}{\left|{\Theta }_h\right|}\int \left|({\Pi }_{\Theta +{\Theta }_h}-{\Pi }_{\Theta }){\Theta }_h^t{\nabla }_{\Theta }g\left(\Theta \right)\right|\mu \left(\mathit{dw}\right),\end{array}$$

(25)

we will have the result if we show that each of the terms on the right tend to zero as |Θ _h |→0. By Cauchy Schwartz inequality,

$$\begin{array}{l}\underset{\left|{\Theta }_h\right|\to 0}{lim}\frac{1}{\left|{\Theta }_h\right|}\int \left|g\left(\Theta \right)\left({\Pi }_{\Theta +{\Theta }_h}-{\Pi }_{\Theta }-{\Theta }_h^t{\nabla }_{\Theta }{\Pi }_{\Theta }\right)\right|\mu \left(\mathit{dw}\right)\\ \le |g{|}_2\underset{\left|{\Theta }_h\right|\to 0}{lim}\frac{1}{\left|{\Theta }_h\right|}\\ \times {\left(\int {\left|\left({\Pi }_{\Theta +{\Theta }_h}-{\Pi }_{\Theta }-{\Theta }_h^t{\nabla }_{\Theta }{\Pi }_{\Theta }\right)\right|}^2\mu \left(\mathit{dw}\right)\right)}^{1/2}\end{array}$$

The right side tends to zero because the gradient ∇ _Θ Π _Θ exists in L₂norm.

The second term on the right hand side of (25) tends to zero by bounded convergence theorem and mean value theorem (as in Appendix 2), because Π _Θ ≤1everywhere for all Θ and as ∇g is uniformly bounded in a neighborhood of Θ by an integrable function.

The third term of (25) tends to zero by Cauchy Schwartz inequality and by the continuity of the stationary density Π in L₂ norm, and by uniform boundedness of the function (∇ _Θ g)in a neighborhood of Θ by a square integrable function. ▀