On almost-sure bounds for the LMS algorithm

• Published in: IEEE transactions on information theory Vol. 40; no. 2; pp. 372 - 383
• Main Author: Kouritzin, M.A.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.03.1994; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive filters; Algorithms; Convergence; Differential equations; Filtering algorithms; Information theory; Least squares approximation; Mathematics
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/18.312160

Almost-sure (a.s.) bounds for linear, constant-gain, adaptive filtering algorithms are investigated. For instance, under general pseudo-stationarity and dependence conditions on the driving data {/spl psi/k,k=1,2,3,...}, {Y/sub k/,k=0,1,2,...} a.s. convergence and rates of a.s. convergence (as the algorithm gain /spl epsiv//spl rarr/0) are established for the LMS algorithm h/sub k+1//sup /spl epsiv//=h/sub k//sup /spl epsiv//+/spl epsiv/Y/sub k/(/spl psi//sub k+1/-Y/sub k//sup T/h/sub k//sup /spl epsiv//) subject to some nonrandom initial condition h/sub 0//sup /spl epsiv//=h/sub 0/. In particular, defining {g/sub k//sup /spl epsiv//}/sub k=0//sup /spl infin// by g/sub 0//sup /spl epsiv//=h/sub 0/ and g/sub k+1//sup /spl epsiv//=g/sub k//sup /spl epsiv//+/spl epsiv/(E[Y/sub k//spl psi//sub k+1/]-E[Y/sub k/Y/sub k//sup T/]g/sub k//sup /spl epsiv//) for k=0,1,2,..., the author shows that for any /spl gamma/>0 max/sub 0/spl les/k/spl les//spl gamma//spl epsiv/(-1/)|h/sub k//sup /spl epsiv//-g/sub k//sup /spl epsiv//|/spl rarr/0 as /spl epsiv//spl rarr/0 a.s. and under a stronger dependency condition, the author shows that for any 0</spl zeta//spl les/1 and /spl gamma/>0, max/sub 0/spl les/k/spl les//spl gamma//spl epsiv/(-/spl zeta//)|h/sub k//sup /spl epsiv//-g/sub k//sup /spl epsiv//| converges (as /spl epsiv//spl rarr/0) a.s. At a rate marginally slower than O((/spl epsiv//sup 2-/spl zeta//log log(/spl epsiv//sup -/spl zeta//))/sup 1/2 /). Then, under a stronger pseudostationarity assumption it is shown that similar results hold if the sequences {g/sub k//sup /spl epsiv//}/sub k=0//sup /spl infin//,/spl epsiv/>0 in the above results are replaced with the solution g/sup 0/(/spl middot/) of a nonrandom linear ordinary differential equation, i.e. one has, max/sub 0/spl les/k/spl les/[/spl gamma//spl epsiv/(-/spl zeta//])|h/sub k//sup /spl epsiv//-g/sup 0/(/spl epsiv/k)|/spl rarr/0 as /spl epsiv//spl rarr/0 a.s., where one can attach a rate to this convergence under the stronger dependency condition. The almost-sure bounds contained in the paper complement previously developed weak convergence results in Kushner and Shwartz [IEEE Trans. Information Theory, IT-30(2), 177-182, 1984] and, as are "near optimal". Moreover, the proofs used to establish these bounds are quite elementary.< >