Stability Analysis of the Modified Levenberg-Marquardt Algorithm for the Artificial Neural Network Training

• Published in: IEEE transaction on neural networks and learning systems Vol. 32; no. 8; pp. 3510 - 3524
• Main Author: Rubio, Jose de Jesus
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.08.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Artificial neural networks; Biological neural networks; Cost function; Error stability; Learning; Levenberg–Marquardt; Machine learning; Neural networks; Newton; Prediction algorithms; Singularities; Stability analysis; Testing; Training; weights boundedness
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2020.3015200

The Levenberg-Marquardt and Newton are two algorithms that use the Hessian for the artificial neural network learning. In this article, we propose a modified Levenberg-Marquardt algorithm for the artificial neural network learning containing the training and testing stages. The modified Levenberg-Marquardt algorithm is based on the Levenberg-Marquardt and Newton algorithms but with the following two differences to assure the error stability and weights boundedness: 1) there is a singularity point in the learning rates of the Levenberg-Marquardt and Newton algorithms, while there is not a singularity point in the learning rate of the modified Levenberg-Marquardt algorithm and 2) the Levenberg-Marquardt and Newton algorithms have three different learning rates, while the modified Levenberg-Marquardt algorithm only has one learning rate. The error stability and weights boundedness of the modified Levenberg-Marquardt algorithm are assured based on the Lyapunov technique. We compare the artificial neural network learning with the modified Levenberg-Marquardt, Levenberg-Marquardt, Newton, and stable gradient algorithms for the learning of the electric and brain signals data set.