A new conjugate gradient algorithm for training neural networks based on a modified secant equation

• Published in: Applied mathematics and computation Vol. 221; pp. 491 - 502
• Main Authors: Livieris, Ioannis E., Pintelas, Panagiotis
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 15.09.2013
• Subjects: Algorithms; Artificial neural networks; Computational efficiency; Computing time; Conjugate gradients; Descent conjugate gradient algorithm; Global convergence; Mathematical analysis; Modified secant equation; Neural networks; Searching; Training; Descent conjugate gradient algorithm; Modified secant equation; Global convergence; Artificial neural networks
• ISSN: 0096-3003; 1873-5649
• DOI: 10.1016/j.amc.2013.06.101

Conjugate gradient methods have been established as excellent neural network training methods, due to the simplicity of their iteration, numerical efficiency and their low memory requirements. In this work, we propose a conjugate gradient neural network training algorithm which guarantees sufficient descent using any line search, avoiding thereby the usually inefficient restarts. Moreover, it approximates the second order curvature information of the error surface with a high-order accuracy by utilizing a new modified secant condition. Under mild conditions, we establish that the global convergence of our proposed method. Experimental results provide evidence that our proposed method is in general superior to the classical conjugate gradient training methods and has a potential to significantly enhance the computational efficiency and robustness of the training process.