Convergence of batch gradient learning algorithm with smoothing L1/2 regularization for Sigma–Pi–Sigma neural networks

• Published in: Neurocomputing (Amsterdam) Vol. 151; pp. 333 - 341
• Main Authors: Liu, Yan, Li, Zhengxue, Yang, Dakun, Mohamed, Kh.Sh, Wang, Jing, Wu, Wei
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 03.03.2015
• Subjects: Batch gradient learning algorithm; Convergence; Sigma–Pi–Sigma neural networks; Smoothing L1/2 regularization; Sigma–Pi–Sigma neural networks; Batch gradient learning algorithm; Convergence; Smoothing L1/2 regularization
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2014.09.031

Sigma–Pi–Sigma neural networks are known to provide more powerful mapping capability than traditional feed-forward neural networks. The L1/2 regularizer is very useful and efficient, and can be taken as a representative of all the Lq(0<q<1) regularizers. However, the nonsmoothness of L1/2 regularization may lead to oscillation phenomenon. The aim of this paper is to develop a novel batch gradient method with smoothing L1/2 regularization for Sigma–Pi–Sigma neural networks. Compared with conventional gradient learning algorithm, this method produces sparser weights and simpler structure, and it improves the learning efficiency. A comprehensive study on the weak and strong convergence results for this algorithm are also presented, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed value, respectively.