A note on the estimation of the generalization error and the prevention of overfitting [machine learning]

• Published in: IEEE International Conference on Neural Networks, 1994 Vol. 1; pp. 321 - 326 vol.1
• Main Authors: Pados, D.A., Papantoni-Kazakos, P.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 1994
• Subjects: Backpropagation algorithms; Contracts; Logic functions; Machine learning; Multilayer perceptrons; Neural networks; Nonhomogeneous media; Radial basis function networks; Supervised learning; Training data
• ISBN: 078031901X; 9780780319011
• DOI: 10.1109/ICNN.1994.374183

Valid generalization and overfitting are closely related issues in the theory of machine learning. In the context of multilayer perceptrons (MLPs) it is usually assumed that improved generalization can be achieved by reducing the functionality of the network. Overfitting is battled by cross-validation methods. A different look on these subjects is proposed by this paper. A Radial-Basis-Function (RBF) network is developed that provides statistically consistent estimates of the expected generalization error induced by arbitrary MLPs. Using the RBF network as a teacher for the MLP, a new batch backpropagation-type learning algorithm is developed that minimizes directly the estimated generalization error. The on-line version of this algorithm calls for a random generalization of the training set. It is concluded that the problem of improved generalization performance is equivalent to the problem of valid generalization of the training data set. Therefore, both problems fall within the statistical context of nonparametric density estimation.< >