Neural net algorithms that learn in polynomial time from examples and queries

• Published in: IEEE transactions on neural networks Vol. 2; no. 1; pp. 5 - 19
• Main Author: Baum, E.B.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.01.1991; Institute of Electrical and Electronics Engineers
• Subjects: Exact sciences and technology; Helium; Humans; Image coding; Intelligent networks; Neural networks; Optical network units; Optical propagation; Other topics in statistical physics, thermodynamics, and nonlinear dynamical systems; Physics; Polynomials; Statistical physics, thermodynamics, and nonlinear dynamical systems; Testing; Workstations; Polynomial; Probability law; Uniform distribution; Backpropagation algorithm; Numerical simulation; Neural network; Learning algorithm
• ISSN: 1045-9227
• DOI: 10.1109/72.80287

An algorithm which trains networks using examples and queries is proposed. In a query, the algorithm supplies a y and is told t(y) by an oracle. Queries appear to be available in practice for most problems of interest, e.g. by appeal to a human expert. The author's algorithm is proved to PAC learn in polynomial time the class of target functions defined by layered, depth two, threshold nets having n inputs connected to k hidden threshold units connected to one or more output units, provided k<or=4. While target functions and input distributions can be described for which the algorithm will fail for larger k, it appears likely to work well in practice. Tests of a variant of the algorithm have consistently and rapidly learned random nets of this type. Computational efficiency figures are given. The algorithm can also be proved to learn intersections of k half-spaces in R/sup n/ in time polynomial in both n and k. A variant of the algorithm can learn arbitrary depth layered threshold networks with n inputs and k units in the first hidden layer in time polynomial in the larger of n and k but exponential in the smaller of the two.< >