Deep, Big, Simple Neural Nets for Handwritten Digit Recognition

• Published in: Neural computation Vol. 22; no. 12; pp. 3207 - 3220
• Main Authors: Cireşan, Dan Claudiu, Meier, Ueli, Gambardella, Luca Maria, Schmidhuber, Jürgen
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.12.2010; MIT Press Journals, The
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Back propagation; Biological and medical sciences; Computer science; control theory; systems; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects; Handwriting; Inference from stochastic processes; time series analysis; Learning; Learning and adaptive systems; Letters; Mathematics; Mathematics in biology. Statistical analysis. Models. Metrology. Data processing in biology (general aspects); Miscellaneous; Neural networks; Neurons; Pattern Recognition, Automated - methods; Probability and statistics; Recognition; Sciences and techniques of general use; Statistics; Learning; Graphics; Backpropagation; Neural computation; Neuron; Error rate; Benchmarks; Multilayer perceptrons; Neural network; Recognition
• ISSN: 0899-7667; 1530-888X; 1530-888X
• DOI: 10.1162/NECO_a_00052

Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35% error rate on the MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images to avoid overfitting, and graphics cards to greatly speed up learning.