The Tradeoffs of Large-Scale Learning

• Published in: Optimization for Machine Learning p. 351
• Main Authors: Léon Bottou, Olivier Bousquet
• Format: Book Chapter
• Language: English
• Published: United States The MIT Press 30.09.2011; MIT Press
• Subjects: Algorithms; Applied mathematics; Applied psychology; Applied sciences; Applied statistics; Approximation; Artificial Intelligence; Behavioral sciences; Communications; Compromises; Computational complexity; Computer Science; Education; Educational psychology; Error rates; Estimate reliability; Inferential statistics; Learning disabilities; Machine learning; Machine Learning & Neural Networks; Mathematical procedures; Mathematics; Negotiation; Psychology; Social sciences; Specialized education; Statistical estimation; Statistical properties; Statistical results; Statistics; Theoretical computer science; Tradeoffs; Training
• ISBN: 026201646X; 9780262016469
• DOI: 10.7551/mitpress/8996.003.0015

The computational complexity of learning algorithms has seldom been taken into account by the learning theory. Valiant (1984) states that a problem is “learnable” when there exists a “probably approximately correct” learning algorithmwith polynomial complexity. Whereas much progress has been made on the statistical aspect (e.g., Vapnik, 1982; Boucheron et al., 2005; Bartlett and Mendelson, 2006), very little has been said about the complexity side of this proposal (e.g., Judd, 1988). Computational complexity becomes the limiting factor when one envisions large amounts of training data. Two important examples come to mind: Data mining exists because competitive advantages can be