A mean field view of the landscape of two-layer neural networks

• Published in: Proceedings of the National Academy of Sciences - PNAS Vol. 115; no. 33; pp. E7665 - E7671
• Main Authors: Mei, Song, Montanari, Andrea, Nguyen, Phan-Minh
• Format: Journal Article
• Language: English
• Published: United States National Academy of Sciences 14.08.2018
• Series: PNAS Plus
• Subjects: Artificial intelligence; Convergence; Learning algorithms; Machine learning; Mathematical models; Minima; Multilayers; Neural networks; Nonlinear differential equations; Optimization; Partial differential equations; Physical Sciences; PNAS Plus; Scaling; Stochastic models; gradient flow; partial differential equations; stochastic gradient descent; Wasserstein space; neural networks
• ISSN: 0027-8424; 1091-6490; 1091-6490
• DOI: 10.1073/pnas.1806579115

Multilayer neural networks are among the most powerful models in machine learning, yet the fundamental reasons for this success defy mathematical understanding. Learning a neural network requires optimizing a nonconvex high-dimensional objective (risk function), a problem that is usually attacked using stochastic gradient descent (SGD). Does SGD converge to a global optimum of the risk or only to a local optimum? In the former case, does this happen because local minima are absent or because SGD some-how avoids them? In the latter, why do local minima reached by SGD have good generalization properties? In this paper, we consider a simple case, namely two-layer neural networks, and prove that—in a suitable scaling limit—SGD dynamics is captured by a certain nonlinear partial differential equation (PDE) that we call distributional dynamics (DD). We then consider several specific examples and show how DD can be used to prove convergence of SGD to networks with nearly ideal generalization error. This description allows for “averaging out” some of the complexities of the landscape of neural networks and can be used to prove a general convergence result for noisy SGD.