The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionality

• Published in: The Journal of artificial intelligence research Vol. 74; pp. 1775 - 1790
• Main Authors: Froese, Vincent, Hertrich, Christoph, Niedermeier, Rolf
• Format: Journal Article
• Language: English
• Published: San Francisco AI Access Foundation 01.01.2022
• Subjects: Algorithms; Artificial intelligence; Complexity; Lower bounds; Neural networks; Parameterization; Parameters; Polynomials; Training
• ISSN: 1076-9757; 1076-9757; 1943-5037
• DOI: 10.1613/jair.1.13547

Understanding the computational complexity of training simple neural networks with rectified linear units (ReLUs) has recently been a subject of intensive research. Closing gaps and complementing results from the literature, we present several results on the parameterized complexity of training two-layer ReLU networks with respect to various loss functions. After a brief discussion of other parameters, we focus on analyzing the influence of the dimension d of the training data on the computational complexity. We provide running time lower bounds in terms of W[1]-hardness for parameter d and prove that known brute-force strategies are essentially optimal (assuming the Exponential Time Hypothesis). In comparison with previous work, our results hold for a broad(er) range of loss functions, including lp-loss for all p ∈ [0, ∞]. In particular, we improve a known polynomial-time algorithm for constant d and convex loss functions to a more general class of loss functions, matching our running time lower bounds also in these cases.