Fast Algorithms for ℓp-Regression - Adil, Deeksha; Kyng, Rasmus; Peng, Richard; Sachdeva, Sushant (Journal of the ACM, 2024)
The \(\ell _p\) -norm regression problem is a classic problem in optimization with wide ranging applications in machine learning and theoretical computer science. The goal is to compute \(\boldsymbol {x}^{\star } =\arg \min _{\boldsymbol {A}\boldsymbol {x}=\boldsymbol {b}}\Vert \boldsymbol {x}\Vert _p^p\) , where \(\boldsymbol {x}^{\star }\in \mathbb {R}^n,\boldsymbol {A}\in \mathbb {R}^{d× n},\boldsymbol {b}\in \mathbb {R}^d\) and \(d\le n\) . Efficient high-accuracy algorithms for the problem have been challenging both in theory and practice and the state-of-the-art algorithms require \(poly(p)· n^{\frac{1}{2}-\frac{1}{p}}\) linear system solves for \(p\ge 2\) . In this article, we provide new algorithms for \(\ell _p\) -regression (and a more general formulation of the problem) that obtain a high-accuracy solution in \(O(p n^{ {(p-2)}{(3p-2)}})\) linear system solves. We further propose a new inverse maintenance procedure that speeds-up our algorithm to \(\widetilde{O}(n^{ω })\) total runtime, where \(O(n^{ω })\) denotes the running time for multiplying \(n × n\) matrices. Additionally, we give the first Iteratively Reweighted Least Squares (IRLS) algorithm that is guaranteed to converge to an optimum in a few iterations. Our IRLS algorithm has shown exceptional practical performance, beating the currently available implementations in MATLAB/CVX by 10–50×.