A highly efficient ADMM-based algorithm for outlier-robust regression with Huber loss

• Published in: Applied intelligence (Dordrecht, Netherlands) Vol. 54; no. 6; pp. 5147 - 5166
• Main Authors: Wang, Tianlei, Lai, Xiaoping, Cao, Jiuwen
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2024; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Computational efficiency; Computer Science; Convergence; Least squares method; Machine learning; Machines; Manufacturing; Mechanical Engineering; Neural networks; Newton methods; Parameter modification; Processes; Quadratic programming; Regression; Robustness (mathematics); Outliers; Alternating direction method of multipliers; Parallel algorithm; GPU acceleration ratio; Huber function; Machine learning; Least-squares regression; Huber robust regression
• ISSN: 0924-669X; 1573-7497
• DOI: 10.1007/s10489-024-05370-9

Huber robust regression (HRR) has attracted much attention in machine learning due to its greater robustness to outliers compared to least-squares regression. However, existing algorithms for HRR are computationally much less efficient than those for least-squares regression. Based on a maximally split alternating direction method of multipliers (MS-ADMM) for model fitting, a highly computationally efficient algorithm referred to as the modified MS-ADMM is derived in this article for HRR. After analyzing the convergence of the modified MS-ADMM, a parameter selection scheme is presented for the algorithm. With the parameter values calculated via this scheme, the modified MS-ADMM converges very rapidly, much faster than several typical HRR algorithms. Through applications in the training of stochastic neural networks and comparisons with existing algorithms, the modified MS-ADMM is shown to be computationally much more efficient than the convex quadratic programming method, the Newton method, the iterative reweighted least-squares method, and Nesterov’s accelerated gradient method. Implementation of the proposed algorithm on a GPU-based parallel computing platform demonstrates its higher GPU acceleration ratio compared to the competing methods and, thus, its greater superiority in computational efficiency over the competing methods.