Convolution smoothing and online updating estimation for support vector machine

• Published in: Test (Madrid, Spain) Vol. 34; no. 1; pp. 288 - 323
• Main Authors: Wang, Kangning, Meng, Xiaoqing, Sun, Xiaofei
• Format: Journal Article
• Language: English
• Published: Heidelberg Springer Nature B.V 01.03.2025
• Subjects: Algorithms; Approximation; Classification; Convolution; Datasets; Distance learning; Feature selection; Regression analysis; Smoothing; Smoothness; Statistical inference; Support vector machines; Survival analysis
• ISSN: 1133-0686; 1863-8260
• DOI: 10.1007/s11749-024-00959-1

Support vector machine (SVM) is a powerful binary classification statistical learning tool. In real applications, streaming data are common, which arrive in batches and have unbounded cumulative size. Because of the memory constraints of one single computer, the classical SVM solving the entire data together is unsuitable. Furthermore, the non-smoothness of hinge loss in SVM also poses high computational complexity. To overcome these issues, we first develop a convolution smoothing approach that achieves smooth and convex approximation to SVM. Then an online updating SVM is proposed, in which the estimators are renewed with current data and historical summary statistics. In theory, we prove that the convolution smoothing SVM achieves adequate approximation to SVM, and they are asymptotically equivalent in inference. Furthermore, the online updating SVM achieves the same efficiency as the classical SVM applying to the entire dataset. Numerical experiments on both synthetic and real data also validate our new methods.