Control-based algorithms for high dimensional online learning

• Published in: Journal of the Franklin Institute Vol. 357; no. 3; pp. 1909 - 1942
• Main Authors: Ning, Hanwen, Zhang, Jiaming, Feng, Ting-Ting, Chu, Eric King-wah, Tian, Tianhai
• Format: Journal Article
• Language: English
• Published: Elmsford Elsevier Ltd 01.02.2020; Elsevier Science Ltd
• Subjects: Algorithms; Classification; Data transmission; Feedback control; Machine learning; Numerical analysis; Online instruction; Optimal control; Robust control
• ISSN: 0016-0032; 1879-2693; 0016-0032
• DOI: 10.1016/j.jfranklin.2019.12.039

In the era of big data, the high-dimensional online learning problems require huge computing power. This paper proposes a novel approach for high-dimensional online learning. Two new algorithms are developed for online high-dimensional regression and classification problems, respectively. The problems are formulated as feedback control problems for some low dimensional systems. The novel learning algorithms are then developed via the control problems. Via an efficient polar decomposition, we derive the explicit solutions of the control problems, substantially reducing the corresponding computational complexity, especially for high dimensional large-scale data streams. Comparing with conventional methods, the new algorithm can achieve more robust and accurate performance with faster convergence. This paper demonstrates that optimal control can be an effective approach for developing high dimensional learning algorithms. We have also for the first time proposed a control-based robust algorithm for classification problems. Numerical results support our theory and illustrate the efficiency of our algorithm.