An Optimal High-Order Tensor Method for Convex Optimization

• Published in: Mathematics of operations research Vol. 46; no. 4; pp. 1390 - 1412
• Main Authors: Jiang, Bo, Wang, Haoyue, Zhang, Shuzhong
• Format: Journal Article
• Language: English
• Published: Linthicum INFORMS 01.11.2021; Institute for Operations Research and the Management Sciences
• Subjects: acceleration; Algorithms; Complexity; Computational geometry; Convex analysis; convex optimization; Convexity; iteration complexity; Iterative methods; Lower bounds; Mathematical analysis; Operations research; Optimization; Primary: 90C60, 90C25, 47H05; Primary: programming/nonlinear/convex; tensor method; Tensors
• ISSN: 0364-765X; 1526-5471
• DOI: 10.1287/moor.2020.1103

This paper is concerned with finding an optimal algorithm for minimizing a composite convex objective function. The basic setting is that the objective is the sum of two convex functions: the first function is smooth with up to the d th-order derivative information available, and the second function is possibly nonsmooth, but its proximal tensor mappings can be computed approximately in an efficient manner. The problem is to find—in that setting—the best possible (optimal) iteration complexity for convex optimization. Along that line, for the smooth case (without the second nonsmooth part in the objective), Nesterov proposed an optimal algorithm for the first-order methods ( d = 1 ) with iteration complexity O ( 1 / k 2 ) , whereas high-order tensor algorithms (using up to general d th-order tensor information) with iteration complexity O ( 1 / k d + 1 ) were recently established. In this paper, we propose a new high-order tensor algorithm for the general composite case, with the iteration complexity of O ( 1 / k ( 3 d +1 ) / 2 ) , which matches the lower bound for the d th-order methods as previously established and hence is optimal. Our approach is based on the accelerated hybrid proximal extragradient (A-HPE) framework proposed by Monteiro and Svaiter, where a bisection procedure is installed for each A-HPE iteration. At each bisection step, a proximal tensor subproblem is approximately solved, and the total number of bisection steps per A-HPE iteration is shown to be bounded by a logarithmic factor in the precision required.