A projected decentralized variance-reduction algorithm for constrained optimization problems

• Published in: Neural computing & applications Vol. 36; no. 2; pp. 913 - 928
• Main Authors: Deng, Shaojiang, Gao, Shanfu, Lü, Qingguo, Li, Yantao, Li, Huaqing
• Format: Journal Article
• Language: English
• Published: London Springer London 01.01.2024; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Computational Biology/Bioinformatics; Computational efficiency; Computational Science and Engineering; Computer Science; Computing time; Convergence; Convexity; Data Mining and Knowledge Discovery; Digital video recorders; Efficiency; Image Processing and Computer Vision; Iterative methods; Operators (mathematics); Optimization; Original Article; Probability and Statistics in Computer Science; Reduction; Variance-reduction; Stochastic algorithm; Linear convergence; Decentralized constrained optimization
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-023-09067-x

Solving constrained optimization problems that require processing large-scale data is of significant value in practical applications, and such problems can be described as the minimization of a finite-sum of local convex functions. Many existing works addressing constrained optimization problems have achieved a linear convergence rate to the exact optimal solution if the constant step-size was sufficiently small. However, they still suffer from low computational efficiency because of the computation of the local batch gradients at each iteration. Considering high computational efficiency to resolve the constrained optimization problems, we introduce the projection operator and variance-reduction technique to propose a novel projected decentralized variance-reduction algorithm, namely P-DVR, to tackle the constrained optimization problem subject to a closed convex set. Theoretical analysis shows that if the local function is strongly convex and smooth, the P-DVR algorithm can converge to the exact optimal solution at a linear convergence rate O ( λ ^ k ) with a sufficiently small step-size, where 0 < λ ^ < 1 . Finally, we experimentally validate the effectiveness of the algorithm, i.e., the algorithm possesses high computational efficiency and exact convergence.