Quantization for Decentralized Learning Under Subspace Constraints

• Published in: IEEE transactions on signal processing Vol. 71; pp. 2320 - 2335
• Main Authors: Nassif, Roula, Vlaski, Stefan, Carpentiero, Marco, Matta, Vincenzo, Antonini, Marc, Sayed, Ali H.
• Format: Journal Article
• Language: English
• Published: New York IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Bit rate; Communication; Computer Science; Constraints; Cost function; Costs; Counters; decentralized learning; decentralized optimization; decentralized subspace projection; differential quantization; Engineering Sciences; Learning; mixing parameter; Optimization; Quantization (signal); randomized quantizers; Servers; Signal processing algorithms; Smoothness; Stochastic optimization; stochastic performance analysis; Stochastic processes; Subspaces; decentralized learning; decentralized subspace projection; decentralized optimization; differential quantization; randomized quantizers; mixing parameter; stochastic performance analysis; Stochastic optimization
• ISSN: 1053-587X; 1941-0476; 1941-0476
• DOI: 10.1109/TSP.2023.3287333

In this article, we consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints that require the minimizers across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus or single-task optimization as special cases, and allows for more general task relatedness models such as multitask smoothness and coupled optimization. In order to cope with communication constraints, we propose and study an adaptive decentralized strategy where the agents employ differential randomized quantizers to compress their estimates before communicating with their neighbors. The analysis shows that, under some general conditions on the quantization noise, and for sufficiently small step-sizes <inline-formula><tex-math notation="LaTeX">\mu</tex-math></inline-formula>, the strategy is stable both in terms of mean-square error and average bit rate: by reducing <inline-formula><tex-math notation="LaTeX">\mu</tex-math></inline-formula>, it is possible to keep the estimation errors small (on the order of <inline-formula><tex-math notation="LaTeX">\mu</tex-math></inline-formula>) without increasing indefinitely the bit rate as <inline-formula><tex-math notation="LaTeX">\mu \rightarrow 0</tex-math></inline-formula> when variable-rate quantizers are used. Simulations illustrate the theoretical findings and the effectiveness of the proposed approach, revealing that decentralized learning is achievable at the expense of only a few bits.