Gradient Coding With Iterative Block Leverage Score Sampling

• Published in: IEEE transactions on information theory Vol. 70; no. 9; pp. 6639 - 6664
• Main Authors: Charalambides, Neophytos, Pilanci, Mert, Hero, Alfred O.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.09.2024
• Subjects: Accuracy; Approximation algorithms; coded computing; Encoding; erasure-coding; least-squares regression; Linear algebra; Low-rank approximations; MATHEMATICS AND COMPUTING; Monte Carlo methods; randomized algorithms; randomized numerical linear algebra; replication-coding; Servers; stragglers; Task analysis
• ISSN: 0018-9448; 1557-9654; 1557-9654
• DOI: 10.1109/TIT.2024.3420222

Gradient coding is a method for mitigating straggling servers in a centralized computing network that uses erasure-coding techniques to distributively carry out first-order optimization methods. Randomized numerical linear algebra uses randomization to develop improved algorithms for large-scale linear algebra computations. In this paper, we propose a method for distributed optimization that combines gradient coding and randomized numerical linear algebra. The proposed method uses a randomized <inline-formula> <tex-math notation="LaTeX">\ell _{2} </tex-math></inline-formula>-subspace embedding and a gradient coding technique to distribute blocks of data to the computational nodes of a centralized network, and at each iteration the central server only requires a small number of computations to obtain the steepest descent update. The novelty of our approach is that the data is replicated according to importance scores, called block leverage scores, in contrast to most gradient coding approaches that uniformly replicate the data blocks. Furthermore, we do not require a decoding step at each iteration, avoiding a bottleneck in previous gradient coding schemes. We show that our approach results in a valid <inline-formula> <tex-math notation="LaTeX">\ell _{2} </tex-math></inline-formula>-subspace embedding, and that our resulting approximation converges to the optimal solution.