Training generalizable quantized deep neural nets

• Published in: Expert systems with applications Vol. 213; p. 118736
• Main Authors: Hernandez, Charles, Taslimi, Bijan, Lee, Hung Yi, Liu, Hongcheng, Pardalos, Panos M.
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.03.2023
• Subjects: Deep learning; Generalizability; Quantized neural networks; Deep learning; Quantized neural networks; 62M45; Generalizability; 68Q32
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2022.118736

While a number of practical methods for training quantized DL models have been presented in the literature, there exists a critical gap in the theoretical generalizability results for such approaches. Although empirical evidence often suggests a high tolerance of DL architectures to variations of training procedures, existing theoretical generalization analyses are often contingent on the specific designs of training algorithms, e.g., in stochastic gradient descent (SGD). This specialization makes such generalizability results inapplicable to the case of quantized DL models. In view of this critical vacuum, this paper provides several almost-algorithm-independent results to ensure the generalizability of a quantized neural network at different levels of optimality. These results include the characterizations of a computable, quantized local solution that ensures the generalization performance and an algorithm that is provably convergent to such a local solution. •A novel generalizability theory for quantized deep neural nets trained globally.•The first generalization error bound for tractable local solutions of quantized DL.•Proposing a provably effective and efficient algorithm for quantized DL models.