Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models

• Published in: IEEE transactions on information theory Vol. 71; no. 4; pp. 2930 - 2954
• Main Authors: Mei, Song, Wu, Yuchen
• Format: Journal Article
• Language: English
• Published: IEEE 01.04.2025
• Subjects: Approximation algorithms; Computational modeling; Diffusion model; Diffusion models; graphical model; Graphical models; Inference algorithms; Mathematical models; Noise reduction; residual neural network; Residual neural networks; Risk minimization
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2025.3535923

We investigate the efficiency of deep neural networks for approximating scoring functions in diffusion-based generative modeling. While existing approximation theories leverage the smoothness of score functions, they suffer from the curse of dimensionality for intrinsically high-dimensional data. This limitation is pronounced in graphical models such as Markov random fields, where the approximation efficiency of score functions remains unestablished. To address this, we note score functions can often be well-approximated in graphical models through variational inference denoising algorithms. Furthermore, these algorithms can be efficiently represented by neural networks. We demonstrate this through examples, including Ising models, conditional Ising models, restricted Boltzmann machines, and sparse encoding models. Combined with off-the-shelf discretization error bounds for diffusion-based sampling, we provide an efficient sample complexity bound for diffusion-based generative modeling when the score function is learned by deep neural networks.