Winograd Convolution for Deep Neural Networks: Efficient Point Selection

• Published in: ACM transactions on embedded computing systems Vol. 21; no. 6; pp. 1 - 28
• Main Authors: Alam, Syed Asad, Anderson, Andrew, Barabasz, Barbara, Gregg, David
• Format: Journal Article
• Language: English
• Published: New York, NY ACM 12.12.2022
• Subjects: Computer systems organization; Embedded systems; Winograd convolution; Toom-Cook algorithm; deep neural networks
• ISSN: 1539-9087; 1558-3465
• DOI: 10.1145/3524069

Convolutional neural networks (CNNs) have dramatically improved the accuracy of image, video, and audio processing for tasks such as object recognition, image segmentation, and interactive speech systems. CNNs require large amounts of computing resources for both training and inference, primarily because the convolution layers are computationally intensive. Fast convolution algorithms such as Winograd convolution can greatly reduce the computational cost of these layers. However, Winograd convolution has poor numeric properties, such that greater savings in computation cause exponentially increasing floating point errors.A defining feature of each Winograd convolution algorithm is a set of real-value points where polynomials are sampled. The choice of points impacts the numeric accuracy of the algorithm, but the optimal set of points for small convolutions remains unknown. Existing work considers only small integers and simple fractions as candidate points. In this work, we propose a novel approach to point selection using points of the form \(\lbrace -\frac{1}{c},-c,c,\frac{1}{c}\rbrace\) using the full range of real-valued numbers for c. We show that groups of this form cause cancellations in the Winograd transform matrices that reduce numeric error. We find empirically that the error for different values of c forms a rough curve across the range of real-value numbers. It is therefore possible to localize the values of c that lead to lower error. We show that it is not necessary to choose integers or simple fractions as evaluation points, and that lower errors can be achieved with non-obvious real-valued points. We study a range of sizes for small convolutions and achieve reduction in error ranging from 2% to around 59% for both 1D and 2D convolution, when compared to state of the art. Furthermore, we identify patterns in cases when we select a subset of our proposed points that will always lead to a lower error. Finally, we implement a complete Winograd convolution layer and use it to run state-of-the-art deep convolution neural networks on real datasets and show that our proposed points achieve reduction in error, ranging from 22% to 63%, while also showing how an increased Winograd output size can result in execution speed-up for some cases.