Adaptive learning rule for hardware-based deep neural networks using electronic synapse devices

• Published in: Neural computing & applications Vol. 31; no. 11; pp. 8101 - 8116
• Main Authors: Lim, Suhwan, Bae, Jong-Ho, Eum, Jai-Ho, Lee, Sungtae, Kim, Chul-Heung, Kwon, Dongseok, Park, Byung-Gook, Lee, Jong-Ho
• Format: Journal Article
• Language: English
• Published: London Springer London 01.11.2019; Springer Nature B.V
• Subjects: Accuracy; Adaptive learning; Algorithms; Artificial Intelligence; Artificial neural networks; Computational Biology/Bioinformatics; Computational Science and Engineering; Computer Science; Computer simulation; Data Mining and Knowledge Discovery; Electronic devices; Hardware; Image Processing and Computer Vision; Machine learning; Neural networks; Original Article; Probability and Statistics in Computer Science; Resistance; Upgrading; Weight; Hardware-based deep neural networks (HW-DNNs); Neuromorphic; Classification accuracy; Back-propagation; Synapse device; Deep neural networks (DNNs)
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-018-3659-y

In this paper, we propose a learning rule based on a back-propagation (BP) algorithm that can be applied to a hardware-based deep neural network using electronic devices that exhibit discrete and limited conductance characteristics. This adaptive learning rule, which enables forward, backward propagation, as well as weight updates in hardware, is helpful during the implementation of power-efficient and high-speed deep neural networks. In simulations using a three-layer perceptron network, we evaluate the learning performance according to various conductance responses of electronic synapse devices and weight-updating methods. It is shown that the learning accuracy is comparable to that obtained when using a software-based BP algorithm when the electronic synapse device has a linear conductance response with a high dynamic range. Furthermore, the proposed unidirectional weight-updating method is suitable for electronic synapse devices which have nonlinear and finite conductance responses. Because this weight-updating method can compensate the demerit of asymmetric weight updates, we can obtain better accuracy compared to other methods. This adaptive learning rule, which can be applied to full hardware implementation, can also compensate the degradation of learning accuracy due to the probable device-to-device variation in an actual electronic synapse device.