DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error

• Published in: Entropy (Basel, Switzerland) Vol. 20; no. 12; p. 927
• Main Authors: Jiang, Jianjun, Zhang, Jing, Zhang, Lijia, Ran, Xiaomin, Jiang, Jun, Wu, Yifan
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 04.12.2018; MDPI
• Subjects: Algorithms; artificial intelligence; Belief networks; Datasets; DBN; deep learning; Entropy (Information theory); Errors; improved simulated annealing algorithm; information entropy; Machine learning; Mathematical models; Neurons; Optimization; Reconstruction; reconstruction error; Simulated annealing; Structural design; structure design; reconstruction error; DBN; deep learning; information entropy; structure design; artificial intelligence; improved simulated annealing algorithm
• ISSN: 1099-4300; 1099-4300
• DOI: 10.3390/e20120927

Deep belief networks (DBNs) of deep learning technology have been successfully used in many fields. However, the structure of a DBN is difficult to design for different datasets. Hence, a DBN structure design algorithm based on information entropy and reconstruction error is proposed. Unlike previous algorithms, we innovatively combine network depth and node number and optimizes them simultaneously. First, the mathematical model of the structural design problem is established, and the boundary constraint for node number based on information entropy is derived by introducing the idea of information compression. Moreover, the optimization objective of the network performance based on reconstruction error is proposed by deriving the fact that network energy is proportional to reconstruction error. Finally, the improved simulated annealing (ISA) algorithm is used to adjust the DBN network layers and nodes simultaneously. Experiments were carried out on three public datasets (MNIST, Cifar-10 and Cifar-100). The results show that the proposed algorithm can design its proper structure to different datasets, yielding a trained DBN which has the lowest reconstruction error and prediction error rate. The proposed algorithm is shown to have the best performance compared with other algorithms and can be used to assist the setting of DBN structural parameters for different datasets.