A Large-Scale Architecture for Restricted Boltzmann Machines

• Published in: 2010 18th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines pp. 201 - 208
• Main Authors: Sang Kyun Kim, McMahon, Peter L, Olukotun, Kunle
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.05.2010
• Subjects: Bandwidth; Boltzmann machines; Computational efficiency; Computer architecture; Field programmable gate arrays; Hardware; Image processing; Large-scale systems; Logic; Machine learning; Neural network hardware; Neurons; Parallel processing; Process design
• ISBN: 9781424471423; 0769540562; 9780769540566; 1424471427
• DOI: 10.1109/FCCM.2010.38

Deep Belief Nets (DBNs) are an emerging application in the machine learning domain, which use Restricted Boltzmann Machines (RBMs) as their basic building block. Although small scale DBNs have shown great potential, the computational cost of RBM training has been a major challenge in scaling to large networks. In this paper we present a highly scalable architecture for Deep Belief Net processing on hardware systems that can handle hundreds of boards, if not more, of customized logic with near linear performance increase. We elucidate tradeoffs between flexibility in the neuron connections, and the hardware resources, such as memory and communication bandwidth, required to build a custom processor design that has optimal efficiency. We illustrate how our architecture can easily support sparse networks with dense regions of connections between neighboring sets of neurons, which is relevant to applications where there are obvious spatial correlations in the data, such as in image processing. We demonstrate the feasibility of our approach by implementing a multi-FPGA system. We show that a speedup of 46X-112X over an optimized single core CPU implementation can be achieved for a four-FPGA implementation.