Learning in compressed space

• Published in: Neural networks Vol. 42; pp. 83 - 93
• Main Authors: Fabisch, Alexander, Kassahun, Yohannes, Wöhrle, Hendrik, Kirchner, Frank
• Format: Journal Article
• Language: English
• Published: Kidlington Elsevier Ltd 01.06.2013; Elsevier
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Artificial neural networks; Brain-Computer Interfaces; Compressed sensing; Computer science; control theory; systems; Computer Simulation; Connectionism. Neural networks; Data Compression; Data processing. List processing. Character string processing; Detection, estimation, filtering, equalization, prediction; Electroencephalography; Event-Related Potentials, P300; Exact sciences and technology; Humans; Information, signal and communications theory; Learning; Memory organisation. Data processing; Model compression; Nerve Net - physiology; Neural Networks (Computer); Neurons - physiology; Perception; Probability; Reinforcement learning; Signal and communications theory; Signal, noise; Software; Supervised learning; Telecommunications and information theory; Supervised learning; Model compression; Compressed sensing; Artificial neural networks; Reinforcement learning; Backpropagation; Data analysis; compressed sensing; Neural network; Modeling; Projection method; Dimension reduction; Backpropagation algorithm; Orthogonal function; Multilayer perceptrons; Sparse representation; Feedforward; Learning algorithm; Artificial intelligence; Signal detection
• ISSN: 0893-6080; 1879-2782; 1879-2782
• DOI: 10.1016/j.neunet.2013.01.020

We examine two methods which are used to deal with complex machine learning problems: compressed sensing and model compression. We discuss both methods in the context of feed-forward artificial neural networks and develop the backpropagation method in compressed parameter space. We further show that compressing the weights of a layer of a multilayer perceptron is equivalent to compressing the input of the layer. Based on this theoretical framework, we will use orthogonal functions and especially random projections for compression and perform experiments in supervised and reinforcement learning to demonstrate that the presented methods reduce training time significantly.