SpaRCe: Improved Learning of Reservoir Computing Systems Through Sparse Representations

• Published in: IEEE transaction on neural networks and learning systems Vol. 34; no. 2; pp. 824 - 838
• Main Authors: Manneschi, Luca, Lin, Andrew C., Vasilaki, Eleni
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Brain; Catastrophic forgetting; Computation; Computational modeling; Decision making; echo state networks (ESNs); Error functions; Firing pattern; Learning algorithms; Machine Learning; Mathematical model; Nervous system; Neural networks; Neural Networks, Computer; Neurons; Neurons - physiology; online learning; Recurrent neural networks; reservoir computing; Reservoirs; Sparsity; Task analysis; Thresholds
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2021.3102378

"Sparse" neural networks, in which relatively few neurons or connections are active, are common in both machine learning and neuroscience. While, in machine learning, "sparsity" is related to a penalty term that leads to some connecting weights becoming small or zero, in biological brains, sparsity is often created when high spiking thresholds prevent neuronal activity. Here, we introduce sparsity into a reservoir computing network via neuron-specific learnable thresholds of activity, allowing neurons with low thresholds to contribute to decision-making but suppressing information from neurons with high thresholds. This approach, which we term "SpaRCe," optimizes the sparsity level of the reservoir without affecting the reservoir dynamics. The read-out weights and the thresholds are learned by an online gradient rule that minimizes an error function on the outputs of the network. Threshold learning occurs by the balance of two opposing forces: reducing interneuronal correlations in the reservoir by deactivating redundant neurons, while increasing the activity of neurons participating in correct decisions. We test SpaRCe on classification problems and find that threshold learning improves performance compared to standard reservoir computing. SpaRCe alleviates the problem of catastrophic forgetting, a problem most evident in standard echo state networks (ESNs) and recurrent neural networks in general, due to increasing the number of task-specialized neurons that are included in the network decisions.