Information Losses in Neural Classifiers From Sampling

• Published in: IEEE transaction on neural networks and learning systems Vol. 31; no. 10; pp. 4073 - 4083
• Main Authors: Foggo, Brandon, Yu, Nanpeng, Shi, Jie, Gao, Yuanqi
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Classifiers; Compression; Datasets; Deep learning; information theory; large deviations theory; Learning systems; Machine learning; mutual information; Neural networks; Probability distribution; Random variables; statistical learning theory; Training; Training data
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2019.2952029

This article considers the subject of information losses arising from the finite data sets used in the training of neural classifiers. It proves a relationship between such losses as the product of the expected total variation of the estimated neural model with the information about the feature space contained in the hidden representation of that model. It then bounds this expected total variation as a function of the size of randomly sampled data sets in a fairly general setting, and without bringing in any additional dependence on model complexity. It ultimately obtains bounds on information losses that are less sensitive to input compression and in general much smaller than existing bounds. This article then uses these bounds to explain some recent experimental findings of information compression in neural networks that cannot be explained by previous work. Finally, this article shows that not only are these bounds much smaller than existing ones, but they also correspond well with experiments.