A Learning Algorithm with Compression-Based Regularization

This paper investigates, from information theoretic principles, a learning problem based on the principle that any regularity in a given dataset can be exploited to extract compact features from data, in order to build meaningful representations of a relevant content. We begin by introducing the fun...

Full description

Saved in:
Bibliographic Details
Published in2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 2836 - 2840
Main Authors Vera, Matias, Vega, Leonardo Rey, Piantanida, Pablo
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.04.2018
Subjects
Arimoto-Blahut Algorithm
Complexity theory
Decoding
Feature extraction
Information Bottleneck
Information rates
Multi-Task Learning
Side Information
Task analysis
Text categorization
Training
Online AccessGet full text
ISSN2379-190X
DOI10.1109/ICASSP.2018.8461441

Cover

More Information
Summary:This paper investigates, from information theoretic principles, a learning problem based on the principle that any regularity in a given dataset can be exploited to extract compact features from data, in order to build meaningful representations of a relevant content. We begin by introducing the fundamental tradeoff between the average risk and the model complexity. Interestingly, our formulation allows an information theoretic formulation of the multi-task learning (MTL) problem. Then, we present an iterative algorithm for computing the optimal tradeoffs. Remarkably, empirical results illustrate that there exists an optimal information rate minimizing the excess risk which depends on the nature and the amount of available training data. An application to hierarchical text categorization is also investigated, extending previous works.
ISSN:2379-190X
DOI:10.1109/ICASSP.2018.8461441