Dictionary Learning Algorithms for Sparse Representation

• Published in: Neural computation Vol. 15; no. 2; pp. 349 - 396
• Main Authors: Kreutz-Delgado, Kenneth, Murray, Joseph F., Rao, Bhaskar D., Engan, Kjersti, Lee, Te-Won, Sejnowski, Terrence J.
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.02.2003
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Biological and medical sciences; Computer science; control theory; systems; Exact sciences and technology; Fundamental and applied biological sciences. Psychology; General aspects; General aspects. Models. Methods; Learning - physiology; Learning and adaptive systems; Mathematical foundations; Mathematics; Mathematics in biology. Statistical analysis. Models. Metrology. Data processing in biology (general aspects); Probability and statistics; Sciences and techniques of general use; Statistics; Stochastic Processes; Vertebrates: nervous system and sense organs; Stochastic model; Compression; Dictionaries; Description; Word; Noise; Complete; Elements; Generalization; Image; Mean square error; Learning; Probability density; Accuracy; Schur concavity; Gaussian noise; Noise source; Coding; Efficiency; Test; Bayesian model; A posteriori estimation; Sparse representation; Performance analysis; Component; Learning algorithm; Vector; Concept; Dictionary learning algorithm; Vector quantization; Use; Independent component analysis; Data; Representation; Neural network; Algorithm; Neurophysiology; Domains; Analysis method; Algorithm performance; Sparse set; Environment; Density function; Models; Maximum likelihood; Pixel; Signal to noise ratio
• ISSN: 0899-7667; 1530-888X; 1530-888X
• DOI: 10.1162/089976603762552951

Algorithms for data-driven learning of domain-specific overcomplete dictionaries are developed to obtain maximum likelihood and maximum a posteriori dictionary estimates based on the use of Bayesian models with concave/Schur-concave (CSC) negative log priors. Such priors are appropriate for obtaining sparse representations of environmental signals within an appropriately chosen (environmentally matched) dictionary. The elements of the dictionary can be interpreted as concepts, features, or words capable of succinct expression of events encountered in the environment (the source of the measured signals). This is a generalization of vector quantization in that one is interested in a description involving a few dictionary entries (the proverbial “25 words or less”), but not necessarily as succinct as one entry. To learn an environmentally adapted dictionary capable of concise expression of signals generated by the environment, we develop algorithms that iterate between a representative set of sparse representations found by variants of FOCUSS and an update of the dictionary using these sparse representations. Experiments were performed using synthetic data and natural images. For complete dictionaries, we demonstrate that our algorithms have improved performance over other independent component analysis (ICA) methods, measured in terms of signal-to-noise ratios of separated sources. In the overcomplete case, we show that the true underlying dictionary and sparse sources can be accurately recovered. In tests with natural images, learned overcomplete dictionaries are shown to have higher coding efficiency than complete dictionaries; that is, images encoded with an overcomplete dictionary have both higher compression (fewer bits per pixel) and higher accuracy (lower mean square error).