Rate Distortion and Denoising of Individual Data Using Kolmogorov Complexity

• Published in: IEEE transactions on information theory Vol. 56; no. 7; pp. 3438 - 3454
• Main Authors: Vereshchagin, Nikolai K, Vitányi, Paul M B
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.07.2010; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithmic rate distortion; Algorithms; characterization; Complexity; denoising; Distortion; distortion families; Distortion measurement; fitness of destination words; Formulations; Frequency; individual data; Information analysis; Information processing; Information science; Information theory; Kolmogorov complexity; Lists; Measurement standards; Mutual information; Noise reduction; Rate distortion theory; Rate-distortion; Region 8; Shape; shapes of curves; Theory
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2010.2048491

We examine the structure of families of distortion balls from the perspective of Kolmogorov complexity. Special attention is paid to the canonical rate-distortion function of a source word which returns the minimal Kolmogorov complexity of all distortion balls containing that word subject to a bound on their cardinality. This canonical rate-distortion function is related to the more standard algorithmic rate-distortion function for the given distortion measure. Examples are given of list distortion, Hamming distortion, and Euclidean distortion. The algorithmic rate-distortion function can behave differently from Shannon's rate-distortion function. To this end, we show that the canonical rate-distortion function can and does assume a wide class of shapes (unlike Shannon's); we relate low algorithmic mutual information to low Kolmogorov complexity (and consequently suggest that certain aspects of the mutual information formulation of Shannon's rate-distortion function behave differently than would an analogous formulation using algorithmic mutual information); we explore the notion that low Kolmogorov complexity distortion balls containing a given word capture the interesting properties of that word (which is hard to formalize in Shannon's theory) and this suggests an approach to denoising.