On the Rényi Divergence, Joint Range of Relative Entropies, and a Channel Coding Theorem

• Published in: IEEE transactions on information theory Vol. 62; no. 1; pp. 23 - 34
• Main Author: Sason, Igal
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.01.2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Alphabets; Binary codes; Binomial distribution; Block codes; Channels; Chernoff information; distance spectrum; Divergence; Entropy; error exponent; Information theory; Maximum likelihood decoding; Maximum likelihood method; Minimization; Optimization; Q measurement; relative entropy; Renyi divergence; total variation distance; Upper bound; maximum-likelihood decoding; Rényi divergence; error exponent; relative entropy; distance spectrum; Chernoff information; total variation distance
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2015.2504100

This paper starts by considering the minimization of the Rényi divergence subject to a constraint on the total variation distance. Based on the solution of this optimization problem, the exact locus of the points (D(QIIP 1 ), D(QIIP 2 )) is determined when P 1 , P 2 , and Q are arbitrary probability measures which are mutually absolutely continuous, and the total variation distance between P 1 and P 2 is not below a given value. It is further shown that all the points of this convex region are attained by probability measures which are defined on a binary alphabet. This characterization yields a geometric interpretation of the minimal Chernoff information subject to a constraint on the variational distance. This paper also derives an exponential upper bound on the performance of binary linear block codes (or code ensembles) under maximum-likelihood decoding. Its derivation relies on the Gallager bounding technique, and it reproduces the Shulman-Feder bound as a special case. The bound is expressed in terms of the Rényi divergence from the normalized distance spectrum of the code (or the average distance spectrum of the ensemble) to the binomially distributed distance spectrum of the capacity-achieving ensemble of random block codes. This exponential bound provides a quantitative measure of the degradation in performance of binary linear block codes (or code ensembles) as a function of the deviation of their distance spectra from the binomial distribution. An efficient use of this bound is considered.