Information Without Rolling Dice

• Published in: IEEE transactions on information theory Vol. 63; no. 3; pp. 1349 - 1363
• Main Authors: Lim, Taehyung J., Franceschetti, Massimo
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.03.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: <italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">N -width; approximation theory; Bandlimited signals; capacity; Communication; degrees of freedom; Entropy; Information storage; Information theory; Rate distortion theory; rate-distortion function; Signal to noise ratio; Stochastic models; Stochastic processes; zero-error capacity; ϵ-capacity; ϵ-entropy
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2016.2647602

The deterministic notions of capacity and entropy are studied in the context of communication and storage of information using square-integrable and bandlimited signals subject to perturbation. The (E, δ)-capacity that extends the Kolmogorov E-capacity to packing sets of overlap at most δ is introduced and compared with the Shannon capacity. The functional form of the results indicates that in both Kolmogorov and Shannon's settings, capacity and entropy grow linearly with the number of degrees of freedom, but only logarithmically with the signal to noise ratio. This basic insight transcends the details of the stochastic or deterministic description of the information theoretic model. For δ = 0, the analysis leads to a tight asymptotic expression of the Kolmogorov E-entropy of bandlimited signals. A deterministic notion of error exponent is introduced. Applications of the theory are briefly discussed.