Sumset and Inverse Sumset Inequalities for Differential Entropy and Mutual Information

作者：Kontoyiannis Ioannis^*; Madiman Mokshay

来源：IEEE Transactions on Information Theory, 2014, 60(8): 4503-4514.

DOI：10.1109/TIT.2014.2322861

• Shannon entropy
• differential entropy
• sumset bounds
• inequalities
• submodularity
• data processing
• mutual information

The sumset and inverse sumset theories of Freiman, Plunnecke, and Ruzsa, give bounds connecting the cardinality of the sumset A + B = {a + b; a is an element of A, b is an element of B} of two discrete sets A, B, to the cardinalities (or the finer structure) of the original sets A, B. For example, the sum-difference bound of Ruzsa states that, vertical bar A + B vertical bar vertical bar A vertical bar vertical bar B vertical bar = vertical bar A - B vertical bar(3), where the difference set A - B = {a - b; a is an element of A, b is an element of B}. Interpreting the differential entropy h(X) of a continuous random variable X as (the logarithm of) the size of the effective support of X, the main contribution of this paper is a series of natural information-theoretic analogs for these results. For example, the Ruzsa sum-difference bound becomes the new inequality, h(X + Y)+h(X)+h(Y) %26lt;= 3h(X - Y), for any pair of independent continuous random variables X and Y. Our results include differential-entropy versions of Ruzsa%26apos;s triangle inequality, the Plunnecke-Ruzsa inequality, and the Balog-Szemeredi-Gowers lemma. In addition, we give a differential entropy version of a Freiman-type inverse-sumset theorem, which can be seen as a quantitative converse to the entropy power inequality. Versions of most of these results for the discrete entropy H(X) were recently proved by Tao, relying heavily on a strong, functional form of the submodularity property of H(X). Since differential entropy is not functionally submodular, in the continuous case many of the corresponding discrete proofs fail, in many cases requiring substantially new proof strategies. We find that the basic property that naturally replaces the discrete functional submodularity, is the data processing property of mutual information.

• 出版日期2014-8