Counting Distinctions: On the Conceptual Foundations of Shannon's Information Theory

Abstract

Ordinary "propositional" logic can be interpreted as the logic of subsets. The concept of a partition on a universe set U is dual to the concept of a subset of the universe set in the category-theoretic sense of duality between epimorphisms and monomorpisms. The dual to the notion of an element being in a subset is that of a distinction being made by a partition (i.e., a pair of elements being in distinct blocks of the partition). Probability theory started by moving beyond the logic of subsets on a universe to assign a "probability" to each subset of a finite universe which was the number of elements in the subset normalized by the size of the universe set (using the Laplacian assumption of each element having equal probability). This paper works out the corresponding conceptual development starting with the logic of partitions. Each partition on U is assigned a "logical entropy" which is the number of distinctions made by the partition normalized by the number of ordered pairs in UxU. This notion of logical entropy is then precisely related to Shannon's notion of entropy showing that information theory is conceptually based on the logical notion of distinctions.

Keywords: information theory, Shannon's entropy, logical entropy, logic of partitions

JEL Classification: D80