Information theory began with Claude Shannon’s paper “A Mathematical Theory of Communication”, published in two parts in the Bell System Technical Journal in 1948, the same year as Norbert Wiener’s Cybernetics.
Core ideas
Shannon treated communication as the problem of reproducing at one point a message selected at another. The information in a message is measured by how much it reduces uncertainty about which message, out of all possible ones, was sent. The unit is the bit: the information in a choice between two equally likely options. The average information of a source is its entropy.
Every channel has a capacity. Shannon proved that messages can be sent at any rate below capacity with as few errors as desired, by suitable coding, and not at rates above it. This made it possible to design reliable communication over unreliable lines.
Shannon deliberately set meaning aside: the theory concerns how many messages could have been sent, not what any of them means.
Relation to cybernetics
Wiener developed a closely related measure of information at the same time, and the two theories were discussed together at the Macy Conferences. W. Ross Ashby’s Law of Requisite Variety is closely related to Shannon’s theorem on noisy channels. Cyberneticians also argued about what Shannon’s theory left out. Gregory Bateson’s “a difference which makes a difference” and Donald MacKay’s work on meaning tried to bring the receiver and their context back in.
Further reading
- Claude Shannon and Warren Weaver, The Mathematical Theory of Communication (1949)
- James Gleick, The Information (2011)
Mentioned in
- Article A Brief History of Cybernetics
- Concept First-order cybernetics
- Community Ratio Club
- Person Claude Shannon