All the essential topics in information theory are covered in detail, including entropy, data compression, channel capacity, rate distortion, network information theory, and hypothesis testing. The authors provide readers with a solid understanding of the underlying theory and applications.

Information, in Shannon’s theory of information, is viewed stochastically, or probabilistically. It is carried discretely as symbols, which are selected from a set of possible symbols. It merely means that each symbol is equally likely from the point of view of the receiver.

Shannon defined the quantity of information produced by a source–for example, the quantity in a message–by a formula similar to the equation that defines thermodynamic entropy in physics. In its most basic terms, Shannon’s informational entropy is the number of binary digits required to encode a message.

Applications of fundamental topics of information theory include lossless data compression (e.g. ZIP files), lossy data compression (e.g. MP3s and JPEGs), and channel coding (e.g. for DSL).

Information theory is the mathematical treatment of the concepts, parameters and rules governing the transmission of messages through communication systems.

The mathematical theory of information is based on probability theory and statistics, and measures information with several quantities of information. The most common unit of information is the bit, based on the binary logarithm.

In this sense, the “amount of information” is determined by how contributive the message is to the system. In this sense, even a single word, or a question, can “carry a lot of information” in a certain context, which cannot be explained in the previous two usages.

We can call log(1/p) information. Why? Because if all events happen with probability p, it means that there are 1/p events. To tell which event have happened, we need to use log(1/p) bits (each bit doubles the number of events we can tell apart).

Information is the source of a communication system, whether it is analog or digital. Information theory is a mathematical approach to the study of coding of information along with the quantification, storage, and communication of information.

There are four types of coding:

• Data compression (or source coding)
• Error control (or channel coding)
• Cryptographic coding.
• Line coding.

Coding is a basic literacy in the digital age, and it is important for kids to understand and be able to work with and understand the technology around them. Coding helps children with communication, creativity, math,writing, and confidence.

Claude Elwood Shannon