information theory

The idea of measuring information might at first seem as senseless as that of weighing the theorem of Pythagoras, but it was in fact among hard-headed communication engineers that the need to do so was first recognized. A communication channel exists (and is paid for) in order to 'transmit information'. In order to compare the efficiencies of alternative methods we must be able to estimate the 'capacity' of each to do the job.

In ordinary language we say we have received information when what we know has changed. The bigger the change in what we know, the more information we have received. Information, then, like energy, does work, but whereas energy does physical work, information does logical work. There are various ways of measuring the size of the logical job done when a communication signal is received. Different measures are relevant in different contexts. The main consideration is whether the output of the communication channel has to be constructed by the signal, or merely selected (identified) by it from a range of prefabricated forms. Some examples will make this clear.

1. Construction
2. Selection

1. Construction

When a camera shutter is opened, we say that 'information' is transmitted from the scene to the film. By this we mean that the image on the film is constructed by the action of the light signal received. In comparing results from two cameras, two quite different criteria may be relevant. (i) We might compare the number of distinguishable picture elements, which is called the structural information content of each picture. A picture taken with poor optical resolution would have a low structural information content. (The unit of structural information content, 1 logon, specifies one independent element or 'degree of freedom' of a signal.) (ii) Alternatively, or additionally, we might compare the statistical weight of evidence gathered by the two films, or their metrical information content. A picture taken with too short an exposure, for example, would be deficient in metrical information content. (The unit of metrical information content, 1 metron, represents a certain minimum weight of statistical evidence.)

2. Selection

Although in many telecommunication systems (such as domestic telephones, radio, and TV) the signal has to construct the output, in other cases (such as telegraphy and teleprinting), where the range of possible outputs is small and is known in advance, a much more economical approach is possible. This is known as encoding. Instead of transmitting a complete description of the output required, a code system transmits only instructions to select (identify) that output from a range of prefabricated outputs (for example, letters of the alphabet) available at the receiving end. Thus, whereas the construction of a television picture of a page of type might require several million independent signals, the same page of type can be specified by only a few thousand selective code signals controlling a teleprinter.

In this context, the size of the selective job done by a signal depends not on the size or complexity of the output as such, but on the number of alternative forms that it might have taken, and on the relative likelihood of each. The simplest selective operation is the identification of one out of two equally likely possibilities — as in the game of 'Twenty Questions'. This, in the jargon of communication engineers, has a selective information content of one 'bit' or binary digit. Selection of one out of four equally likely possibilities requires 2 bits; one out of eight requires 3; and so on. In general, then, selective information content measures the statistical unexpectedness of the event in question. The precise form of the event is irrelevant, except as it may affect its prior probability. The more improbable an event, the larger its selective information content. This way of measuring information flow was developed chiefly by the American engineer C. E. Shannon (Shannon and Weaver 1949) and the mathematician Norbert Wiener (1948). (Mathematically, the number of bits per event is log₂(1/pi). Where events 1, 2 ... i have prior probabilities pi₁, pi₂ ... pi_i, the average selective information content or 'entropy'The average unexpectedness or selective information content per event is greatest when all possible events are equally probable. The communication channel is then being used with full 'informational efficiency'. If some events are much more or much less probable than others, the average number of bits per event is correspondingly reduced, and the sequence is said to have 'redundancy'. (Redundancy is defined as 1 – H/H_max, where H_max is the value of H when all pi are equal.) A redundancy of 50 per cent values means that the average number of bits per event is only half what it could be if all events were equally probable.

Despite its pejorative label, redundancy has one great merit. It allows a communication system in principle to tolerate a corresponding amount of random transmission error or 'noise'. If the right kind of code is used, it is even possible to achieve almost error-free transmission up to a certain rate, despite the presence of 'noise', by building in redundancy in such a way that errors can be identified and corrected. Although this may sound almost magical, it is similar in principle to what a human reader does when spotting and correcting printers' errors. The detection of misprints in an unfamiliar passage is possible only because the sequence of letters in typical English text is about 50 per cent redundant. With a table of random numbers, it would be impossible! Saying the same thing several times in different ways (a device familiar to teachers and public speakers) is another sensible way of building in redundancy, so as to make a message more resistant to distortion, either by noise in the communication channel or through misperception by the recipient.

By analogous reasoning, it has been proved that networks of computing elements can be constructed with redundant connections in such a way as to perform without errors, even if individual elements were to break down at random. Here again, the amount of malfunctioning that can be tolerated depends directly on the amount of redundancy built in. There is reason to believe that the amazing reliability of the human brain (despite a steady loss of nerve cells throughout life) depends on a sophisticated use of redundancy on these lines.

Since the theory of information embraces communication processes of all kinds, whether in human societies, in nervous systems, or in machines, it inspired at first some exaggerated expectations. Early efforts to measure the flow of information through sense organs, or through human operators controlling machines, were sometimes frustrated because the probabilities attached to events by the experimenter were not the same as those represented in the subject's nervous system. In other cases, where there was no reason to believe that the neural systems concerned worked on a selective principle, the use of Shannon's measure gave irrelevant or trivial results.

On the other hand, the development and spread of information-theoretical ideas has made notable contributions to brain research by suggesting new ways of looking at the function of the nervous system and new kinds of experimental questions. To take one of the earliest examples, Hick (1952) found that the reaction time of human subjects to a stimulus depended in a particularly simple way on its selective information content. In other words, what mattered in his experiment was not the form of the stimulus per se, but rather the number of alternative forms that it might have taken but did not. The idea that the range of forms not taken by the input might be an important part of its specification is typical of the shift in emphasis brought to psychology by information theory.

Again, both physiologists and psychologists now make extensive use of test signals originally developed by communication engineers to measure the performance of TV and radio channels. Both highly redundant signals such as regularly repetitive patterns (Fig. 1a–c), and completely non-redundant patterns of 'noise' (Fig. 1d), can induce the nervous system to reveal characteristics that might have remained unsuspected without their use.


Fig. 1. a to c. Examples of visual stimuli incorporating a high degree of structural redundancy. d. A sample of 'visual noise' with minimal redundancy.
It is now commonplace to regard the impulses that flow along nerve fibres as 'conveying information', but just how information about the world is represented in the brain remains an unresolved question. Although nerve impulses do not function like the code signals in a digital computer, the general ideas of information engineering are proving increasingly useful in suggesting experiments to throw light on the way they do operate. One of the chief advantages of these ideas is that they belong in a sense to both the psychological and the physiological levels. They thus offer an invaluable working link or 'conceptual bridge' by which data and questions at either level can be brought to bear on hypotheses at the other.

For the application of information theory to the visual system, see information rate of vision.

(Published 1987)

— Donald MacKay

Bibliography
• Attneave, F. (1959). Applications of Information Theory to Psychology: A Summary of Basic Concepts, Methods and Results.
• Hick, W. E. (1952). 'On the rate of gain of information'. Quarterly Journal of Experimental Psychology, 4.
• MacKay, D. M. (1969). Information, Mechanism and Meaning.
• — —  (1970). 'Perception and brain function'. In Schmitt, F. O. (ed.), The Neurosciences: Second Study Program.
• Shannon, C. E., and Weaver, W. (1949). The Mathematical Theory of Communication.
• Wiener, N. (1948). Cybernetics.

---

elements of information theory