retinex theory and colour constancy

As we view an object from different distances, at different angles, and in different illuminations, there occur vast changes in the physical image on our retina, but our sensations prove much more stable than would be expected from our changeful retinal image. One of the several 'constancies' that characterize our sensory experience is colour constancy. Just as our visual system is built to tell us about the permanent size of objects rather than about the ever-fluctuating size of our retinal image, so too it is built to tell us about the permanent colours of objects rather than about the spectral composition of the light falling on a local area of retina. The spectral composition of the light, that is, the relative proportions of the different wavelengths it contains, will depend on two factors:

(i) the spectral reflectance of the object, its tendency to reflect some wavelengths more than others, and
(ii) the spectral composition of the illuminant, the relative proportions of different wavelengths in the light that falls on the object.

When we pass from, say, tungsten illumination, which is rich in long wavelengths, to the bluer environment of daylight, our perception of an object's colour remains dependent on the object's spectral reflectance, and we are aware of little change. To achieve this constancy, the visual system must be taking into account not merely the local absorptions in the three classes of cone cell (see colour vision: eye mechanisms) but also the pattern of absorptions in other parts of the visual field, for the local absorptions can depend only on the local spectral flux, and the latter, being dependent on factors (i) and (ii) above, will vary greatly as the illumination changes.

A particularly impressive and instructive demonstration of colour constancy has been provided by Edwin Land. He performed an experiment in which a single mixture of red, green, and blue light produced many different colour sensations. Observers reported white, pink, green, red, brown, yellow, purple, blue, and black sensations from identical mixtures of red, green, and blue lights. The experiment used a large, complex display that Land called a 'colour Mondrian'. The display had approximately 100 different matt papers arranged arbitrarily so that each colour was surrounded by several others. The display was illuminated with three projectors, each with a different broad-band interference filter. One filter transmitted long-wave, or reddish light; one transmitted middle-wave, or greenish light; and one transmitted short-wave, or bluish light. Each projector had an independent brightness control. The observers picked an image segment — say a white paper — and the experimenter measured the amounts of red, green, and blue light coming from the white paper. Then the observers picked a second paper — say a red one — and the experimenter measured the amounts of red, green, and blue coming to each observer's eyes from the red paper. The measurements showed that roughly the same amount of red light and much less green and blue light are reflected from a red paper. The experimenter then changed the illumination so that the red light from the red paper was exactly equal to the red light from the white paper. This was a small change. The experimenter then substantially increased the brightness of the other two projectors so that exactly the same amounts of green and blue light came from the red paper as had previously come from the white paper. When all three projectors were turned on together, each observer reported the sensation red despite the fact that the physical properties of the light from that image region were the same as that from the white with the previous illumination. In this manner, Land went from paper to paper and showed that nearly the full gamut of colour sensations can be produced from a single mixture of red, green, and blue light.

This experiment led Land to propose that the information from the long-wave receptors is intercompared to compute a biological analogue of reflectance from the long-wave light across the entire image. Similarly, the information from the middle-wave receptors is intercompared to form the biological analogue of middle-wave reflectance, and short-wave information for short-wave reflectance. The information from each of the sets of receptor mechanisms generates a separate lightness image; the comparison of three lightnesses is the determinant of colour. In Land's theory, which he called the retinex theory, these three lightnesses provide the coordinates of a three-dimensional space. Whereas a colour space based on the absolute absorptions in the three classes of receptor will predict only whether or not two physical stimuli will match, a space based on the three lightnesses of retinex theory will predict how colours actually look. For, between them, the three lightnesses give the reflectance of the object in different parts of the spectrum — in other words, its spectral reflectance.

The formation of lightnesses and their comparison could occur in the retina or in the cortex. Land coined the word retinex — a combination of retina and cortex — to designate the mechanism that generates independent long-, middle-, and short-wave lightness images. This system independence does not require that the retinal receptors with the same spectral sensitivity are directly connected to each other. Instead it argues that the retinal–cortical structure acts in total, as if all the colour mechanisms of the same sensitivity form independent lightness images.

McCann, McKee, and Taylor (1976) described a series of quantitative experiments that were patterned after Land's. In these experiments they measured the sensations of each area in a 'Mondrian' using a series of different illuminants. In each situation they measured the sensation by matching each area in the display to a Munsell book of colours in constant illumination. Secondly, they tested how well the sensations correlated with the reflectance of the papers as measured with spectrophotometers. They showed that there was an excellent correlation between sensation and reflectance measured with light meters that had the same spectral sensitivity as the three human cone pigments. (This correlation was particularly good when the reflectance numbers were scaled by Glasser's Munsell lightness function.)

Land and McCann proposed a lightness model that is based on the comparison of receptor-mechanism responses from all parts of the image. This lightness model does not use local averages or global averages, but rather comparisons that are based on the relationships of image segments. Local relationships are calculated by the visual system using the ratio of energies at nearby points, and this information is propagated to other parts of the image by multiplying ratios to form products. These products propagate relationships across the entire image. The mean of many different products is used as the prediction of lightness.

McCann, McKee, and Taylor showed that the ratio-product lightness model combined with retinex colour-mechanism independence accurately predicted the colour sensation reported by the observer in the 'colour Mondrian' experiments.

The rods provide an opportunity to study sets of receptors forming an image in terms of lightness. With the appropriate low-intensity light source it is possible to see, below cone threshold, that the rods interact to form a lightness image — just as above cone threshold, lightness of an image segment is dependent on the relationship of objects in all parts of the image. With a low-intensity light that is very rich in long-wave light, it is possible to see a wide variety of colour sensations from rod and long-wave cone interactions. Colour is determined by the lightness of the image generated by the rods and the lightness generated by the long-wave colour mechanism.

In summary, Land's retinex theory proposed that colour is determined by three lightnesses — each computed from comparisons using intensity information from the entire image. Each lightness is computed independently using intensity information from each spectral region. The Land and McCann lightness model uses the multiplication of ratios to form products that relate each image segment to each of the others. The mean of many different normalized products is used to predict lightness. Quantitative tests have shown that retinex colour-mechanism independence and the ratio-product lightness model can predict colour sensations in experiments with unknown changes in the spectral distribution of the illumination.

(Published 1987)

— J. J. McCann

Bibliography
• Land, E. H. (1964). 'The retinex'. American Scientist, 52.
• — —  (1977). 'The retinex theory of color vision'. Scientific American, 237/6.
• — —  and McCann, J. J. (1971). 'Lightness and retinex theory'. Journal of the Optical Society of America, 61.
• McCann, J. J. (1973). 'Human color perception'. In Color Theory and Imaging Systems.
• — —  and Houston, K. L. (1983). 'Color sensation, color perception and mathematical models of color vision'. In Mollon, J. D., and Sharpe, L. T., Colour Vision.
• — —  McKee, S., and Taylor, T. H. (1976). 'Quantitative studies in retinex theory'. Vision Research, 16.

---