Judgements of identity or attractiveness depend on relatively enduring features of an individual's face. But important messages are also contained in the momentary configuration of the face as it speaks and expresses. It is these signals upon which we focus here.
1. Facial speech
2. Face gaze
3. Facial expressions
4. Non-verbal communication
1. Facial speech
Everyone lip-reads, not just those who are hard of hearing. We are much better at deciphering speech in a noisy environment if we can see as well as hear the speaker. Phonemic distinctions which are difficult to hear may easily be visible. For example, the difference between an 'F' and an 'S' in a postal or zip code may need spelling out when we give an address by telephone, but when the speaker is present the difference is easy to see — the lips close for an 'F' but not for an 'S'. Lip-reading is well demonstrated by the illusion first reported by McGurk and MacDonald (1976). If a face is shown mouthing one speech sound, but simultaneously the soundtrack plays a different sound, in some circumstances the perceiver hears a phoneme which is neither that seen nor heard, but a blend of the two. For example, an auditory 'ba' with a visual 'ga' will result in the perception of a 'da'. The brain appears to do the best job it can of reconciling the auditory information (for 'ba', that the vocal tract has been closed at the lips), with the visual signal (for 'ga' that the lips are open), and comes up with a compromise perception (the vocal tract must be closed further back — perhaps just behind the teeth, as in 'da'). However, the precise rules of combining these signals remain unclear (Summerfield 1987). Neuroimaging has shown that silent lip-read speech activates areas of the cortex which otherwise respond only to auditory signals, suggesting that the integration of visual with heard speech sounds occurs at a very early level of cortical analysis (Calvert et al. 1997).Lip-reading skills dissociate from other aspects of face perception. Campbell, Landis, and Regard (1986) studied two patients, one of whom had impaired face perception but intact lip-reading, while the other was proficient at all face processing tasks except lip-reading. So, the first patient could tell that pursed lips sounded 'p', but not that downturned ones signalled sadness, while the second showed the opposite pattern. Such patterns of dissociation are strongly suggestive that the neural structures which allow us to lip-read do not overlap with those that are used to derive other kinds of facial meaning. Research with non-brain-injured volunteers has also shown that the McGurk 'blend' illusion is unaffected by whether or not the face and voice match in gender. The brain will combine a male spoken 'ba' with a female mouthing 'ga', suggesting that the visual analysis of face gender and the use of visual speech are kept quite distinct (Green et al. 1991). However, when the faces and voices are of familiar individuals this blending is suppressed (Walker, Bruce, and O'Malley 1995), suggesting that audio-visual speech perception is not completely immune from other aspects of person perception.
2. Face gaze
The direction and timing of movements of the eyes provide important cues which regulate conversation and can convey a sense of intimacy or dominance (see Kleinke 1986 for a review). We may stare directly at people if we wish to intimidate them, and look down when embarrassed or ashamed. People look away from their conversational partners when they are speaking but look at the other person when it is their turn to speak. Our gaze patterns are also informative about the locus of our attention. We look away or upwards when we are thinking, and look at an object that captures our attention. Recent work on gaze and perception has shown how other people's gaze patterns have cognitive consequences for the observer. A shift in gaze triggers a shift in direction of attention by the observer, who finds it easier to detect targets in the direction indicated by the gaze shift. For example, if a face is seen to gaze to the left, an observer will spot targets to the left more easily than those to the right — showing that the perceived gaze shift has had an effect on the observer's attentional focus. Such observations demonstrate how gaze can mediate shared attention between people. Such shifts appear to occur automatically, and at an early age. Some models of social attention suggest that it is the eyes that are the main key to other minds, and that people suffering from autism may be impaired in their use of such signals from gaze. However, developmentally, and evolutionarily, manual pointing seems to be the most salient cue, followed by head direction, with eye gaze cues being used later in the developmental sequence and only by higher primates (Langton, Watt, and Bruce 2000 provide a recent review). (See also eye contact.)3. Facial expressions
The extensive facial musculature pulls the face into distinctive postures which reveal different emotional states. According to Ekman (e.g. 1992) there is a small number of expressive categories which are universally perceived by people of all cultures (though cultures may differ in the 'display rules' which dictate when certain emotions are appropriate to display). The universal basic emotions include happiness, sadness, anger, fear, disgust, and surprise. Recent research into the neuropsychological underpinnings of emotion has revealed that part of the brain called the amygdala is strongly involved in the perception and expression of negative emotions, particularly fear. Patients with damage to the amygdala are poor at recognizing fear, from voices as well as from faces, and in normal people the amygdala becomes active when faces are more fearful or less happy. Work with patients with Huntington's disease has implicated different brain areas, including the insula, in the perception of disgust (Calder, Lawrence, and Young 2001 review these findings). However, other psychologists have been critical of the idea that there is a small number of discrete expression categories which are perceived universally (e.g. see Russell 1994).Much work on the perception of emotions from faces has examined the perception of static images, whereas inzeveryday life we perceive faces in continuous change. Accuracy and intensity of expression perception seem affected by the timing as well as the spatial pattern of the facial gesture (Kamachi et al. 2001). Kamachi and colleagues used image sequences in which a neutral face transformed into an expressive one at different speeds. Sadness was seen more intensely when sequences unfolded slowly, while happiness appeared more intense when changes occurred quickly.
4. Non-verbal communication
Given the important information conveyed by the face, in association with other gestures, it is perhaps not surprising that communication can suffer when people cannot see each other. Speech is harder to hear when faces are not visible, but conversational structure may also be affected more subtly when adults communicate by telephone. For example, nodding the head can signal to a speaker that the listener is still attending, and vocalizations such as 'uh-huh' may be needed to substitute when speakers rely on voice alone. Video-mediated communication does not necessarily substitute simply for face to face, and considerable research is being conducted to try to optimize the design of remote VMC systems to capture the best aspects of face-to-face communication (e.g. Doherty-Sneddon et al. 1997).While adults can adapt well to different communication media, young children may not. Young children depend more upon non-verbal cues than do adults, and this dependence can have both positive and negative effects. Children are generally better at communicating when they can see as well as hear each other, since they can use gestures to supplement speech, particularly for things that may be difficult to express in words.
However, the social presence of an adult can have an intimidating effect on a child. Recent research has shown that children may be more resistant to leading questions when they communicate via a live video link rather than face to face — perhaps because the link removes some of the social pressure (Doherty-Sneddon and McAuley 2000).
(Published 2004)
— Vicki Bruce
- Bibliography
- Bruce, V., and Young, A. (1998). In the Eye of the Beholder: The Science of Face Perception.
- Calder, A. J., Lawrence, A. D., and Young, A. W. (2001). 'Neuropsychology of fear and loathing'. Nature Reviews Neuroscience, 2.
- Calvert, G. A., Bullmore, E. T., Brammer, M. J., et al. (1997). 'Activation of auditory cortex during silent lipreading'. Science, 276.
- Campbell, R., Landis, T., and Regard, M. (1986). 'Face recognition and lipreading: a neurological dissociation'. Brain, 109.
- Doherty-Sneddon G., and McAuley, S. (2000). 'Influence of video-mediation on adult–child interviews: Implications for the use of the live link with child witnesses'. Applied Cognitive Psychology, 14.
- — — Anderson, A., O'Malley, C., Langton, S., Garrod, S., and Bruce, V. (1997). 'Face-to-face and video-mediated communication: a comparison of dialogue structure and task performance'. Journal of Experimental Psychology: Applied, 3.
- Ekman, P. (1992). 'Facial expressions of emotion: an old controversy and new findings'. Philosophical Transactions of the Royal Society of London, B, 335.
- Green, K. P., Kuhl, P. K., Meltzoff, A. N., and Stevens, E. B. (1991). 'Integrating speech information across talkers, gender and sensory modality: female faces and male voices in the McGurk effect'. Perception & Psychophysics, 50.
- Kamachi, M., Bruce, V., Mukaida, S., Gyoba, J., Yoshikawa S., and Akamatsu, S. (2001). 'Dynamic properties influence the perception of facial expressions'. Perception, 30.
- Kleinke, C. L. (1986). 'Gaze and eye contact: a research review'. Psychological Bulletin, 100.
- Langton, S. R. H., Watt, R. J., and Bruce, V. (2000), 'Do the eyes have it? Cues to the direction of social attention'. Trends in Cognitive Sciences, 4/2.
- McGurk, H., and MacDonald, J. (1976). 'Hearing lips and seeing voices'. Nature, 264.
- Russell, J. A. (1994). 'Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies'. Psychological Bulletin, 115.
- Summerfield, Q. (1987). 'Some preliminaries to a comprehensive account of audio-visual speech perception'. In Dodd, B., and Campbell, R. (eds.), Hearing by Eye: The Psychology of Lip-Reading.
- Walker, S., Bruce, V., and O'Malley, C. (1995). 'Facial identity and facial speech processing: familiar faces and voices in the McGurk effect'. Perception & Psychophysics, 57.
