How do our brains extract information from faces?

Talk like a cognitive and social neuroscientist

Autism spectrum disorder (ASD) — a neurodevelopmental condition impacting social interaction and communication

Cognitive neuroscience — the study of the brain mechanisms underlying mental processes such as reasoning and remembering

Electroencephalography (EEG) — a technique for measuring electrical activity in the brain using electrodes placed on the scalp

Empathy — affective empathy is the ability to relate emotionally to the feelings of others, while cognitive empathy is the ability to take the perspective of and to feel concern for another without experiencing their emotion

Event related potentials (ERPs) — brain responses that are directly triggered by specific sensory, cognitive or motor events detected using EEG

Neurons — specialised cells in the brain and wider nervous system that transmit information through electrical and chemical signals

Social neuroscience — the study of how the brain processes and understands social behaviours and interactions

Theory of mind (ToM) — the ability to understand that other people have their own thoughts and feelings

Imagine you are meeting a friend outside your local shopping centre on a sunny Saturday morning. You spot them in the crowd of faces, greet them, and start to chat about your plans for the weekend. However, you notice they are frowning slightly, and their eyes keep darting elsewhere. You wonder what is on their mind.

Dr Roxane Itier, from the University of Waterloo, is interested in situations like this. “My research focuses on understanding how the brain extracts relevant information from the face to allow us to process identity, gender, emotion and social attention,” says Roxane. She is studying how factors such as empathy, personality traits and context influence this ability, and how this may change during our lives.

How does Roxane study the brain?

Roxane uses electroencephalography (EEG), a brain imaging technique, to study the temporal dynamics of cognitive processes – that is, when these processes happen. Her participants wear caps (that look like swimming caps) fitted with electrodes. These electrodes record people’s electrical brain activity while they look at pictures of faces on a screen and complete tasks such as recognising gender, emotions and familiar faces.

“EEG records the electrical fields created by large populations of hundreds of thousands to millions of neurons,” explains Roxane. “These electrical fields spread in real time through the brain and skull to the surface of the scalp, where the electrodes record them.” In addition to neural signals, EEG electrodes also record unwanted ‘noise’, caused by signals from activity such as eye movement, muscle activity and heart rate. Researchers spend a lot of time getting rid of this noise.

Roxane uses event related potentials (ERPs) to analyse the EEG signal. Roxane explains, “ERPs allow us to track cognitive processes as they unfold across time, from the moment a stimulus, such as a face, is presented onscreen, to the moment people press a button to register their decision, categorising the face as ‘new face’, ‘angry’ or ‘woman’, for example.”

How can eye tracking be combined with EEG?

One tool for tracking where on a face people look is a remote eye tracker, positioned at the base of the screen. “We ask participants not to move their eyes, and instead, we present a little cross that we ask them to fixate on,” explains Roxane. “When the eye tracker registers that the eyes are fixated on the cross, we present the picture of the face so that a specific feature, such as the mouth, is now located where the cross used to be.” The neural activity that is recorded when the participant fixates on one facial feature can then be compared with the activity that is recorded when they are looking at other features.

Roxane is interested in how changing the features that we look at impacts our facial processing. The experiments using the eye tracker have shown that where we look may influence how we recognise faces and how we identify other people’s emotions.

How do we identify emotions from facial expressions?

After our brain recognises that we are looking at a face, it looks for clues in facial features to identify emotions. For example, a smile may indicate happiness, whereas wide eyes can show fear. “The integration of all these pieces of information leads to the understanding of the emotion conveyed by the face, which we often categorise verbally as joy, anger, fear or disgust, for example,” explains Roxane.

This all takes place in less than half a second. Roxane’s research has found that our brains recognise emotions automatically, regardless of the task or context. However, these factors can affect later cognitive processes related to the motor or verbal response “The early neural activity related to emotion is automatic, but task demands do influence the behavioural response that follows,” explains Roxane.

How does the brain process theory of mind and empathy?

“Theory of mind (ToM) and empathy are very complex processes that are not fully understood,” says Roxane. Researchers have identified areas of the brain and neural networks related to ToM and empathy, but not how they unfold over time. “What we know is that extracting information about the face promotes the emergence of these complex processes,” says Roxane. “Someone looking sad or in pain will typically trigger a response of empathy from another person. However, ToM and empathy are modulated by a lot of factors including context and individual traits.”

Roxane is studying how different factors change the neural activity shown during empathy. She says, “These studies will help us determine whether the neural activity we record on the scalp can be used as a neural signature of empathy or not.”

What are the real-life implications of this work?

Not everyone finds it easy to identify faces or emotions. For example, individuals with an autism spectrum disorder (ASD) can find this difficult. “People with ASD often avoid looking at faces, and when they do, they typically avoid the eyes,” says Roxane. “They also often show impairments in theory of mind, but not in affective empathy.”

Previous research has not combined EEG with eye tracking, so whether differences in neural activity are due to participants with ASD avoiding faces or due to differences in their neural processes is unclear. Roxane believes that the use of eye trackers alongside EEG will help provide answers.

What are the next steps for the research?

“I am starting research on self-relevance and its impact on cognitive processes,” says Roxane. “For example, do we remember faces of individuals who express opinions of us better than faces of people who express opinions about others?”

Roxane is also exploring how individual differences, such as anxious tendencies, psychopathic traits, ASD behaviours and attention deficit hyperactivity disorder (ADHD), can impact neural activity, as many of us are affected by some of these conditions.