When Mismatched Voices and Lips Make Your Brain Play Tricks
The good news is, the human brain is
flexible and efficient. This helps us make sense of the world. But the
bad news is, the human brain is flexible and efficient. This means the
brain can sometimes make mistakes.
You
can watch this tension play out when the brain tries to connect
auditory and visual speech. It’s why we may find a poorly dubbed kung fu
movie hard to believe, and why we love believing the gibberish in those
Bad Lip Reading Videos on YouTube.
“By
dubbing speech that is reasonably consistent with the available mouth
movements, we can utterly change the meaning of what the original talker
was saying,” said John Magnotti,
a neuroscientist at Baylor College of Medicine in Texas. “Sometimes we
can detect that something is a little off, but the illusion is usually
quite compelling.”
In a study published Thursday in PLOS Computational Biology, Dr. Magnotti and Michael Beauchamp,
also a neuroscientist at Baylor College of Medicine, tried to pin down
why our brains are susceptible to these kinds of perceptual mistakes by
looking at a well-known speech illusion called the McGurk effect. By
comparing mathematical models for how the brain integrates senses
important in detecting speech, they found that the brain uses vision,
hearing and experience when making sense of speech. If the mouth and
voice are likely to come from the same person, the brain combines them;
otherwise, they are kept separate.
“You
may think that when you’re talking to someone you’re just listening to
their voice,” said Dr. Beauchamp, who led the study. “But it turns out
that what their face is doing is actually profoundly influencing what
you are perceiving.”
The McGurk effect was discovered
serendipitously when a psychologist named Harry McGurk mistakenly
believed he heard the sound “da” after watching a video in which the
sound “ba” was dubbed over a girl’s lips saying “ga.”
Forty
years later, scientists have demonstrated this effect in thousands of
studies, in infants and adults, in different languages, in cases where
the genders of the voice and mouth don’t match and even in cases when a
person touches, instead of watches, a moving mouth.
But
the McGurk effect is also limited. If you close your eyes and just
listen, you’ll hear “ba,” and if you watch the lips silently, you’ll
hear “ga.” If you reverse the dub, you don’t integrate the sounds
either. The effect accompanies only certain syllables.
Why do some mouth and voice combinations produce the illusion and others do not?
As
we encounter people talking — voices babbling, mouths moving — our
brains are constantly making calculations to help them pin down what
voices belong to what mouths. The brain asks itself: “Is it likely that
these two things go together or not?” Dr. Magnotti said.
The McGurk effect is an instance when we mistakenly put them together.
“If
they’re similar enough that there is something in the middle that
matches with them, then you go and put them together,” Dr. Beauchamp
said. That’s how “ba” and “ga” appeared to turn into the sound “da” for
Dr. McGurk.
This discrepancy, where
the sound “ba” and the mouth making “ga” do not equal the sound “ga” and
the mouth making “ba,” exists because the brain chooses a likely voice
and mouth match based on its experience with particular combinations in
the past, the researchers say. The “ga” sound just doesn’t emerge from
the closed lips it takes to say “ba.”
“In
science we’ve assumed that people do that, always put together
everything,” Dr. Beauchamp said. “It turns out that might not be right.”
He
thinks the same decision process governing the McGurk effect causes us
to put together President Trump’s lip movements with the bizarre things
we hear on the soundtrack in the Bad Lip Reading video of his inauguration speech.
“The
idea is that in the real world, if a voice and a mouth movement are so
time-locked, they would almost have to be from the same talker,” Dr.
Magnotti said. “As long as they match well enough, you can still see
something that makes sense.”
In a complementary study published this month in The Journal of Neuroscience, Dr. Beauchamp and his student Lin Zhu
determined that the brain most likely processes information about lip
movements and sound in a part of the brain called the superior temporal
sulcus.
Knowing where or how the brain
makes these mistakes may one day help clinicians understand and improve
your hearing as you age, Dr. Magnotti said. For now, you can keep
enjoying the funny lip-reading videos.
posted by dubbing at 3:37 AM
0 comments
