When Mismatched Voices and Lips Make Your Brain Play Tricks

 https://mobile.nytimes.com/2017/02/21/science/lip-reading-mcgurk-effect.html

The good news is, the human brain is flexible and efficient. This helps us make sense of the world. But the bad news is, the human brain is flexible and efficient. This means the brain can sometimes make mistakes.

You can watch this tension play out when the brain tries to connect auditory and visual speech. It’s why we may find a poorly dubbed kung fu movie hard to believe, and why we love believing the gibberish in those Bad Lip Reading Videos on YouTube.

“By dubbing speech that is reasonably consistent with the available mouth movements, we can utterly change the meaning of what the original talker was saying,” said John Magnotti, a neuroscientist at Baylor College of Medicine in Texas. “Sometimes we can detect that something is a little off, but the illusion is usually quite compelling.”

In a study published Thursday in PLOS Computational Biology, Dr. Magnotti and Michael Beauchamp, also a neuroscientist at Baylor College of Medicine, tried to pin down why our brains are susceptible to these kinds of perceptual mistakes by looking at a well-known speech illusion called the McGurk effect. By comparing mathematical models for how the brain integrates senses important in detecting speech, they found that the brain uses vision, hearing and experience when making sense of speech. If the mouth and voice are likely to come from the same person, the brain combines them; otherwise, they are kept separate.

“You may think that when you’re talking to someone you’re just listening to their voice,” said Dr. Beauchamp, who led the study. “But it turns out that what their face is doing is actually profoundly influencing what you are perceiving.”

The McGurk effect was discovered serendipitously when a psychologist named Harry McGurk mistakenly believed he heard the sound “da” after watching a video in which the sound “ba” was dubbed over a girl’s lips saying “ga.”

Forty years later, scientists have demonstrated this effect in thousands of studies, in infants and adults, in different languages, in cases where the genders of the voice and mouth don’t match and even in cases when a person touches, instead of watches, a moving mouth.

But the McGurk effect is also limited. If you close your eyes and just listen, you’ll hear “ba,” and if you watch the lips silently, you’ll hear “ga.” If you reverse the dub, you don’t integrate the sounds either. The effect accompanies only certain syllables.

Why do some mouth and voice combinations produce the illusion and others do not?

As we encounter people talking — voices babbling, mouths moving — our brains are constantly making calculations to help them pin down what voices belong to what mouths. The brain asks itself: “Is it likely that these two things go together or not?” Dr. Magnotti said.

The McGurk effect is an instance when we mistakenly put them together.

“If they’re similar enough that there is something in the middle that matches with them, then you go and put them together,” Dr. Beauchamp said. That’s how “ba” and “ga” appeared to turn into the sound “da” for Dr. McGurk.

This discrepancy, where the sound “ba” and the mouth making “ga” do not equal the sound “ga” and the mouth making “ba,” exists because the brain chooses a likely voice and mouth match based on its experience with particular combinations in the past, the researchers say. The “ga” sound just doesn’t emerge from the closed lips it takes to say “ba.”

“In science we’ve assumed that people do that, always put together everything,” Dr. Beauchamp said. “It turns out that might not be right.”

He thinks the same decision process governing the McGurk effect causes us to put together President Trump’s lip movements with the bizarre things we hear on the soundtrack in the Bad Lip Reading video of his inauguration speech.

“The idea is that in the real world, if a voice and a mouth movement are so time-locked, they would almost have to be from the same talker,” Dr. Magnotti said. “As long as they match well enough, you can still see something that makes sense.”

In a complementary study published this month in The Journal of Neuroscience, Dr. Beauchamp and his student Lin Zhu determined that the brain most likely processes information about lip movements and sound in a part of the brain called the superior temporal sulcus.

Knowing where or how the brain makes these mistakes may one day help clinicians understand and improve your hearing as you age, Dr. Magnotti said. For now, you can keep enjoying the funny lip-reading videos.