Matching Heard and Seen Speech: An ERP Study of Audiovisual Word Recognition

Overview

Journal Brain Lang

Publisher Elsevier

Specialties Neurology
Otorhinolaryngology
Psychiatry

Date 2016 May 8

PMID 27155219

Citations 3

Authors

Natalya Kaganovich

Jennifer Schumaker

Courtney Rowland

Affiliations

Soon will be listed here.

Abstract

Seeing articulatory gestures while listening to speech-in-noise (SIN) significantly improves speech understanding. However, the degree of this improvement varies greatly among individuals. We examined a relationship between two distinct stages of visual articulatory processing and the SIN accuracy by combining a cross-modal repetition priming task with ERP recordings. Participants first heard a word referring to a common object (e.g., pumpkin) and then decided whether the subsequently presented visual silent articulation matched the word they had just heard. Incongruent articulations elicited a significantly enhanced N400, indicative of a mismatch detection at the pre-lexical level. Congruent articulations elicited a significantly larger LPC, indexing articulatory word recognition. Only the N400 difference between incongruent and congruent trials was significantly correlated with individuals' SIN accuracy improvement in the presence of the talker's face.

Citing Articles

Electrophysiological Dynamics of Visual Speech Processing and the Role of Orofacial Effectors for Cross-Modal Predictions.

Michon M, Boncompte G, Lopez V Front Hum Neurosci. 2020; 14:538619.

PMID: 33192386 PMC: 7653187. DOI: 10.3389/fnhum.2020.538619.

The Cross-Modal Suppressive Role of Visual Context on Speech Intelligibility: An ERP Study.

Shen S, Kerlin J, Bortfeld H, Shahin A Brain Sci. 2020; 10(11).

PMID: 33147691 PMC: 7692090. DOI: 10.3390/brainsci10110810.

Atypical audiovisual word processing in school-age children with a history of specific language impairment: an event-related potential study.

Kaganovich N, Schumaker J, Rowland C J Neurodev Disord. 2016; 8(1):33.

PMID: 27597881 PMC: 5011345. DOI: 10.1186/s11689-016-9168-3.

References

Norrix L, Plante E, Vance R . Auditory-visual speech integration by adults with and without language-learning disabilities. J Commun Disord. 2005; 39(1):22-36. DOI: 10.1016/j.jcomdis.2005.05.003. View

Tye-Murray N, Sommers M, Spehar B . Auditory and visual lexical neighborhoods in audiovisual speech perception. Trends Amplif. 2007; 11(4):233-41. PMC: 4111531. DOI: 10.1177/1084713807307409. View

Praamstra P, Stegeman D . Phonological effects on the auditory N400 event-related brain potential. Brain Res Cogn Brain Res. 1993; 1(2):73-86. DOI: 10.1016/0926-6410(93)90013-u. View

Ten Oever S, Schroeder C, Poeppel D, van Atteveldt N, Zion-Golumbic E . Rhythmicity and cross-modal temporal cues facilitate detection. Neuropsychologia. 2014; 63:43-50. PMC: 4209287. DOI: 10.1016/j.neuropsychologia.2014.08.008. View

Hickok G, Okada K, Serences J . Area Spt in the human planum temporale supports sensory-motor integration for speech processing. J Neurophysiol. 2009; 101(5):2725-32. DOI: 10.1152/jn.91099.2008. View

McGrath M, Summerfield Q . Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. J Acoust Soc Am. 1985; 77(2):678-85. DOI: 10.1121/1.392336. View

Friedman D, Johnson Jr R . Event-related potential (ERP) studies of memory encoding and retrieval: a selective review. Microsc Res Tech. 2000; 51(1):6-28. DOI: 10.1002/1097-0029(20001001)51:1<6::AID-JEMT2>3.0.CO;2-R. View

Conrey B, Pisoni D . Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. J Acoust Soc Am. 2006; 119(6):4065-73. PMC: 3314884. DOI: 10.1121/1.2195091. View

Guiraud J, Tomalski P, Kushnerenko E, Ribeiro H, Davies K, Charman T . Atypical audiovisual speech integration in infants at risk for autism. PLoS One. 2012; 7(5):e36428. PMC: 3352915. DOI: 10.1371/journal.pone.0036428. View

10.

Praamstra P, Meyer A, Levelt W . Neurophysiological Manifestations of Phonological Processing: Latency Variation of a Negative ERP Component Timelocked to Phonological Mismatch. J Cogn Neurosci. 2013; 6(3):204-19. DOI: 10.1162/jocn.1994.6.3.204. View

11.

Groppe D, Urbach T, Kutas M . Mass univariate analysis of event-related brain potentials/fields I: a critical tutorial review. Psychophysiology. 2011; 48(12):1711-25. PMC: 4060794. DOI: 10.1111/j.1469-8986.2011.01273.x. View

12.

Leybaert J, Macchi L, Huyse A, Champoux F, Bayard C, Colin C . Atypical audio-visual speech perception and McGurk effects in children with specific language impairment. Front Psychol. 2014; 5:422. PMC: 4033223. DOI: 10.3389/fpsyg.2014.00422. View

13.

Stevenson R, Nelms C, Baum S, Zurkovsky L, Barense M, Newhouse P . Deficits in audiovisual speech perception in normal aging emerge at the level of whole-word recognition. Neurobiol Aging. 2014; 36(1):283-91. PMC: 4268368. DOI: 10.1016/j.neurobiolaging.2014.08.003. View

14.

Dodd B, McIntosh B, Erdener D, Burnham D . Perception of the auditory-visual illusion in speech perception by children with phonological disorders. Clin Linguist Phon. 2007; 22(1):69-82. DOI: 10.1080/02699200701660100. View

15.

Brysbaert M, New B . Moving beyond Kucera and Francis: a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behav Res Methods. 2009; 41(4):977-90. DOI: 10.3758/BRM.41.4.977. View

16.

Van Petten C, Coulson S, Rubin S, Plante E, Parks M . Time course of word identification and semantic integration in spoken language. J Exp Psychol Learn Mem Cogn. 1999; 25(2):394-417. DOI: 10.1037//0278-7393.25.2.394. View

17.

Grant K, SEITZ P . Measures of auditory-visual integration in nonsense syllables and sentences. J Acoust Soc Am. 1999; 104(4):2438-50. DOI: 10.1121/1.423751. View

18.

Foxe J, Molholm S, Del Bene V, Frey H, Russo N, Blanco D . Severe multisensory speech integration deficits in high-functioning school-aged children with Autism Spectrum Disorder (ASD) and their resolution during early adolescence. Cereb Cortex. 2013; 25(2):298-312. PMC: 4303800. DOI: 10.1093/cercor/bht213. View

19.

Barutchu A, Danaher J, Crewther S, Innes-Brown H, Shivdasani M, Paolini A . Audiovisual integration in noise by children and adults. J Exp Child Psychol. 2009; 105(1-2):38-50. DOI: 10.1016/j.jecp.2009.08.005. View

20.

Taylor N, Isaac C, Milne E . A comparison of the development of audiovisual integration in children with autism spectrum disorders and typically developing children. J Autism Dev Disord. 2010; 40(11):1403-11. DOI: 10.1007/s10803-010-1000-4. View