Real-time Classification of Auditory Sentences Using Evoked Cortical Activity in Humans

Overview

Journal J Neural Eng

Specialties Biomedical Engineering
Neurology

Date 2018 Jan 31

PMID 29378977

Citations 18

Authors

David A Moses

Matthew K Leonard

Edward F Chang

Affiliations

Soon will be listed here.

Abstract

Objective: Recent research has characterized the anatomical and functional basis of speech perception in the human auditory cortex. These advances have made it possible to decode speech information from activity in brain regions like the superior temporal gyrus, but no published work has demonstrated this ability in real-time, which is necessary for neuroprosthetic brain-computer interfaces.

Approach: Here, we introduce a real-time neural speech recognition (rtNSR) software package, which was used to classify spoken input from high-resolution electrocorticography signals in real-time. We tested the system with two human subjects implanted with electrode arrays over the lateral brain surface. Subjects listened to multiple repetitions of ten sentences, and rtNSR classified what was heard in real-time from neural activity patterns using direct sentence-level and HMM-based phoneme-level classification schemes.

Main Results: We observed single-trial sentence classification accuracies of [Formula: see text] or higher for each subject with less than 7 minutes of training data, demonstrating the ability of rtNSR to use cortical recordings to perform accurate real-time speech decoding in a limited vocabulary setting.

Significance: Further development and testing of the package with different speech paradigms could influence the design of future speech neuroprosthetic applications.

Citing Articles

A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages.

Silva A, Liu J, Metzger S, Bhaya-Grossman I, Dougherty M, Seaton M Nat Biomed Eng. 2024; 8(8):977-991.

PMID: 38769157 PMC: 11554235. DOI: 10.1038/s41551-024-01207-5.

A high-performance neuroprosthesis for speech decoding and avatar control.

Metzger S, Littlejohn K, Silva A, Moses D, Seaton M, Wang R Nature. 2023; 620(7976):1037-1046.

PMID: 37612505 PMC: 10826467. DOI: 10.1038/s41586-023-06443-4.

Wingfield C, Zhang C, Devereux B, Fonteneau E, Thwaites A, Liu X Front Comput Neurosci. 2023; 16:1057439.

PMID: 36618270 PMC: 9811675. DOI: 10.3389/fncom.2022.1057439.

Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis.

Metzger S, Liu J, Moses D, Dougherty M, Seaton M, Littlejohn K Nat Commun. 2022; 13(1):6510.

PMID: 36347863 PMC: 9643551. DOI: 10.1038/s41467-022-33611-3.

Dataset of Speech Production in intracranial.Electroencephalography.

Verwoert M, Ottenhoff M, Goulis S, Colon A, Wagner L, Tousseyn S Sci Data. 2022; 9(1):434.

PMID: 35869138 PMC: 9307753. DOI: 10.1038/s41597-022-01542-9.

References

Leonard M, Baud M, Sjerps M, Chang E . Perceptual restoration of masked speech in human cortex. Nat Commun. 2016; 7:13619. PMC: 5187421. DOI: 10.1038/ncomms13619. View

Rauschecker J, Scott S . Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing. Nat Neurosci. 2009; 12(6):718-24. PMC: 2846110. DOI: 10.1038/nn.2331. View

Leuthardt E, Miller K, Schalk G, Rao R, Ojemann J . Electrocorticography-based brain computer interface--the Seattle experience. IEEE Trans Neural Syst Rehabil Eng. 2006; 14(2):194-8. DOI: 10.1109/TNSRE.2006.875536. View

. Recommendations for use of uniform nomenclature pertinent to patients with severe alterations in consciousness. American Congress of Rehabilitation Medicine. Arch Phys Med Rehabil. 1995; 76(2):205-9. DOI: 10.1016/s0003-9993(95)80031-x. View

Mugler E, Patton J, Flint R, Wright Z, Schuele S, Rosenow J . Direct classification of all American English phonemes using signals from functional speech motor cortex. J Neural Eng. 2014; 11(3):035015. PMC: 4097188. DOI: 10.1088/1741-2560/11/3/035015. View

Sellers E, Ryan D, Hauser C . Noninvasive brain-computer interface enables communication after brainstem stroke. Sci Transl Med. 2014; 6(257):257re7. PMC: 4711808. DOI: 10.1126/scitranslmed.3007801. View

Spuler M, Rosenstiel W, Bogdan M . Online adaptation of a c-VEP Brain-computer Interface(BCI) based on error-related potentials and unsupervised learning. PLoS One. 2012; 7(12):e51077. PMC: 3517594. DOI: 10.1371/journal.pone.0051077. View

Kellis S, Miller K, Thomson K, Brown R, House P, Greger B . Decoding spoken words using local field potentials recorded from the cortical surface. J Neural Eng. 2010; 7(5):056007. PMC: 2970568. DOI: 10.1088/1741-2560/7/5/056007. View

Moses D, Mesgarani N, Leonard M, Chang E . Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity. J Neural Eng. 2016; 13(5):056004. PMC: 5031534. DOI: 10.1088/1741-2560/13/5/056004. View

10.

Crone N, Miglioretti D, Gordon B, Lesser R . Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band. Brain. 1999; 121 ( Pt 12):2301-15. DOI: 10.1093/brain/121.12.2301. View

11.

Martin S, Brunner P, Iturrate I, Millan J, Schalk G, Knight R . Word pair classification during imagined speech using direct brain recordings. Sci Rep. 2016; 6:25803. PMC: 4863149. DOI: 10.1038/srep25803. View

12.

Pei X, Barbour D, Leuthardt E, Schalk G . Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J Neural Eng. 2011; 8(4):046028. PMC: 3772685. DOI: 10.1088/1741-2560/8/4/046028. View

13.

Mainsah B, Collins L, Colwell K, Sellers E, Ryan D, Caves K . Increasing BCI communication rates with dynamic stopping towards more practical use: an ALS study. J Neural Eng. 2015; 12(1):016013. PMC: 4631027. DOI: 10.1088/1741-2560/12/1/016013. View

14.

Hamilton L, Chang D, Lee M, Chang E . Semi-automated Anatomical Labeling and Inter-subject Warping of High-Density Intracranial Recording Electrodes in Electrocorticography. Front Neuroinform. 2017; 11:62. PMC: 5671481. DOI: 10.3389/fninf.2017.00062. View

15.

Binder J, Frost J, Hammeke T, Bellgowan P, Springer J, Kaufman J . Human temporal lobe activation by speech and nonspeech sounds. Cereb Cortex. 2000; 10(5):512-28. DOI: 10.1093/cercor/10.5.512. View

16.

Hotson G, McMullen D, Fifer M, Johannes M, Katyal K, Para M . Individual finger control of a modular prosthetic limb using high-density electrocorticography in a human subject. J Neural Eng. 2016; 13(2):026017-26017. PMC: 4875758. DOI: 10.1088/1741-2560/13/2/026017. View

17.

Mesgarani N, Cheung C, Johnson K, Chang E . Phonetic feature encoding in human superior temporal gyrus. Science. 2014; 343(6174):1006-10. PMC: 4350233. DOI: 10.1126/science.1245994. View

18.

Vansteensel M, Pels E, Bleichner M, Branco M, Denison T, Freudenburg Z . Fully Implanted Brain-Computer Interface in a Locked-In Patient with ALS. N Engl J Med. 2016; 375(21):2060-2066. PMC: 5326682. DOI: 10.1056/NEJMoa1608085. View

19.

Pasley B, David S, Mesgarani N, Flinker A, Shamma S, Crone N . Reconstructing speech from human auditory cortex. PLoS Biol. 2012; 10(1):e1001251. PMC: 3269422. DOI: 10.1371/journal.pbio.1001251. View

20.

Laureys S, Pellas F, Van Eeckhout P, Ghorbel S, Schnakers C, Perrin F . The locked-in syndrome : what is it like to be conscious but paralyzed and voiceless?. Prog Brain Res. 2005; 150:495-511. DOI: 10.1016/S0079-6123(05)50034-7. View