Consonant-Vowel Transition Models Based on Deep Learning for Objective Evaluation of Articulation

Overview

Journal IEEE/ACM Trans Audio Speech Lang Process

Date 2023 Jan 30

PMID 36712557

Authors

Vikram C Mathad

Julie M Liss

Kathy Chapman

Nancy Scherer

Visar Berisha

Affiliations

Soon will be listed here.

Abstract

Spectro-temporal dynamics of consonant-vowel (CV) transition regions are considered to provide robust cues related to articulation. In this work, we propose an objective measure of precise articulation, dubbed the objective articulation measure (OAM), by analyzing the CV transitions segmented around vowel onsets. The OAM is derived based on the posteriors of a convolutional neural network pre-trained to classify between different consonants using CV regions as input. We demonstrate that the OAM is correlated with perceptual measures in a variety of contexts including (a) adult dysarthric speech, (b) the speech of children with cleft lip/palate, and (c) a database of accented English speech from native Mandarin and Spanish speakers.

Citing Articles

Responsible development of clinical speech AI: Bridging the gap between clinical research and technology.

Berisha V, Liss J NPJ Digit Med. 2024; 7(1):208.

PMID: 39122889 PMC: 11316053. DOI: 10.1038/s41746-024-01199-1.

Dysarthria detection based on a deep learning model with a clinically-interpretable layer.

Xu L, Liss J, Berisha V JASA Express Lett. 2023; 3(1):015201.

PMID: 36725533 PMC: 9835557. DOI: 10.1121/10.0016833.

References

Kalita S, Prasanna S, Dandapat S . Intelligibility assessment of cleft lip and palate speech using Gaussian posteriograms based on joint spectro-temporal features. J Acoust Soc Am. 2018; 144(4):2413. DOI: 10.1121/1.5064463. View

Stevens K . Toward a model for lexical access based on acoustic landmarks and distinctive features. J Acoust Soc Am. 2002; 111(4):1872-91. DOI: 10.1121/1.1458026. View

Saxon M, Liss J, Berisha V . OBJECTIVE MEASURES OF PLOSIVE NASALIZATION IN HYPERNASAL SPEECH. Proc IEEE Int Conf Acoust Speech Signal Process. 2020; 2019:6520-6524. PMC: 6954066. DOI: 10.1109/ICASSP.2019.8682339. View

Henningsson G, Kuehn D, Sell D, Sweeney T, Trost-Cardamone J, Whitehill T . Universal parameters for reporting speech outcomes in individuals with cleft palate. Cleft Palate Craniofac J. 2008; 45(1):1-17. DOI: 10.1597/06-086.1. View

Ohde R, Haley K, Barnes C . Perception of the [m]-[n] distinction in consonant-vowel (CV) and vowel-consonant (VC) syllables produced by child and adult talkers. J Acoust Soc Am. 2006; 119(3):1697-711. DOI: 10.1121/1.2140830. View

Hedrick M, Ohde R . Effect of relative amplitude of frication on perception of place of articulation. J Acoust Soc Am. 1993; 94(4):2005-26. DOI: 10.1121/1.407503. View

Stegmann G, Hahn S, Liss J, Shefner J, Rutkove S, Kawabata K . Repeatability of Commonly Used Speech and Language Features for Clinical Applications. Digit Biomark. 2021; 4(3):109-122. PMC: 7772887. DOI: 10.1159/000511671. View

Sussman H, Hoemeke K, Ahmed F . A cross-linguistic investigation of locus equations as a phonetic descriptor for place of articulation. J Acoust Soc Am. 1993; 94(3 Pt 1):1256-68. DOI: 10.1121/1.408178. View

DARLEY F, Aronson A, Brown J . Differential diagnostic patterns of dysarthria. J Speech Hear Res. 1969; 12(2):246-69. DOI: 10.1044/jshr.1202.246. View

10.

Kim H, Martin K, Hasegawa-Johnson M, Perlman A . Frequency of consonant articulation errors in dysarthric speech. Clin Linguist Phon. 2010; 24(10):759-70. DOI: 10.3109/02699206.2010.497238. View

11.

Berisha V, Krantsevich C, Hahn P, Hahn S, Dasarathy G, Turaga P . Digital medicine and the curse of dimensionality. NPJ Digit Med. 2021; 4(1):153. PMC: 8553745. DOI: 10.1038/s41746-021-00521-5. View

12.

Kent R, Kim Y . Toward an acoustic typology of motor speech disorders. Clin Linguist Phon. 2003; 17(6):427-45. DOI: 10.1080/0269920031000086248. View

13.

Pisoni D, Studdert-Kennedy M . Perception of static and dynamic acoustic cues to place of articulation in initial stop consonants. J Acoust Soc Am. 1983; 73(5):1779-93. PMC: 3515858. DOI: 10.1121/1.389402. View

14.

Strombergsson S, Salvi G, House D . Acoustic and perceptual evaluation of category goodness of /t/ and /k/ in typical and misarticulated children's speech. J Acoust Soc Am. 2015; 137(6):3422-35. DOI: 10.1121/1.4921033. View

15.

Saxon M, Tripathi A, Jiao Y, Liss J, Berisha V . Robust Estimation of Hypernasality in Dysarthria with Acoustic Model Likelihood Features. IEEE/ACM Trans Audio Speech Lang Process. 2021; 28:2511-2522. PMC: 7978228. DOI: 10.1109/taslp.2020.3015035. View

16.

Mahr T, Berisha V, Kawabata K, Liss J, Hustad K . Performance of Forced-Alignment Algorithms on Children's Speech. J Speech Lang Hear Res. 2021; 64(6S):2213-2222. PMC: 8740721. DOI: 10.1044/2020_JSLHR-20-00268. View

17.

Chapman K, Baylis A, Trost-Cardamone J, Cordero K, Dixon A, Dobbelsteyn C . The Americleft Speech Project: A Training and Reliability Study. Cleft Palate Craniofac J. 2014; 53(1):93-108. PMC: 5693235. DOI: 10.1597/14-027. View