The Americleft Speech Project: A Training and Reliability Study

Overview

Journal Cleft Palate Craniofac J

Specialties Dentistry
General Surgery

Date 2014 Dec 23

PMID 25531738

Citations 25

Authors

Kathy L Chapman

Adriane Baylis

Judith Trost-Cardamone

Kelly Nett Cordero

Angela Dixon

Cindy Dobbelsteyn

Anna Thurmes

Kristina Wilson

Anne Harding-Bell

Triona Sweeney

Gregory Stoddard

Debbie Sell

Affiliations

Soon will be listed here.

Abstract

Objective: To describe the results of two reliability studies and to assess the effect of training on interrater reliability scores.

Design: The first study (1) examined interrater and intrarater reliability scores (weighted and unweighted kappas) and (2) compared interrater reliability scores before and after training on the use of the Cleft Audit Protocol for Speech-Augmented (CAPS-A) with British English-speaking children. The second study examined interrater and intrarater reliability on a modified version of the CAPS-A (CAPS-A Americleft Modification) with American and Canadian English-speaking children. Finally, comparisons were made between the interrater and intrarater reliability scores obtained for Study 1 and Study 2.

Participants: The participants were speech-language pathologists from the Americleft Speech Project.

Results: In Study 1, interrater reliability scores improved for 6 of the 13 parameters following training on the CAPS-A protocol. Comparison of the reliability results for the two studies indicated lower scores for Study 2 compared with Study 1. However, this appeared to be an artifact of the kappa statistic that occurred due to insufficient variability in the reliability samples for Study 2. When percent agreement scores were also calculated, the ratings appeared similar across Study 1 and Study 2.

Conclusion: The findings of this study suggested that improvements in interrater reliability could be obtained following a program of systematic training. However, improvements were not uniform across all parameters. Acceptable levels of reliability were achieved for those parameters most important for evaluation of velopharyngeal function.

Citing Articles

The Rate of Secondary Speech Surgery After Cleft Palate Repair: A Systematic Review.

Plonkowski A, Choi D, Naidu P, Turk M, Yao C, Magee 3rd W Plast Reconstr Surg Glob Open. 2025; 13(2):e6465.

PMID: 39995475 PMC: 11850049. DOI: 10.1097/GOX.0000000000006465.

Does Successful Surgical Treatment of Velopharyngeal Insufficiency Aid in the Remediation of Compensatory Misarticulation Errors?.

Chee-Williams J, Bunton K, Alvarez-Montoya E, Cordero K, Perry J, Philp J Am J Speech Lang Pathol. 2025; 34(2):868-876.

PMID: 39898817 PMC: 11902990. DOI: 10.1044/2024_AJSLP-24-00349.

Differential Diagnosis of a Pharyngeal Fricative and Therapeutic Monitoring of Velopharyngeal Function Using Magnetic Resonance Imaging.

Mason K, Botz E, Gampper T Am J Speech Lang Pathol. 2024; 34(1):1-11.

PMID: 39637256 PMC: 11745311. DOI: 10.1044/2024_AJSLP-24-00292.

Velopharyngeal Gap Size During Sustained Vowel Production Correlates With Perceptual Ratings of Hypernasality in Connected Speech.

Williams J, Perry J, Bunton K, Cordero K, Snodgrass T, Singh D J Speech Lang Hear Res. 2024; 67(9):2977-2986.

PMID: 39173051 PMC: 11427416. DOI: 10.1044/2024_JSLHR-24-00221.

The Levator Veli Palatini: Are all Segments Created Equal?.

Perry J, Snodgrass T, Gilbert I, Williams J, Singh D, Sitzman T Cleft Palate Craniofac J. 2024; 62(1):28-34.

PMID: 39136069 PMC: 11581906. DOI: 10.1177/10556656241271666.

References

Hutters B, Henningsson G . Speech outcome following treatment in cross-linguistic cleft palate studies: methodological implications. Cleft Palate Craniofac J. 2004; 41(5):544-9. DOI: 10.1597/02-164.1. View

Imatomi S . Effects of breathy voice source on ratings of hypernasality. Cleft Palate Craniofac J. 2005; 42(6):641-8. DOI: 10.1597/03-146.1. View

Dodd B, Holm A, Hua Z, Crosbie S . Phonological development: a normative study of British English-speaking children. Clin Linguist Phon. 2004; 17(8):617-43. DOI: 10.1080/0269920031000111348. View

Bassich C, Ludlow C . The use of perceptual methods by new clinicians for assessing voice quality. J Speech Hear Disord. 1986; 51(2):125-33. DOI: 10.1044/jshd.5102.125. View

Helou L, Solomon N, Henry L, Coppit G, Howard R, Stojadinovic A . The role of listener experience on Consensus Auditory-perceptual Evaluation of Voice (CAPE-V) ratings of postthyroidectomy voice. Am J Speech Lang Pathol. 2010; 19(3):248-58. DOI: 10.1044/1058-0360(2010/09-0012). View

Pereira V, Sell D, Tuomainen J . Effect of maxillary osteotomy on speech in cleft lip and palate: perceptual outcomes of velopharyngeal function. Int J Lang Commun Disord. 2013; 48(6):640-50. DOI: 10.1111/1460-6984.12036. View

Cordes A . The reliability of observational data: I. Theories and methods for speech-language pathology. J Speech Hear Res. 1994; 37(2):264-78. DOI: 10.1044/jshr.3702.264. View

Lee A, Whitehill T, Ciocca V . Effect of listener training on perceptual judgement of hypernasality. Clin Linguist Phon. 2009; 23(5):319-34. DOI: 10.1080/02699200802688596. View

Chan K, Yiu E . A comparison of two perceptual voice evaluation training programs for naive listeners. J Voice. 2005; 20(2):229-41. DOI: 10.1016/j.jvoice.2005.03.007. View

10.

Long Jr R, Hathaway R, Daskalogiannakis J, Mercado A, Russell K, Cohen M . The Americleft study: an inter-center study of treatment outcomes for patients with unilateral cleft lip and palate part 1. Principles and study design. Cleft Palate Craniofac J. 2011; 48(3):239-43. DOI: 10.1597/09-180.1. View

11.

Sell D, Harding A, Grunwell P . GOS.SP.ASS.'98: an assessment for speech disorders associated with cleft palate and/or velopharyngeal dysfunction (revised). Int J Lang Commun Disord. 1999; 34(1):17-33. DOI: 10.1080/136828299247595. View

12.

Kreiman J, Gerratt B, Precoda K . Listener experience and perception of voice quality. J Speech Hear Res. 1990; 33(1):103-15. DOI: 10.1044/jshr.3301.103. View

13.

Kreiman J, Gerratt B, Precoda K, Berke G . Individual differences in voice quality perception. J Speech Hear Res. 1992; 35(3):512-20. DOI: 10.1044/jshr.3503.512. View

14.

Gerratt B, Kreiman J, Berke G . Comparing internal and external standards in voice quality judgments. J Speech Hear Res. 1993; 36(1):14-20. DOI: 10.1044/jshr.3601.14. View

15.

Daniel H . Nasality ratings of single words, phrases, and running speech samples obtained from cleft palate children. Folia Phoniatr (Basel). 1971; 23(1):41-9. DOI: 10.1159/000263481. View

16.

Kreiman J, Gerratt B, Kempster G, Erman A, Berke G . Perceptual evaluation of voice quality: review, tutorial, and a framework for future research. J Speech Hear Res. 1993; 36(1):21-40. DOI: 10.1044/jshr.3601.21. View

17.

Cordes A, Ingham R . The reliability of observational data: II. Issues in the identification and measurement of stuttering events. J Speech Hear Res. 1994; 37(2):279-94. View

18.

Baylis A, Chapman K, Whitehill T . Validity and Reliability of Visual Analog Scaling for Assessment of Hypernasality and Audible Nasal Emission in Children With Repaired Cleft Palate. Cleft Palate Craniofac J. 2014; 52(6):660-70. DOI: 10.1597/14-040. View

19.

Lohmander A, Olsson M . Methodology for perceptual assessment of speech in patients with cleft palate: a critical review of the literature. Cleft Palate Craniofac J. 2003; 41(1):64-70. DOI: 10.1597/02-136. View

20.

Gooch J, Chapman K, Sussman J . Reliability of listener transcriptions of compensatory articulations. Cleft Palate Craniofac J. 2001; 38(1):59-67. DOI: 10.1597/1545-1569_2001_038_0059_roltoc_2.0.co_2. View