The Americleft Speech Project: A Training and Reliability Study
2016, The Cleft Palate-Craniofacial Journal
https://doi.org/10.1597/14-027Abstract
Objective To describe the results of two reliability studies and to assess the effect of training on interrater reliability scores. Design The first study (1) examined interrater and intrarater reliability scores (weighted and unweighted kappas) and (2) compared interrater reliability scores before and after training on the use of the Cleft Audit Protocol for Speech–Augmented (CAPS-A) with British English-speaking children. The second study examined interrater and intrarater reliability on a modified version of the CAPS-A (CAPS-A Americleft Modification) with American and Canadian English-speaking children. Finally, comparisons were made between the interrater and intrarater reliability scores obtained for Study 1 and Study 2. Participants The participants were speech-language pathologists from the Americleft Speech Project. Results In Study 1, interrater reliability scores improved for 6 of the 13 parameters following training on the CAPS-A protocol. Comparison of the reliability r...
References (104)
- Altman, DG. Practical Statistics for Medical Research. London: Chapman and Hall/CRC Press; 1991. p. 404
- Anthony, A., Bogle, D., Ingram, TTS., McIsaac, MW. The Edinburgh Articulation Test. Edinburgh: E & S Livingstone; 1971.
- Arlt PB, Goodban MT. A comparative study of articulation acquisition as based on a study of 240 normals, aged three to six. Lang Speech Hear Serv Sch. 1976; 7:173-180.
- Ball MJ, Rahilly J. Transcribing disordered speech: the segmental and prosodic layers. Clin Linguist Phon. 2002; 16:329-344. [PubMed: 12185981]
- Bassich CJ, Ludlow CL. The use of perceptual methods by new clinicians for assessing voice quality. J Speech Hear Disord. 1986; 51:125-133. [PubMed: 3702360]
- Baylis AL, Munson B, Moller KT. Perceptions of audible nasal emission in speakers with cleft palate: a comparative study of listener judgments. Cleft Palate Craniofac J. 2011; 48:399-411. [PubMed: 20572776]
- Baylis AL, Whitehill TL, Chapman KL. Americleft Speech Group. Validity and reliability of visual analog scaling for assessment of speech outcomes in children with repaired cleft palate. Cleft Palate Craniofac J. 2015; 52:660-670. [PubMed: 25322442]
- Bradford LJ, Brooks AR, Shelton RL. Clinical judgment of hypernasality in cleft palate children. Cleft Palate J. 1964; 6:329-335.
- Britton L, Albery E, Bowden M, Harding-Bell A, Phippen G, Sell D. A cross-sectional cohort study of speech in 5-year-olds with cleft palate ± lip to support development of national audit standards. Cleft Palate Craniofac J. 2014; 51:431-451. [PubMed: 24635034]
- Brunnegård K, Lohmander A. A cross-sectional study of speech in 10-year-old children with cleft palate: results and issues of rater reliability. Cleft Palate Craniofac J. 2007; 44:33-44. [PubMed: 17214536]
- Brunnegård K, Lohmander A, van Doorn J. Untrained listeners' ratings of speech disorders in a group with cleft palate: a comparison with speech and language pathologists' ratings. Int J Lang Commun Disord. 2009; 44:656-674. [PubMed: 18821109]
- Bunton K, Kent RD, Duffy JR, Rosenbek JC, Kent JF. Listener agreement for auditory-perceptual ratings of dysarthria. J Speech Lang Hear Res. 2007; 50:1481-1495. [PubMed: 18055769]
- Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol. 1993; 46:423-429. [PubMed: 8501467]
- Chan KMK, Yiu EML. A comparison of two perceptual voice evaluation training programs for naive listeners. J Voice. 2006; 20:229-241. [PubMed: 16139475]
- Chan KMK, Yiu EML. The effect of anchors and training on the reliability of perceptual voice evaluation. J Speech Lang Hear Res. 2002; 45:111-126. [PubMed: 14748643]
- Chirlian NS, Sharpley CF. Children's articulation development: some regional differences. Aust J Hum Commun Disord. 1982; 10:23-30.
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20:37-46.
- Cordes AK. Individual and consensus judgments of dysfluency types in the speech of persons who stutter. J Speech Lang Hear Res. 2000; 43:951-964. [PubMed: 11386481]
- Cordes AK. The reliability of observational data: I. Theories and methods for speech-language pathology. J Speech Lang Hear Res. 1994; 37:264-278.
- Cordes AK, Ingham RJ. Effects of time-interval judgment training on real-time measurement of stuttering. J Speech Lang Hear Res. 1999; 42:862-879. [PubMed: 10450907]
- Cordes AK, Ingham RJ. The reliability of observational data: II. Issues in the identification and measurement of stuttering events. J Speech Lang Hear Res. 1994; 37:279-294.
- Counihan DT, Cullinan DL. Reliability and dispersion of nasality ratings. Cleft Palate J. 1970; 7:261- 270. [PubMed: 5266336]
- Cucchiarini C. Assessing transcription agreement: methodological aspects. Clin Linguist Phon. 1996; 10:131-155.
- Dalston RM, Marsh JL, Vig KW, Witzel MA, Bumsted RM. Minimal standards for reporting the results of surgery on patients with cleft lip, cleft palate, or both: a proposal. Cleft Palate J. 1988; 25:3-7. [PubMed: 3422597]
- Daniel HJ. Nasality ratings of single words, phrases, and running speech samples obtained from cleft palate children. Folia Phoniatr Logop. 1971; 23:41-49.
- de Vet HCW, Terwee CB, Know DL, Bouter LM. When to use agreement versus reliability measures. J Clin Epidemiol. 2006; 59:1033-1039. [PubMed: 16980142]
- Dodd B, Holm A, Hua Z, Crosbie S. Phonological development: a normative study of British English- speaking children. Clin Linguist Phon. 2003; 17:617-643. [PubMed: 14977026]
- Dotevall H, Lohmander-Agerskov A, Ejnell H, Bake B. Perceptual evaluation of speech and velopharyngeal function in children with and without cleft palate and the relationship to nasal airflow patterns. Cleft Palate Craniofac J. 2002; 39:409-424. [PubMed: 12071789]
- Eadie TL, Baylor CR. The effect of perceptual training on inexperienced listeners' judgments of dysphonic voice. J Voice. 2006; 20:527-544. [PubMed: 16324823]
- Ellis LW, Beltyukova S. Effects of training on naïve listeners' judgments of the speech intelligibility of children with severe-to-profound hearing loss. J Speech Lang Hear Res. 2008; 51:1114-1123. [PubMed: 18664708]
- Bronsted K, Grunwell P, Henningsson G, Jansonius K, Karling J, Meijer M, Ording D, Sell D, Vermei- Zieverink E, Wyatt R. Eurocleft Speech Group. A phonetic framework for the cross-linguistic analysis of cleft palate speech. Clin Linguist Phon. 1994; 8:109-125.
- Grunwell P, Bronsted K, Henningsson G, Jansonius K, Karling J, Meijer M, Ording D, Wyatt R, Sell D, Vermei-Zieverink E. Eurocleft Speech Group. A six-centre international study of the outcome of treatment in patients with clefts of the lip and palate: the results of a cross-linguistic investigation of cleft palate speech. Scand J Plast Reconstr Surg Hand Surg. 2000; 34:219-229. [PubMed: 11020918]
- Fleiss, JL., Levin, B., Pai, MC. Statistical Methods for Rates and Proportions. 3. Hoboken, NJ: John Wiley & Sons; 2003. p. 609
- Fujiwara Y, Henningsson G, Ainoda N. A review of Japanese articles on perceptual assessment of speech sounds in individuals with cleft palate. Jpn J Logoped Phoniatr. 2006; 47:252-257.
- Gerratt BR, Kreiman J, Antonanzas-Barroso N, Berke GS. Comparing internal and external standards in voice quality judgments. J Speech Lang Hear Res. 1993; 36:14-20.
- Gibbon FE, Crampin L. An electropalatographic investigation of middorsum palatal stops in an adult with repaired cleft palate. Cleft Palate Craniofac J. 2001; 38:96-105. [PubMed: 11294548]
- Gooch JL, Hardin-Jones M, Chapman KL, Trost-Cardamone JE, Sussman J. Reliability of listener transcriptions of compensatory articulations. Cleft Palate Craniofac J. 2001; 38:59-67. [PubMed: 11204684]
- Grunwell, P., Sell, D., Harding, A. Describing cleft palate speech. In: Grunwell, P., editor. Analysing Cleft Palate Speech. London: Whurr; 1993.
- Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012; 8:23-34. [PubMed: 22833776]
- Helou LB, Solomon NP, Henry LR, Coppit GL, Howard RS, Stojadinovic A. The role of listener experience on consensus auditory-perceptual evaluation of voice (CAPE-V): ratings of postthyroidectomy voice. Am J Speech Lang Pathol. 2010; 19:248-258. [PubMed: 20484704]
- Henningsson G, Kuehn D, Sell D, Sweeney T, Trost-Cardamone J, Whitehill T. Universal parameters for reporting speech outcomes in individuals with cleft palate. Cleft Palate Craniofac J. 2008; 45:1-17. [PubMed: 18215095]
- Hutters B, Henningsson G. Speech outcome following treatment in cross-linguistic cleft palate studies: methodological implications. Cleft Palate Craniofac J. 2004; 41:544-549. [PubMed: 15352862]
- Imatomi S. Effects of breathy voice source on ratings of hypernasality. Cleft Palate Craniofac J. 2005; 42:641-648. [PubMed: 16241176]
- John A, Sell D, Sweeney T, Harding-Bell A, Williams A. The cleft audit protocol for speech- augmented: a validated and reliable measure for auditing cleft speech. Cleft Palate Craniofac J. 2006; 43:272-288. [PubMed: 16681400]
- Kent RD. Hearing and believing: some limits to the auditory-perceptual assessment of speech and voice disorders. Am J Speech Lang Pathol. 1996; 5:7-23.
- Keuning KHD, Wieneke GH, Dejonckere PH. The intrajudge reliability of the perceptual rating of cleft palate speech before and after pharyngeal flap surgery: the effect of judges and speech samples. Cleft Palate Craniofac J. 1999; 36:328-333. [PubMed: 10426599]
- Kilminster M, Laird E. Articulation development in children aged three to nine years. Aust J Hum Commun Disord. 1978; 6:23-30.
- Klinto K, Slameh E, Svensson H, Lohmander A. The impact of speech material on speech judgment in children with and without cleft palate. Int J Lang Commun Disord. 2011; 46:348-360. [PubMed: 21575075]
- Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, Roberts C, Shoukri M, Steiner DL. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Int J Nurs Stud. 2011; 48:661-671. [PubMed: 21514934]
- Kreiman J, Gerratt BR. Perceptual assessment of voice quality: past, present and future. SIG3 Perspect Voice Voice Disord. 2010; 22:62-67.
- Kreiman J, Gerratt BR, Kempster GB, Erman A, Berke GS. Perceptual evaluation of voice quality: review, tutorial, and a framework for future research. J Speech Lang Hear Res. 1993; 36:21-40.
- Kreiman J, Gerratt BR, Precoda K. Listener experience and perception of voice quality. J Speech Lang Hear Res. 1990; 33:103-115.
- Kreiman J, Gerratt BR, Precoda K, Berke GS. Individual differences in voice quality perception. J Speech Hear Res. 1992; 35:512-520. [PubMed: 1608242]
- Kuehn DP, Moller KT. Speech and language issues in the cleft palate population: the state of the art. Cleft Palate Craniofac J. 2000; 37:348-348. [Accessed March 7, 2012] Available at http:// dx.doi.org/10.1597/1545-1569(2000)037%3C0348:SALIIT%3E2.3.CO;2.
- Lee A, Whitehill T, Ciocca V. Effect of listener training on perceptual judgment of hypernasality. Clin Linguist Phon. 2009; 23:319-334. [PubMed: 19399664]
- Lewis KE. Reporting observer agreement on stuttering event judgments: a survey and evaluation of current practice. J Fluency Disord. 1994; 19:269-284.
- Lewis KE, Watterson TL, Houghton SM. The influence of listener experience and academic training on ratings of nasality. J Commun Disord. 2003; 36:49-58. [PubMed: 12493637]
- Lockhart R, McLeod S. Factors that enhance English-speaking speech-language pathologists' transcription of Cantonese-speaking children's consonants. Am J Speech Lang Pathol. 2013; 22:523-539. [PubMed: 23813201]
- Lohmander, A. Surgical interventions and speech outcomes in cleft lip and palate. In: Howard, S., Lohmander, A., editors. Cleft Palate Speech: Assessment and Intervention. West Sussex: Wiley- Blackwell; 2011. p. 55-85.
- Lohmander, A., Borell, E., Henningsson, G., Havstram, C., Lundeborg, I., Persson, C. SVANTE- Svenskt Artikulations-och Nasalitets Test [manual]. Malmo: Pedogogisk Design; 2005.
- Lohmander A, Friede H, Lilja J. Long-term, longitudinal follow-up of individuals with unilateral cleft lip and palate after the Gothenburg primary early veloplasty and delayed hard palate closure protocol: speech outcomes. Cleft Palate Craniofac J. 2012; 49:657-671. [PubMed: 22364610]
- Lohmander A, Olsson M. Methodology for perceptual assessment of speech in patients with cleft palate: a critical review of the literature. Cleft Palate Craniofac J. 2004; 41:64-70. [PubMed: 14697067]
- Lohmander A, Willadsen E, Persson C, Henningsson M, Bowden B, Hutters B. Methodology for speech assessment in the Scandcleft project-an international randomized clinical trial on palatal surgery: experiences from a pilot study. Cleft Palate Craniofac J. 2009; 46:347-362. [PubMed: 19642772]
- Long RE, Hathaway R, Daskalogiannakis J, Mercado A, Russell K, Cohen M, Semb G, Shaw W. The Americleft study: an inter-center study of treatment outcomes for patients with unilateral cleft lip and palate, part 1. Principles and study design. Cleft Palate Craniofac J. 2011; 48:239-243. [PubMed: 21219224]
- McNutt JC, Wicki L, Paulsen J. Judgments of phoneme errors under four modes of audio-visual presentation. J Speech Lang Pathol Audiol. 1991; 15:37-42.
- McWilliams BJ. Some factors in the intelligibility of cleft-palate speech. J Speech Hear Disord. 1954; 19:524-528. [PubMed: 13222468]
- McWilliams, BJ., Morris, HL., Shelton, RL. Cleft Palate Speech. Philadelphia: BC Decker; 1990.
- McWilliams, BJ., Philips, BJ. Velopharyngeal Incompetence. Philadelphia: BC Decker; 1990.
- McWilliams, BJ., Philips, BJ. Velopharyngeal Incompetence: Audio Seminars in Speech Pathology. Philadelphia: WB Saunders; 1979.
- Munson B, Johnson JM, Edwards J. The role of experience in the perception of phonetic detail in children's speech: a comparison between speech-language pathologists and clinically untrained listeners. Am J Speech Lang Pathol. 2012; 21:124-139. [PubMed: 22230182]
- Oates J. Auditory-perceptual evaluation of disordered voice quality. Folia Phoniatr Logop. 2009; 61:49-56. [PubMed: 19204393]
- Pereira V, Tuomainen J, Sell D. Perceptual outcomes of maxillary osteotomy in patients with cleft lip and palate. Int J Lang Commun Disord. 2013; 48:640-650. [PubMed: 24165361]
- Peterson-Falzone SJ. The relationship between timing of cleft palate surgery and speech outcome: what have we learned, and where do we stand in the 1990s? Semin Orthod. 1996; 2:185-191. [PubMed: 9161287]
- Prather EM, Hedrick DL, Kern CA. Articulation development in children aged two to four years. J Speech Hear Disord. 1975; 40:179-191. [PubMed: 1234929]
- Santelmann H, Sussman J, Chapman K. Perception of middorsum palatal stops from the speech of three children with repaired cleft palate. Cleft Palate Craniofac J. 1999; 36:233-242. [PubMed: 10342611]
- Schellinger, SK., Edwards, J., Munson, B., Beckman, MK. The role of listener expectations on judgments of children's /s/ productions. Presented at the Symposium on Research in Child Language Disorders; June 2008; Madison, Wisconsin.
- Sell D. Issues in perceptual speech analysis in cleft palate and related disorders: a review. Int J Lang Commun Disord. 2005; 40:103-121. [PubMed: 16101269]
- Sell D, Grunwell P, Mildinhall S, Murphy T, Cornish TC, Williams A, Bearn D, Shaw WC, Murray J, Sandy J. Cleft lip and palate care in the United Kingdom-the Clinical Standards Advisory Group (CSAG) study. Part 3: speech outcomes. Cleft Palate Craniofac J. 2001; 38:30-37. [PubMed: 11204679]
- Sell D, Harding A, Grunwell P. Revised GOS.SP.ASS (98): speech assessment for children with cleft palate and/or velopharyngeal dysfunction. Int J Lang Commun Disord. 1999; 34:7-33.
- Sell D, Harding A, Grunwell P. A screening assessment of cleft palate speech (Great Ormond Street Speech Assessment). Eur J Disord Commun. 1994; 29:1-15. [PubMed: 8032102]
- Sell, D., Harding-Bell, A., Sweeney, T., Freeman, J., John, A. CAPS-A Speech Audit Tool Training Programmes-are they necessary?. Presented at the Meeting of the Craniofacial Society of Great Britain and Ireland; April 2010; Liverpool, United Kingdom.
- Sell D, John A, Harding-Bell A, Sweeney T, Hegarty F, Freeman J. Cleft Audit Protocol for Speech (CAPS-A): a comprehensive training package for speech analysis. Int J Lang Commun Disord. 2009; 44:529-548. [PubMed: 18821108]
- Shriberg LD, Lof GL. Reliability studies in broad and narrow phonetic transcription. Clin Linguist Phon. 1991; 5:225-279.
- Shriberg LD, McSweeny JL, Anderson BE, Campbell TF, Chial MR, Green JR, Hauner KK, Moore CA, Rusiewicz HL, Wilson DL. Transitioning from analog to digital audio recording in childhood speech sound disorders. Clin Linguist Phon. 2005; 19:335-359. [PubMed: 16019779]
- Shrivastav R, Sapienza CM, Nandur V. Application of psychometric theory to the measurement of voice quality using rating scales. J Speech Lang Hear Res. 2005; 48:323-335. [PubMed: 15989395]
- Smit AB. Phonologic error distributions in the Iowa-Nebraska Articulation Norms. J Speech Lang Hear Res. 1993; 36:533-537.
- Smit AB, Hand L, Freilinger JJ, Bernthal JE, Bird A. The Iowa articulation norms project and its Nebraska replication. J Speech Hear Disord. 1990; 55:779-798. [PubMed: 2232757]
- Spriestersbach DC, Powers GR. Nasality in isolated vowels and connected speech of cleft palate speakers. J Speech Hear Res. 1959; 2:40-45. [PubMed: 13655290]
- Stevens I, Daniloff RG. Trouble with /s/: a methodological study of factors affecting the judgment of misarticulated /s/. J Commun Disord. 1977; 10:207-220. [PubMed: 903406]
- Stoddard, GJ. Biostatistics and Epidemiology Using Stata: A Course Manual. Salt Lake City, UT: University of Utah School of Medicine; Available at http://www.ccts.utah.edu/biostats/? pageId=5385 [Accessed February 20, 2012]
- Streiner, DL., Norman, GR. Health Measurement Scales: A Practical Guide to Their Development and Use. New York: Oxford University Press; 1995. p. 118
- Sussman, J., Chapman, KL. Acoustic analysis of mid-dorsum, alveolar, and velar stop consonants. Presented at the American Speech-Language-Hearing Association Convention; November 2000; Washington, DC.
- Sweeney, T. Nasality-assessment and intervention. In: Howard, S., Lohmander, A., editors. Cleft Palate Speech Assessment and Intervention. Oxford: Wiley and Blackwell; 2011. p. 199-216.
- Templin, MC. Certain Language Skills in Children [monograph]. Minneapolis: University of Minnesota Press; 1957.
- Tönz M, Schmid I, Graf M, Mischler-Heeb R, Weissen J, Kaiser G. Blinded speech evaluation following pharyngeal flap surgery by speech pathologists and lay people in children with cleft palate. Folia Phoniatr Logop. 2002; 54:228-295.
- Trost-Cardamone, JE. American English Sentence Sample: a controlled sample for assessing cleft palate speech outcome. Presented at the Meeting of the American Cleft Palate-Craniofacial Association; April 2012; San Jose, California.
- Van Hattum RJ. Articulation and nasality in cleft palate speakers. J Speech Hear Disord. 1958; 1:383- 387. [PubMed: 13599159]
- Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005; 37:360-363. [PubMed: 15883903]
- Watterson T, Lewis K, Allord M, Sulprizio S, O'Neill P. Effect of vowel type on reliability of nasality ratings. J Commun Disord. 2007; 40:503-512. [PubMed: 17391692]
- Whitehill TL, Lee ASY, Chun JC. Direct magnitude estimation and interval scaling of hypernasality. J Speech Lang Hear Res. 2002; 45:80-88. [PubMed: 14748640]
- Wolfe V, Martin D, Borton D, Youngblood HC. The effect of clinical experience on cue trading for the /r-w/ contrast. Am J Speech Lang Pathol. 2003; 12:221-228. [PubMed: 12828535]
- Yeung, AC. Dissertation. Pokfulam, Hong Kong: University of Hong Kong; 2010. Effects of Listening Conditions on Perceptual Ratings of Hypernasal Speech.
- Young, MA. Identification of stuttering and stutterers. In: Curlee, RF., Perkins, WH., editors. Nature and Treatment of Stuttering: New Directions. San Diego: College-Hill; 1984. p. 13-30.
- Zraick RI, Liss JM. A comparison of equal-appearing interval scaling and direct magnitude estimation of nasal voice quality. J Speech Lang Hear Res. 2000; 43:979-988. [PubMed: 11386483]