Evaluation of the Departmental Inter-rater Reliability when Scoring Thyroid Nodules According to the British Thyroid Association Ultrasound-classification Model: Is There Significant Disagreement?

Overview

Journal Ultrasound

Date 2024 May 2

PMID 38694831

Authors

Nabil Rtam

Affiliations

Soon will be listed here.

Abstract

Introduction: The British Thyroid Association Ultrasound-classification is a risk stratification model which grades thyroid nodules in U2-5 based on their sonographic appearance. Existence of variability between the ultrasound operators when U-scoring is reported in the literature with some evidence found in the author's department. The aim of this study was to investigate whether there is significant disagreement in the department and identify potential reasons for variability.

Methods: Eight operators, radiologists and sonographers, were recruited to grade 33 TNs and answer a tick box questionnaire using the British Thyroid Association lexicon. The inter-operator variability for the U-categories, indication for fine-needle aspiration biopsy and ultrasound features was assessed using Fleiss' kappa and Gwet-AC1. The operators' accuracy was measured against the most experienced operator in the department using Cohen's kappa and percentage agreement.

Results: Fair agreement (Fleiss' = 0.21) was obtained between the participants when U-scoring (U2-5). Fair-to-moderate agreement was noted between sonographers ( = 0.40). Significant variability was demonstrated between radiologists ( > 0.05). Indication for fine-needle aspiration biopsy reached fair to almost substantial agreement (radiologists' AC1 = 0.34, sonographers' AC1 = 0.58, overall AC1 = 0.41). No significant variability measured for echogenicity ( = 0.29), composition ( = 0.33), shape ( = 0.58), margin ( = 0.45), halo ( = 0.34) and vascularity ( = 0.44). Accuracy reached fair agreement (mean Cohen's = 0.29) and moderate agreement (mean AC1 = 0.53) for the U-categories and fine-needle aspiration biopsy, respectively. Radiologists demonstrated lower accuracy.

Conclusion: No significant inter-rater variability in U-scoring or recommending fine-needle aspiration biopsy was demonstrated between all the operators in the department. Radiologists showed significant variability in U-scoring and lower accuracy. Reliability and accuracy could be improved by addressing those problematic categories and features identified with this study.

References

Koltin D, OGorman C, Murphy A, Ngan B, Daneman A, Navarro O . Pediatric thyroid nodules: ultrasonographic characteristics and inter-observer variability in prediction of malignancy. J Pediatr Endocrinol Metab. 2016; 29(7):789-94. DOI: 10.1515/jpem-2015-0242. View

Wongpakaran N, Wongpakaran T, Wedding D, Gwet K . A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013; 13:61. PMC: 3643869. DOI: 10.1186/1471-2288-13-61. View

Kwak J, Han K, Yoon J, Moon H, Son E, Park S . Thyroid imaging reporting and data system for US features of nodules: a step in establishing better stratification of cancer risk. Radiology. 2011; 260(3):892-9. DOI: 10.1148/radiol.11110206. View

Lim-Dunham J, Toslak I, Alsabban K, Aziz A, Martin B, Okur G . Ultrasound risk stratification for malignancy using the 2015 American Thyroid Association Management Guidelines for Children with Thyroid Nodules and Differentiated Thyroid Cancer. Pediatr Radiol. 2017; 47(4):429-436. DOI: 10.1007/s00247-017-3780-6. View

Couzins M, Forbes S, Vigneswaran G, Mitra I, Rutherford E . Ultrasound grading of thyroid nodules using the BTA U-scoring guidelines - Is there evidence of intra-and interobserver variability?. Ultrasound. 2021; 29(2):100-105. PMC: 8083133. DOI: 10.1177/1742271X20971323. View

Haugen B, Alexander E, Bible K, Doherty G, Mandel S, Nikiforov Y . 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid. 2015; 26(1):1-133. PMC: 4739132. DOI: 10.1089/thy.2015.0020. View

Bunting K, Steeds R, Slater K, Rogers J, Gkoutos G, Kotecha D . A Practical Guide to Assess the Reproducibility of Echocardiographic Measurements. J Am Soc Echocardiogr. 2019; 32(12):1505-1515. DOI: 10.1016/j.echo.2019.08.015. View

Landis J, Koch G . The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1):159-74. View

Tessler F, Middleton W, Grant E, Hoang J, Berland L, Teefey S . ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee. J Am Coll Radiol. 2017; 14(5):587-595. DOI: 10.1016/j.jacr.2017.01.046. View

10.

Youserm D, Huang T, Loevner L, Langlotz C . Clinical and economic impact of incidental thyroid lesions found with CT and MR. AJNR Am J Neuroradiol. 1997; 18(8):1423-8. PMC: 8338147. View

11.

Cicchetti D, Feinstein A . High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990; 43(6):551-8. DOI: 10.1016/0895-4356(90)90159-m. View

12.

Weller A, Sharif B, Qarib M, St Leger D, De Silva H, Lingam R . British Thyroid Association 2014 classification ultrasound scoring of thyroid nodules in predicting malignancy: Diagnostic performance and inter-observer agreement. Ultrasound. 2020; 28(1):4-13. PMC: 6987506. DOI: 10.1177/1742271X19865001. View

13.

McHugh M . Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012; 22(3):276-82. PMC: 3900052. View

14.

Seifert P, Gorges R, Zimny M, Kreissl M, Schenke S . Interobserver agreement and efficacy of consensus reading in Kwak-, EU-, and ACR-thyroid imaging recording and data systems and ATA guidelines for the ultrasound risk stratification of thyroid nodules. Endocrine. 2019; 67(1):143-154. DOI: 10.1007/s12020-019-02134-1. View

15.

Kim E, Park C, Chung W, Oh K, Kim D, Lee J . New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol. 2002; 178(3):687-91. DOI: 10.2214/ajr.178.3.1780687. View

16.

Ahmed S, Horton K, Jeffrey Jr R, Sheth S, Fishman E . Incidental thyroid nodules on chest CT: Review of the literature and management suggestions. AJR Am J Roentgenol. 2010; 195(5):1066-71. DOI: 10.2214/AJR.10.4506. View

17.

Hoang J, Middleton W, Farjat A, Teefey S, Abinanti N, Boschini F . Interobserver Variability of Sonographic Features Used in the American College of Radiology Thyroid Imaging Reporting and Data System. AJR Am J Roentgenol. 2018; 211(1):162-167. DOI: 10.2214/AJR.17.19192. View

18.

Russ G, Bonnema S, Erdogan M, Durante C, Ngu R, Leenhardt L . European Thyroid Association Guidelines for Ultrasound Malignancy Risk Stratification of Thyroid Nodules in Adults: The EU-TIRADS. Eur Thyroid J. 2017; 6(5):225-237. PMC: 5652895. DOI: 10.1159/000478927. View

19.

Kim H, Kwak J, Kim E, Choi S, Moon H . Man to man training: can it help improve the diagnostic performances and interobserver variabilities of thyroid ultrasonography in residents?. Eur J Radiol. 2011; 81(3):e352-6. DOI: 10.1016/j.ejrad.2011.11.011. View