Discordance, Accuracy and Reproducibility Study of Pathologists' Diagnosis of Melanoma and Melanocytic Tumors

Accurate melanoma diagnosis is crucial for patient outcomes and reliability of AI diagnostic tools. We assess interrater variability among eight expert pathologists reviewing histopathological images and clinical metadata of 792 melanoma-suspicious lesions prospectively collected at eight German hospitals. Moreover, we provide access to the largest panel-validated dataset featuring dermoscopic and histopathological images with metadata. Complete agreement is achieved in 53.5% of cases (424/792), and a majority vote ( ≥ five pathologists) in 90.9% (720/792). Considerable discordance is observed for non-invasive melanomas (complete agreement in only 10/73 cases). The expert panel disagrees with the local pathologists' and dermatologists' diagnoses in 14.9% and 33.5% of cases, respectively. This variability highlights the diagnostic challenges of early-stage melanomas and the need to reconsider how ground truth is established in routine care and AI research. Including at least two pathologists or virtual panels may contribute to more consistent diagnostic results.

References

Cherobin A, Wainstein A, Colosimo E, Goulart E, Bittencourt F . Prognostic factors for metastasis in cutaneous melanoma. An Bras Dermatol. 2018; 93(1):19-26. PMC: 5871357. DOI: 10.1590/abd1806-4841.20184779. View

Hosler G, Goldberg M, Estrada S, ONeil B, Amin S, Plaza J . Diagnostic discordance among histopathological reviewers of melanocytic lesions. J Cutan Pathol. 2024; 51(8):624-633. DOI: 10.1111/cup.14635. View

Mitteldorf C, Tronnier M . Dermatopathologie - aktueller Stand und Entwicklung in der deutschsprachigen Dermatologie. J Dtsch Dermatol Ges. 2023; 21(4):393-398. DOI: 10.1111/ddg.15047_g. View

Williams K, Jackson S, Beeken R, Steptoe A, Wardle J . The impact of a cancer diagnosis on health and well-being: a prospective, population-based study. Psychooncology. 2015; 25(6):626-32. PMC: 4915489. DOI: 10.1002/pon.3998. View

Brinker T, Hekler A, Enk A, Klode J, Hauschild A, Berking C . Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur J Cancer. 2019; 113:47-54. DOI: 10.1016/j.ejca.2019.04.001. View

Lodha S, Saggar S, Celebi J, Silvers D . Discordance in the histopathologic diagnosis of difficult melanocytic neoplasms in the clinical setting. J Cutan Pathol. 2008; 35(4):349-52. DOI: 10.1111/j.1600-0560.2007.00970.x. View

Hekler A, Utikal J, Enk A, Solass W, Schmitt M, Klode J . Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images. Eur J Cancer. 2019; 118:91-96. DOI: 10.1016/j.ejca.2019.06.012. View

Corona R, Mele A, Amini M, De Rosa G, Coppola G, Piccardi P . Interobserver variability on the histopathologic diagnosis of cutaneous melanoma and other pigmented skin lesions. J Clin Oncol. 1996; 14(4):1218-23. DOI: 10.1200/JCO.1996.14.4.1218. View

Veenhuizen K, de Wit P, Mooi W, Scheffer E, Verbeek A, Ruiter D . Quality assessment by expert opinion in melanoma pathology: experience of the pathology panel of the Dutch Melanoma Working Party. J Pathol. 1997; 182(3):266-72. DOI: 10.1002/(SICI)1096-9896(199707)182:3<266::AID-PATH812>3.0.CO;2-#. View

Bauer J, Leinweber B, Metzler G, Blum A, Hofmann-Wellenhof R, Leitz N . Correlation with digital dermoscopic images can help dermatopathologists to diagnose equivocal skin tumours. Br J Dermatol. 2006; 155(3):546-51. DOI: 10.1111/j.1365-2133.2006.07342.x. View

10.

Ferrara G, Annessi G, Argenyi Z, Argenziano G, Beltraminelli H, Cerio R . Prior knowledge of the clinical picture does not introduce bias in the histopathologic diagnosis of melanocytic skin lesions. J Cutan Pathol. 2015; 42(12):953-958. DOI: 10.1111/cup.12589. View

11.

Mitteldorf C, Tronnier M . Dermatopathology - Current status and development in German-speaking dermatology. J Dtsch Dermatol Ges. 2023; 21(4):393-397. DOI: 10.1111/ddg.15047. View

12.

Song H, Kim M, Park D, Shin Y, Lee J . Learning From Noisy Labels With Deep Neural Networks: A Survey. IEEE Trans Neural Netw Learn Syst. 2022; 34(11):8135-8153. DOI: 10.1109/TNNLS.2022.3152527. View

13.

Arnold M, Singh D, Laversanne M, Vignat J, Vaccarella S, Meheus F . Global Burden of Cutaneous Melanoma in 2020 and Projections to 2040. JAMA Dermatol. 2022; 158(5):495-503. PMC: 8968696. DOI: 10.1001/jamadermatol.2022.0160. View

14.

Cohen J, Korevaar D, Altman D, Bruns D, Gatsonis C, Hooft L . STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open. 2017; 6(11):e012799. PMC: 5128957. DOI: 10.1136/bmjopen-2016-012799. View

15.

Ferrara G, Argenyi Z, Argenziano G, Cerio R, Cerroni L, Di Blasi A . The influence of clinical information in the histopathologic diagnosis of melanocytic skin neoplasms. PLoS One. 2009; 4(4):e5375. PMC: 2671836. DOI: 10.1371/journal.pone.0005375. View

16.

Barnhill R, Argenyi Z, From L, Glass L, Maize J, Mihm Jr M . Atypical Spitz nevi/tumors: lack of consensus for diagnosis, discrimination from melanoma, and prediction of outcome. Hum Pathol. 1999; 30(5):513-20. DOI: 10.1016/s0046-8177(99)90193-4. View

17.

Brinker T, Schmitt M, Krieghoff-Henning E, Barnhill R, Beltraminelli H, Braun S . Diagnostic performance of artificial intelligence for histologic melanoma recognition compared to 18 international expert pathologists. J Am Acad Dermatol. 2021; 86(3):640-642. DOI: 10.1016/j.jaad.2021.02.009. View

18.

Esteva A, Kuprel B, Novoa R, Ko J, Swetter S, Blau H . Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542(7639):115-118. PMC: 8382232. DOI: 10.1038/nature21056. View

19.

Hekler A, Kather J, Krieghoff-Henning E, Utikal J, Meier F, Gellrich F . Effects of Label Noise on Deep Learning-Based Skin Cancer Classification. Front Med (Lausanne). 2020; 7:177. PMC: 7218064. DOI: 10.3389/fmed.2020.00177. View