Bias Analysis on Public X-Ray Image Datasets of Pneumonia and COVID-19 Patients

Overview

Journal IEEE Access

Date 2021 Nov 23

PMID 34812384

Citations 5

Authors

Omar Del Tejo Catala

Ismael Salvador Igual

Francisco Javier Perez-Benito

David Millan Escriva

Vicent Ortiz Castello

Rafael Llobet

Juan-Carlos Perez-Cortes

Affiliations

Soon will be listed here.

Abstract

Chest X-ray images are useful for early COVID-19 diagnosis with the advantage that X-ray devices are already available in health centers and images are obtained immediately. Some datasets containing X-ray images with cases (pneumonia or COVID-19) and controls have been made available to develop machine-learning-based methods to aid in diagnosing the disease. However, these datasets are mainly composed of different sources coming from pre-COVID-19 datasets and COVID-19 datasets. Particularly, we have detected a significant bias in some of the released datasets used to train and test diagnostic systems, which might imply that the results published are optimistic and may overestimate the actual predictive capacity of the techniques proposed. In this article, we analyze the existing bias in some commonly used datasets and propose a series of preliminary steps to carry out before the classic machine learning pipeline in order to detect possible biases, to avoid them if possible and to report results that are more representative of the actual predictive power of the methods under analysis.

Citing Articles

Challenges issues and future recommendations of deep learning techniques for SARS-CoV-2 detection utilising X-ray and CT images: a comprehensive review.

Islam M, Al Farid F, Shamrat F, Islam M, Rashid M, Bari B PeerJ Comput Sci. 2025; 10:e2517.

PMID: 39896401 PMC: 11784792. DOI: 10.7717/peerj-cs.2517.

Deep Learning for Pneumonia Detection in Chest X-ray Images: A Comprehensive Survey.

Siddiqi R, Javaid S J Imaging. 2024; 10(8).

PMID: 39194965 PMC: 11355845. DOI: 10.3390/jimaging10080176.

Digital Determinants of Health: Health data poverty amplifies existing health disparities-A scoping review.

Paik K, Hicklen R, Kaggwa F, Puyat C, Nakayama L, Ong B PLOS Digit Health. 2023; 2(10):e0000313.

PMID: 37824445 PMC: 10569513. DOI: 10.1371/journal.pdig.0000313.

Validating Automatic Concept-Based Explanations for AI-Based Digital Histopathology.

Sauter D, Lodde G, Nensa F, Schadendorf D, Livingstone E, Kukuk M Sensors (Basel). 2022; 22(14).

PMID: 35891026 PMC: 9319808. DOI: 10.3390/s22145346.

Explainable artificial intelligence-based edge fuzzy images for COVID-19 detection and identification.

Hu Q, Gois F, Costa R, Zhang L, Yin L, Magaia N Appl Soft Comput. 2022; 123:108966.

PMID: 35582662 PMC: 9102011. DOI: 10.1016/j.asoc.2022.108966.

References

Saez C, Zurriaga O, Perez-Panades J, Melchor I, Robles M, Garcia-Gomez J . Applying probabilistic temporal and multisite data quality control methods to a public health mortality registry in Spain: a systematic approach to quality control of repositories. J Am Med Inform Assoc. 2016; 23(6):1085-1095. PMC: 11741068. DOI: 10.1093/jamia/ocw010. View

Dansana D, Kumar R, Bhattacharjee A, Hemanth D, Gupta D, Khanna A . Early diagnosis of COVID-19-affected patients based on X-ray and computed tomography images using deep learning algorithm. Soft comput. 2020; 27(5):2635-2643. PMC: 7453871. DOI: 10.1007/s00500-020-05275-y. View

Ismael A, Sengur A . Deep learning approaches for COVID-19 detection based on chest X-ray images. Expert Syst Appl. 2020; 164:114054. PMC: 7521412. DOI: 10.1016/j.eswa.2020.114054. View

Oh Y, Park S, Ye J . Deep Learning COVID-19 Features on CXR Using Limited Training Data Sets. IEEE Trans Med Imaging. 2020; 39(8):2688-2700. DOI: 10.1109/TMI.2020.2993291. View

Perez-Benito F, Saez C, Conejero J, Tortajada S, Valdivieso B, Garcia-Gomez J . Temporal variability analysis reveals biases in electronic health records due to hospital process reengineering interventions over seven years. PLoS One. 2019; 14(8):e0220369. PMC: 6685618. DOI: 10.1371/journal.pone.0220369. View

Perez-Benito F, Conejero J, Saez C, Garcia-Gomez J, Navarro-Pardo E, Florencio L . Subgrouping Factors Influencing Migraine Intensity in Women: A Semi-automatic Methodology Based on Machine Learning and Information Geometry. Pain Pract. 2019; 20(3):297-309. DOI: 10.1111/papr.12854. View

Litjens G, Kooi T, Bejnordi B, Setio A, Ciompi F, Ghafoorian M . A survey on deep learning in medical image analysis. Med Image Anal. 2017; 42:60-88. DOI: 10.1016/j.media.2017.07.005. View

Waheed A, Goyal M, Gupta D, Khanna A, Al-Turjman F, Pinheiro P . CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection. IEEE Access. 2021; 8:91916-91923. PMC: 8043420. DOI: 10.1109/ACCESS.2020.2994762. View

Tartaglione E, Barbano C, Berzovini C, Calandri M, Grangetto M . Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data. Int J Environ Res Public Health. 2020; 17(18). PMC: 7557723. DOI: 10.3390/ijerph17186933. View

10.

Polsinelli M, Cinque L, Placidi G . A light CNN for detecting COVID-19 from CT scans of the chest. Pattern Recognit Lett. 2020; 140:95-100. PMC: 7532353. DOI: 10.1016/j.patrec.2020.10.001. View

11.

Minaee S, Kafieh R, Sonka M, Yazdani S, Jamalipour Soufi G . Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning. Med Image Anal. 2020; 65:101794. PMC: 7372265. DOI: 10.1016/j.media.2020.101794. View

12.

Rubin G, Ryerson C, Haramati L, Sverzellati N, Kanne J, Raoof S . The Role of Chest Imaging in Patient Management During the COVID-19 Pandemic: A Multinational Consensus Statement From the Fleischner Society. Chest. 2020; 158(1):106-116. PMC: 7138384. DOI: 10.1016/j.chest.2020.04.003. View

13.

Zebin T, Rezvy S . COVID-19 detection and disease progression visualization: Deep learning on chest X-rays for classification and coarse localization. Appl Intell (Dordr). 2021; 51(2):1010-1021. PMC: 7486976. DOI: 10.1007/s10489-020-01867-1. View

14.

Apostolopoulos I, Mpesiana T . Covid-19: automatic detection from X-ray images utilizing transfer learning with convolutional neural networks. Phys Eng Sci Med. 2020; 43(2):635-640. PMC: 7118364. DOI: 10.1007/s13246-020-00865-4. View

15.

Ismael A, Sengur A . The investigation of multiresolution approaches for chest X-ray image based COVID-19 detection. Health Inf Sci Syst. 2020; 8(1):29. PMC: 7522455. DOI: 10.1007/s13755-020-00116-6. View

16.

Saez C, Robles M, Garcia-Gomez J . Stability metrics for multi-source biomedical data based on simplicial projections from probability distribution distances. Stat Methods Med Res. 2014; 26(1):312-336. DOI: 10.1177/0962280214545122. View

17.

Perez-Benito F, Signol F, Perez-Cortes J, Fuster-Baggetto A, Pollan M, Perez-Gomez B . A deep learning system to obtain the optimal parameters for a threshold-based breast and dense tissue segmentation. Comput Methods Programs Biomed. 2020; 195:105668. DOI: 10.1016/j.cmpb.2020.105668. View

18.

Rahimzadeh M, Attar A . A modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based on the concatenation of Xception and ResNet50V2. Inform Med Unlocked. 2020; 19:100360. PMC: 7255267. DOI: 10.1016/j.imu.2020.100360. View

19.

Candemir S, Antani S . A review on lung boundary detection in chest X-rays. Int J Comput Assist Radiol Surg. 2019; 14(4):563-576. PMC: 6420899. DOI: 10.1007/s11548-019-01917-1. View

20.

Bustos A, Pertusa A, Salinas J, Iglesia-Vaya M . PadChest: A large chest x-ray image dataset with multi-label annotated reports. Med Image Anal. 2020; 66:101797. DOI: 10.1016/j.media.2020.101797. View