Large-scale Biomedical Concept Recognition: an Evaluation of Current Automatic Annotators and Their Parameters

Overview

Journal BMC Bioinformatics

Publisher Biomed Central

Specialty Biology

Date 2014 Feb 28

PMID 24571547

Citations 57

Authors

Christopher Funk

William Baumgartner Jr

Benjamin Garcia

Christophe Roeder

Michael Bada

K Bretonnel Cohen

Lawrence E Hunter

Karin Verspoor

Affiliations

Soon will be listed here.

Abstract

Background: Ontological concepts are useful for many different biomedical tasks. Concepts are difficult to recognize in text due to a disconnect between what is captured in an ontology and how the concepts are expressed in text. There are many recognizers for specific ontologies, but a general approach for concept recognition is an open problem.

Results: Three dictionary-based systems (MetaMap, NCBO Annotator, and ConceptMapper) are evaluated on eight biomedical ontologies in the Colorado Richly Annotated Full-Text (CRAFT) Corpus. Over 1,000 parameter combinations are examined, and best-performing parameters for each system-ontology pair are presented.

Conclusions: Baselines for concept recognition by three systems on eight biomedical ontologies are established (F-measures range from 0.14-0.83). Out of the three systems we tested, ConceptMapper is generally the best-performing system; it produces the highest F-measure of seven out of eight ontologies. Default parameters are not ideal for most systems on most ontologies; by changing parameters F-measure can be increased by up to 0.4. Not only are best performing parameters presented, but suggestions for choosing the best parameters based on ontology characteristics are presented.

Citing Articles

Integration of background knowledge for automatic detection of inconsistencies in gene ontology annotation.

Chen J, Goudey B, Geard N, Verspoor K Bioinformatics. 2024; 40(Suppl 1):i390-i400.

PMID: 38940182 PMC: 11256942. DOI: 10.1093/bioinformatics/btae246.

Causal feature selection using a knowledge graph combining structured knowledge from the biomedical literature and ontologies: A use case studying depression as a risk factor for Alzheimer's disease.

Malec S, Taneja S, Albert S, Shaaban C, Karim H, Levine A J Biomed Inform. 2023; 142:104368.

PMID: 37086959 PMC: 10355339. DOI: 10.1016/j.jbi.2023.104368.

Classifying literature mentions of biological pathogens as experimentally studied using natural language processing.

Jimeno Yepes A, Verspoor K J Biomed Semantics. 2023; 14(1):1.

PMID: 36721225 PMC: 9889128. DOI: 10.1186/s13326-023-00282-y.

Enhanced neurologic concept recognition using a named entity recognition model based on transformers.

Azizi S, Hier D, Wunsch Ii D Front Digit Health. 2022; 4:1065581.

PMID: 36569804 PMC: 9772022. DOI: 10.3389/fdgth.2022.1065581.

A Gated Recurrent Unit based architecture for recognizing ontology concepts from biological literature.

Devkota P, Mohanty S, Manda P BioData Min. 2022; 15(1):22.

PMID: 36171616 PMC: 9516808. DOI: 10.1186/s13040-022-00310-0.

References

Verspoor K, Cohn J, Joslyn C, Mniszewski S, Rechtsteiner A, Rocha L . Protein annotation as term categorization in the gene ontology using word proximity networks. BMC Bioinformatics. 2005; 6 Suppl 1:S20. PMC: 1869013. DOI: 10.1186/1471-2105-6-S1-S20. View

Eilbeck K, Lewis S, Mungall C, Yandell M, Stein L, Durbin R . The Sequence Ontology: a tool for the unification of genome annotations. Genome Biol. 2005; 6(5):R44. PMC: 1175956. DOI: 10.1186/gb-2005-6-5-r44. View

Rocktaschel T, Weidlich M, Leser U . ChemSpot: a hybrid system for chemical named entity recognition. Bioinformatics. 2012; 28(12):1633-40. DOI: 10.1093/bioinformatics/bts183. View

Cohen K, Palmer M, Hunter L . Nominalization and alternations in biomedical language. PLoS One. 2008; 3(9):e3158. PMC: 2527518. DOI: 10.1371/journal.pone.0003158. View

Denny J, Smithers J, Miller R, Spickard 3rd A . "Understanding" medical school curriculum content using KnowledgeMap. J Am Med Inform Assoc. 2003; 10(4):351-62. PMC: 181986. DOI: 10.1197/jamia.M1176. View

Sokolov A, Funk C, Graim K, Verspoor K, Ben-Hur A . Combining heterogeneous data sources for accurate functional annotation of proteins. BMC Bioinformatics. 2013; 14 Suppl 3:S10. PMC: 3584846. DOI: 10.1186/1471-2105-14-S3-S10. View

Denny J, Spickard 3rd A, Miller R, Schildcrout J, Darbar D, Rosenbloom S . Identifying UMLS concepts from ECG Impressions using KnowledgeMap. AMIA Annu Symp Proc. 2006; :196-200. PMC: 1479847. View

Ray S, Craven M . Learning statistical models for annotating proteins with function information using biomedical text. BMC Bioinformatics. 2005; 6 Suppl 1:S18. PMC: 1869010. DOI: 10.1186/1471-2105-6-S1-S18. View

Roeder C, Jonquet C, Shah N, Baumgartner Jr W, Verspoor K, Hunter L . A UIMA wrapper for the NCBO annotator. Bioinformatics. 2010; 26(14):1800-1. PMC: 2894505. DOI: 10.1093/bioinformatics/btq250. View

10.

Shah N, Bhatia N, Jonquet C, Rubin D, Chiang A, Musen M . Comparison of concept recognizers for building the Open Biomedical Annotator. BMC Bioinformatics. 2009; 10 Suppl 9:S14. PMC: 2745685. DOI: 10.1186/1471-2105-10-S9-S14. View

11.

Muller H, Kenny E, Sternberg P . Textpresso: an ontology-based information retrieval and extraction system for biological literature. PLoS Biol. 2004; 2(11):e309. PMC: 517822. DOI: 10.1371/journal.pbio.0020309. View

12.

Rebholz-Schuhmann D, Arregui M, Gaudan S, Kirsch H, Jimeno A . Text processing through Web services: calling Whatizit. Bioinformatics. 2007; 24(2):296-8. DOI: 10.1093/bioinformatics/btm557. View

13.

Yu H, Hatzivassiloglou V, Friedman C, Rzhetsky A, Wilbur W . Automatic extraction of gene and protein synonyms from MEDLINE and journal articles. Proc AMIA Symp. 2002; :919-23. PMC: 2244511. View

14.

Wheeler D, Barrett T, Benson D, Bryant S, Canese K, Chetvernin V . Database resources of the National Center for Biotechnology Information. Nucleic Acids Res. 2006; 35(Database issue):D5-12. PMC: 1781113. DOI: 10.1093/nar/gkl1031. View

15.

Zou Q, Chu W, Morioka C, Leazer G, Kangarloo H . IndexFinder: a method of extracting key concepts from clinical texts for indexing. AMIA Annu Symp Proc. 2004; :763-7. PMC: 1480259. View

16.

Settles B . ABNER: an open source tool for automatically tagging genes, proteins and other entity names in text. Bioinformatics. 2005; 21(14):3191-2. DOI: 10.1093/bioinformatics/bti475. View

17.

Aronson A . Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp. 2002; :17-21. PMC: 2243666. View

18.

Couto F, Silva M, Coutinho P . Finding genomic ontology terms in text using evidence content. BMC Bioinformatics. 2005; 6 Suppl 1:S21. PMC: 1869014. DOI: 10.1186/1471-2105-6-S1-S21. View

19.

Kang N, Singh B, Afzal Z, van Mulligen E, Kors J . Using rule-based natural language processing to improve disease normalization in biomedical text. J Am Med Inform Assoc. 2012; 20(5):876-81. PMC: 3756254. DOI: 10.1136/amiajnl-2012-001173. View

20.

Verspoor K, Cohn J, Mniszewski S, Joslyn C . A categorization approach to automated ontological function annotation. Protein Sci. 2006; 15(6):1544-9. PMC: 2242540. DOI: 10.1110/ps.062184006. View