Biomedical Named Entity Recognition Based on Multi-cross Attention Feature Fusion

Overview

Journal PLoS One

Specialties General Medicine
Science

Date 2024 May 28

PMID 38805478

Authors

Dequan Zheng

Rong Han

Feng Yu

Yannan Li

Affiliations

Soon will be listed here.

Abstract

Currently, in the field of biomedical named entity recognition, CharCNN (Character-level Convolutional Neural Networks) or CharRNN (Character-level Recurrent Neural Network) is typically used independently to extract character features. However, this approach does not consider the complementary capabilities between them and only concatenates word features, ignoring the feature information during the process of word integration. Based on this, this paper proposes a method of multi-cross attention feature fusion. First, DistilBioBERT and CharCNN and CharLSTM are used to perform cross-attention word-char (word features and character features) fusion separately. Then, the two feature vectors obtained from cross-attention fusion are fused again through cross-attention to obtain the final feature vector. Subsequently, a BiLSTM is introduced with a multi-head attention mechanism to enhance the model's ability to focus on key information features and further improve model performance. Finally, the output layer is used to output the final result. Experimental results show that the proposed model achieves the best F1 values of 90.76%, 89.79%, 94.98%, 80.27% and 88.84% on NCBI-Disease, BC5CDR-Disease, BC5CDR-Chem, JNLPBA and BC2GM biomedical datasets respectively. This indicates that our model can capture richer semantic features and improve the ability to recognize entities.

References

Li J, Sun Y, Johnson R, Sciaky D, Wei C, Leaman R . BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database (Oxford). 2016; 2016. PMC: 4860626. DOI: 10.1093/database/baw068. View

Sayers E, Cavanaugh M, Clark K, Ostell J, Pruitt K, Karsch-Mizrachi I . GenBank. Nucleic Acids Res. 2019; 48(D1):D84-D86. PMC: 7145611. DOI: 10.1093/nar/gkz956. View

Zhang J, Shen D, Zhou G, Su J, Tan C . Enhancing HMM-based biomedical named entity recognition by studying special phenomena. J Biomed Inform. 2004; 37(6):411-22. DOI: 10.1016/j.jbi.2004.08.005. View

Islamaj Dogan R, Leaman R, Lu Z . NCBI disease corpus: a resource for disease name recognition and concept normalization. J Biomed Inform. 2014; 47:1-10. PMC: 3951655. DOI: 10.1016/j.jbi.2013.12.006. View

Lee J, Yoon W, Kim S, Kim D, Kim S, So C . BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2019; 36(4):1234-1240. PMC: 7703786. DOI: 10.1093/bioinformatics/btz682. View

Boeckmann B, Bairoch A, Apweiler R, Blatter M, Estreicher A, Gasteiger E . The SWISS-PROT protein knowledgebase and its supplement TrEMBL in 2003. Nucleic Acids Res. 2003; 31(1):365-70. PMC: 165542. DOI: 10.1093/nar/gkg095. View

Zhang Z, Chen A . Biomedical named entity recognition with the combined feature attention and fully-shared multi-task learning. BMC Bioinformatics. 2022; 23(1):458. PMC: 9632084. DOI: 10.1186/s12859-022-04994-3. View

Caporaso J, Baumgartner Jr W, Randolph D, Cohen K, Hunter L . MutationFinder: a high-performance system for extracting point mutation mentions from text. Bioinformatics. 2007; 23(14):1862-5. PMC: 2516306. DOI: 10.1093/bioinformatics/btm235. View

Weber L, Sanger M, Munchmeyer J, Habibi M, Leser U, Akbik A . HunFlair: an easy-to-use tool for state-of-the-art biomedical named entity recognition. Bioinformatics. 2021; 37(17):2792-2794. PMC: 8428609. DOI: 10.1093/bioinformatics/btab042. View

10.

Klinger R, Kolarik C, Fluck J, Hofmann-Apitius M, Friedrich C . Detection of IUPAC and IUPAC-like chemical names. Bioinformatics. 2008; 24(13):i268-76. PMC: 2718657. DOI: 10.1093/bioinformatics/btn181. View

11.

Saha S, Sarkar S, Mitra P . Feature selection techniques for maximum entropy based biomedical named entity recognition. J Biomed Inform. 2009; 42(5):905-11. DOI: 10.1016/j.jbi.2008.12.012. View

12.

Smith L, Tanabe L, Ando R, Kuo C, Chung I, Hsu C . Overview of BioCreative II gene mention recognition. Genome Biol. 2008; 9 Suppl 2:S2. PMC: 2559986. DOI: 10.1186/gb-2008-9-s2-s2. View

13.

Yang Z, Lin H, Li Y . Exploiting the performance of dictionary-based bio-entity name recognition in biomedical literature. Comput Biol Chem. 2008; 32(4):287-91. DOI: 10.1016/j.compbiolchem.2008.03.008. View

14.

Cho M, Ha J, Park C, Park S . Combinatorial feature embedding based on CNN and LSTM for biomedical named entity recognition. J Biomed Inform. 2020; 103:103381. DOI: 10.1016/j.jbi.2020.103381. View

15.

Lee K, Hwang Y, Kim S, Rim H . Biomedical named entity recognition using two-phase model based on SVMs. J Biomed Inform. 2004; 37(6):436-47. DOI: 10.1016/j.jbi.2004.08.012. View

16.

Chai Z, Jin H, Shi S, Zhan S, Zhuo L, Yang Y . Hierarchical shared transfer learning for biomedical named entity recognition. BMC Bioinformatics. 2022; 23(1):8. PMC: 8729142. DOI: 10.1186/s12859-021-04551-4. View

17.

Guan Z, Zhou X . A prefix and attention map discrimination fusion guided attention for biomedical named entity recognition. BMC Bioinformatics. 2023; 24(1):42. PMC: 9907889. DOI: 10.1186/s12859-023-05172-9. View

18.

Rohanian O, Nouriborji M, Kouchaki S, Clifton D . On the effectiveness of compact biomedical transformers. Bioinformatics. 2023; 39(3). PMC: 10027428. DOI: 10.1093/bioinformatics/btad103. View