A Qualitative Assessment of Using ChatGPT As Large Language Model for Scientific Workflow Development

Overview

Journal Gigascience

Publisher Oxford University Press

Specialties Biology
Genetics

Date 2024 Jun 19

PMID 38896539

Authors

Mario Sanger

Ninon De Mecquenem

Katarzyna Ewa Lewinska

Vasilis Bountris

Fabian Lehmann

Ulf Leser

Thomas Kosch

Affiliations

Soon will be listed here.

Abstract

Background: Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages.

Results: To address these challenges, we investigate the efficiency of large language models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed 3 user studies in 2 scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.

Conclusions: Our results show a high accuracy for comprehending and explaining scientific workflows while achieving a reduced performance for modifying and extending workflow descriptions. These findings clearly illustrate the need for further research in this area.

Citing Articles

A qualitative assessment of using ChatGPT as large language model for scientific workflow development.

Sanger M, De Mecquenem N, Lewinska K, Bountris V, Lehmann F, Leser U Gigascience. 2024; 13.

PMID: 38896539 PMC: 11186067. DOI: 10.1093/gigascience/giae030.

References

Ison J, Ienasescu H, Chmura P, Rydza E, Menager H, Kalas M . The bio.tools registry of software tools and data resources for the life sciences. Genome Biol. 2019; 20(1):164. PMC: 6691543. DOI: 10.1186/s13059-019-1772-6. View

Wratten L, Wilm A, Goke J . Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nat Methods. 2021; 18(10):1161-1168. DOI: 10.1038/s41592-021-01254-9. View

Kim D, Paggi J, Park C, Bennett C, Salzberg S . Graph-based genome alignment and genotyping with HISAT2 and HISAT-genotype. Nat Biotechnol. 2019; 37(8):907-915. PMC: 7605509. DOI: 10.1038/s41587-019-0201-4. View

Chen S, Zhou Y, Chen Y, Gu J . fastp: an ultra-fast all-in-one FASTQ preprocessor. Bioinformatics. 2018; 34(17):i884-i890. PMC: 6129281. DOI: 10.1093/bioinformatics/bty560. View

Goecks J, Nekrutenko A, Taylor J . Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol. 2010; 11(8):R86. PMC: 2945788. DOI: 10.1186/gb-2010-11-8-r86. View

Oinn T, Addis M, Ferris J, Marvin D, Senger M, Greenwood M . Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics. 2004; 20(17):3045-54. DOI: 10.1093/bioinformatics/bth361. View

Langmead B, Salzberg S . Fast gapped-read alignment with Bowtie 2. Nat Methods. 2012; 9(4):357-9. PMC: 3322381. DOI: 10.1038/nmeth.1923. View

Dobin A, Davis C, Schlesinger F, Drenkow J, Zaleski C, Jha S . STAR: ultrafast universal RNA-seq aligner. Bioinformatics. 2012; 29(1):15-21. PMC: 3530905. DOI: 10.1093/bioinformatics/bts635. View

Sedgwick P, Greenwood N . Understanding the Hawthorne effect. BMJ. 2015; 351:h4672. DOI: 10.1136/bmj.h4672. View

10.

van Dijk E, Auger H, Jaszczyszyn Y, Thermes C . Ten years of next-generation sequencing technology. Trends Genet. 2014; 30(9):418-26. DOI: 10.1016/j.tig.2014.07.001. View

11.

Almarie B, Teixeira P, Pacheco-Barrios K, Rossetti C, Fregni F . Editorial - The Use of Large Language Models in Science: Opportunities and Challenges. Princ Pract Clin Res. 2023; 9(1):1-4. PMC: 10485814. DOI: 10.21801/ppcrj.2023.91.1. View

12.

Hu T, Chitnis N, Monos D, Dinh A . Next-generation sequencing technologies: An overview. Hum Immunol. 2021; 82(11):801-811. DOI: 10.1016/j.humimm.2021.02.012. View

13.

Sanger M, De Mecquenem N, Lewinska K, Bountris V, Lehmann F, Leser U . A qualitative assessment of using ChatGPT as large language model for scientific workflow development. Gigascience. 2024; 13. PMC: 11186067. DOI: 10.1093/gigascience/giae030. View

14.

Langmead B, Trapnell C, Pop M, Salzberg S . Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol. 2009; 10(3):R25. PMC: 2690996. DOI: 10.1186/gb-2009-10-3-r25. View

15.

Trapnell C, Williams B, Pertea G, Mortazavi A, Kwan G, van Baren M . Transcript assembly and quantification by RNA-Seq reveals unannotated transcripts and isoform switching during cell differentiation. Nat Biotechnol. 2010; 28(5):511-5. PMC: 3146043. DOI: 10.1038/nbt.1621. View

16.

Bolger A, Lohse M, Usadel B . Trimmomatic: a flexible trimmer for Illumina sequence data. Bioinformatics. 2014; 30(15):2114-20. PMC: 4103590. DOI: 10.1093/bioinformatics/btu170. View

17.

Sudmanns M, Tiede D, Lang S, Bergstedt H, Trost G, Augustin H . Big Earth data: disruptive changes in Earth observation data management and analysis?. Int J Digit Earth. 2020; 13(7):832-850. PMC: 7455055. DOI: 10.1080/17538947.2019.1585976. View

18.

Koster J, Rahmann S . Snakemake--a scalable bioinformatics workflow engine. Bioinformatics. 2012; 28(19):2520-2. DOI: 10.1093/bioinformatics/bts480. View

19.

Ewels P, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A . The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020; 38(3):276-278. DOI: 10.1038/s41587-020-0439-x. View

20.

Sanvicente-Garcia M, Garcia-Valiente A, Jouide S, Jaraba-Wallace J, Bautista E, Escobosa M . CRISPR-Analytics (CRISPR-A): A platform for precise analytics and simulations for gene editing. PLoS Comput Biol. 2023; 19(5):e1011137. PMC: 10256225. DOI: 10.1371/journal.pcbi.1011137. View