Reproducibility of Computational Workflows is Automated Using Continuous Analysis

Overview

Journal Nat Biotechnol

Specialty Biotechnology

Date 2017 Mar 14

PMID 28288103

Citations 58

Authors

Brett K Beaulieu-Jones

Casey S Greene

Affiliations

Soon will be listed here.

Abstract

Replication, validation and extension of experiments are crucial for scientific progress. Computational experiments are scriptable and should be easy to reproduce. However, computational analyses are designed and run in a specific computing environment, which may be difficult or impossible to match using written instructions. We report the development of continuous analysis, a workflow that enables reproducible computational analyses. Continuous analysis combines Docker, a container technology akin to virtual machines, with continuous integration, a software development technique, to automatically rerun a computational analysis whenever updates or improvements are made to source code or data. This enables researchers to reproduce results without contacting the study authors. Continuous analysis allows reviewers, editors or readers to verify reproducibility without manually downloading and rerunning code and can provide an audit trail for analyses of data that cannot be shared.

Citing Articles

Multistage deep learning methods for automating radiographic sharp score prediction in rheumatoid arthritis.

Moradmand H, Ren L Sci Rep. 2025; 15(1):3391.

PMID: 39870749 PMC: 11772782. DOI: 10.1038/s41598-025-86073-0.

Ten challenges and opportunities in computational immuno-oncology.

Bao R, Hutson A, Madabhushi A, Jonsson V, Rosario S, Barnholtz-Sloan J J Immunother Cancer. 2024; 12(10).

PMID: 39461879 PMC: 11529678. DOI: 10.1136/jitc-2024-009721.

Systematic data analysis pipeline for quantitative morphological cell phenotyping.

Ghanegolmohammadi F, Eslami M, Ohya Y Comput Struct Biotechnol J. 2024; 23:2949-2962.

PMID: 39104709 PMC: 11298594. DOI: 10.1016/j.csbj.2024.07.012.

Systematic evaluation with practical guidelines for single-cell and spatially resolved transcriptomics data simulation under multiple scenarios.

Duo H, Li Y, Lan Y, Tao J, Yang Q, Xiao Y Genome Biol. 2024; 25(1):145.

PMID: 38831386 PMC: 11149245. DOI: 10.1186/s13059-024-03290-y.

DExplore: An Online Tool for Detecting Differentially Expressed Genes from mRNA Microarray Experiments.

Katsiki A, Karatzas P, De Lastic H, Georgakilas A, Tsitsilonis O, Vorgias C Biology (Basel). 2024; 13(5).

PMID: 38785833 PMC: 11117493. DOI: 10.3390/biology13050351.

References

Bray N, Pimentel H, Melsted P, Pachter L . Near-optimal probabilistic RNA-seq quantification. Nat Biotechnol. 2016; 34(5):525-7. DOI: 10.1038/nbt.3519. View

McNutt M . Reproducibility. Science. 2014; 343(6168):229. DOI: 10.1126/science.1250475. View

Stodden V, McNutt M, Bailey D, Deelman E, Gil Y, Hanson B . Enhancing reproducibility for computational methods. Science. 2016; 354(6317):1240-1241. DOI: 10.1126/science.aah6168. View

Ioannidis J, Allison D, Ball C, Coulibaly I, Cui X, Culhane A . Repeatability of published microarray gene expression analyses. Nat Genet. 2009; 41(2):149-55. DOI: 10.1038/ng.295. View

Groves T, Godlee F . Open science and reproducible research. BMJ. 2012; 344:e4383. DOI: 10.1136/bmj.e4383. View

Peng R . Reproducible research in computational science. Science. 2011; 334(6060):1226-7. PMC: 3383002. DOI: 10.1126/science.1213847. View

Katoh K, Misawa K, Kuma K, Miyata T . MAFFT: a novel method for rapid multiple sequence alignment based on fast Fourier transform. Nucleic Acids Res. 2002; 30(14):3059-66. PMC: 135756. DOI: 10.1093/nar/gkf436. View

Baker M . 1,500 scientists lift the lid on reproducibility. Nature. 2016; 533(7604):452-4. DOI: 10.1038/533452a. View

Rice P, Longden I, Bleasby A . EMBOSS: the European Molecular Biology Open Software Suite. Trends Genet. 2000; 16(6):276-7. DOI: 10.1016/s0168-9525(00)02024-2. View

10.

Smyth G . Linear models and empirical bayes methods for assessing differential expression in microarray experiments. Stat Appl Genet Mol Biol. 2006; 3:Article3. DOI: 10.2202/1544-6115.1027. View

11.

. Illuminating the black box. Nature. 2006; 442(7098):1. DOI: 10.1038/442001a. View

12.

Ritchie M, Phipson B, Wu D, Hu Y, Law C, Shi W . limma powers differential expression analyses for RNA-sequencing and microarray studies. Nucleic Acids Res. 2015; 43(7):e47. PMC: 4402510. DOI: 10.1093/nar/gkv007. View

13.

. Rebooting review. Nat Biotechnol. 2015; 33(4):319. DOI: 10.1038/nbt.3202. View

14.

Garijo D, Kinnings S, Xie L, Xie L, Zhang Y, Bourne P . Quantifying reproducibility in computational biology: the case of the tuberculosis drugome. PLoS One. 2013; 8(11):e80278. PMC: 3842296. DOI: 10.1371/journal.pone.0080278. View

15.

Nunez M, Sanchez-Jimenez C, Alcalde J, Izquierdo J . Long-term reduction of T-cell intracellular antigens reveals a transcriptome associated with extracellular matrix and cell adhesion components. PLoS One. 2014; 9(11):e113141. PMC: 4236147. DOI: 10.1371/journal.pone.0113141. View

16.

. Software with impact. Nat Methods. 2014; 11(3):211. DOI: 10.1038/nmeth.2880. View

17.

Souilmi Y, Lancaster A, Jung J, Rizzo E, Hawkins J, Powles R . Scalable and cost-effective NGS genotyping in the cloud. BMC Med Genomics. 2015; 8:64. PMC: 4608296. DOI: 10.1186/s12920-015-0134-9. View

18.

Beaulieu-Jones B, Greene C . Semi-supervised learning of the electronic health record for phenotype stratification. J Biomed Inform. 2016; 64:168-178. DOI: 10.1016/j.jbi.2016.10.007. View

19.

Hothorn T, Leisch F . Case studies in reproducibility. Brief Bioinform. 2011; 12(3):288-300. DOI: 10.1093/bib/bbq084. View

20.

Dai M, Wang P, Boyd A, Kostov G, Athey B, Jones E . Evolving gene/transcript definitions significantly alter the interpretation of GeneChip data. Nucleic Acids Res. 2005; 33(20):e175. PMC: 1283542. DOI: 10.1093/nar/gni179. View