How to Do an Evaluation: Pitfalls and Traps

Overview

Journal J Comput Aided Mol Des

Publisher Springer

Specialties Biomedical Engineering
Molecular Biology

Date 2008 Jan 25

PMID 18217218

Citations 42

Authors

Paul C D Hawkins

Gregory L Warren

A Geoffrey Skillman

Anthony Nicholls

Affiliations

Soon will be listed here.

Abstract

The recent literature is replete with papers evaluating computational tools (often those operating on 3D structures) for their performance in a certain set of tasks. Most commonly these papers compare a number of docking tools for their performance in cognate re-docking (pose prediction) and/or virtual screening. Related papers have been published on ligand-based tools: pose prediction by conformer generators and virtual screening using a variety of ligand-based approaches. The reliability of these comparisons is critically affected by a number of factors usually ignored by the authors, including bias in the datasets used in virtual screening, the metrics used to assess performance in virtual screening and pose prediction and errors in crystal structures used.

Citing Articles

Modeling High Energy Molecules and Screening to Find Novel High Energy Candidates.

Rachamim M, Domb A, Goldblum A ACS Omega. 2024; 9(42):42709-42720.

PMID: 39464471 PMC: 11500160. DOI: 10.1021/acsomega.4c01070.

Structure-based pose prediction: Non-cognate docking extended to macrocyclic ligands.

Cleves A, Tandon H, Jain A J Comput Aided Mol Des. 2024; 38(1):33.

PMID: 39414633 PMC: 11485047. DOI: 10.1007/s10822-024-00574-0.

Considerations Around Structure-Based Drug Discovery for KRAS Using DOCK.

Chakrabarti M, Tan Y, Balius T Methods Mol Biol. 2024; 2797:67-90.

PMID: 38570453 DOI: 10.1007/978-1-0716-3822-4_6.

Improvement of binding pose prediction of the MR1 covalent ligands by inclusion of simple pharmacophore constraints and structural waters in the docking process.

Shamsara J, Schuurmann G 3 Biotech. 2023; 13(8):279.

PMID: 37483466 PMC: 10356737. DOI: 10.1007/s13205-023-03694-w.

Confidence bands and hypothesis tests for hit enrichment curves.

Ash J, Hughes-Oliver J J Cheminform. 2022; 14(1):50.

PMID: 35902962 PMC: 9334420. DOI: 10.1186/s13321-022-00629-0.

References

Triballeau N, Acher F, Brabet I, Pin J, Bertrand H . Virtual screening workflow development guided by the "receiver operating characteristic" curve approach. Application to high-throughput docking on metabotropic glutamate receptor subtype 4. J Med Chem. 2005; 48(7):2534-47. DOI: 10.1021/jm049092j. View

Bostrom J, Greenwood J, Gottfries J . Assessing the performance of OMEGA with respect to retrieving bioactive conformations. J Mol Graph Model. 2003; 21(5):449-62. DOI: 10.1016/s1093-3263(02)00204-8. View

Sheridan R, Singh S, Fluder E, Kearsley S . Protocols for bridging the peptide to nonpeptide gap in topological similarity searches. J Chem Inf Comput Sci. 2001; 41(5):1395-406. DOI: 10.1021/ci0100144. View

Cruickshank D . Remarks about protein structure precision. Acta Crystallogr D Biol Crystallogr. 1999; 55(Pt 3):583-601. DOI: 10.1107/s0907444998012645. View

Verdonk M, Berdini V, Hartshorn M, Mooij W, Murray C, Taylor R . Virtual screening using protein-ligand docking: avoiding artificial enrichment. J Chem Inf Comput Sci. 2004; 44(3):793-806. DOI: 10.1021/ci034289q. View

Hartshorn M, Verdonk M, Chessari G, Brewerton S, Mooij W, Mortenson P . Diverse, high-quality test set for the validation of protein-ligand docking performance. J Med Chem. 2007; 50(4):726-41. DOI: 10.1021/jm061277y. View

Jain A . Surflex-Dock 2.1: robust performance from ligand energetic modeling, ring flexibility, and knowledge-based search. J Comput Aided Mol Des. 2007; 21(5):281-306. DOI: 10.1007/s10822-007-9114-2. View

Bostrom J . Reproducing the conformations of protein-bound ligands: a critical evaluation of several popular conformational searching tools. J Comput Aided Mol Des. 2002; 15(12):1137-52. DOI: 10.1023/a:1015930826903. View

Hanley J, McNeil B . The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143(1):29-36. DOI: 10.1148/radiology.143.1.7063747. View

10.

Leite T, Gomes D, Miteva M, Chomilier J, Villoutreix B, Tuffery P . Frog: a FRee Online druG 3D conformation generator. Nucleic Acids Res. 2007; 35(Web Server issue):W568-72. PMC: 1933180. DOI: 10.1093/nar/gkm289. View

11.

Huang N, Shoichet B, Irwin J . Benchmarking sets for molecular docking. J Med Chem. 2006; 49(23):6789-801. PMC: 3383317. DOI: 10.1021/jm0608356. View

12.

Warren G, Andrews C, Capelli A, Clarke B, LaLonde J, Lambert M . A critical assessment of docking programs and scoring functions. J Med Chem. 2006; 49(20):5912-31. DOI: 10.1021/jm050362n. View

13.

Kroemer R, Vulpetti A, McDonald J, Rohrer D, Trosset J, Giordanetto F . Assessment of docking poses: interactions-based accuracy classification (IBAC) versus crystal structure deviations. J Chem Inf Comput Sci. 2004; 44(3):871-81. DOI: 10.1021/ci049970m. View

14.

Erickson J, Jalaie M, Robertson D, Lewis R, Vieth M . Lessons in molecular recognition: the effects of ligand and protein flexibility on molecular docking accuracy. J Med Chem. 2003; 47(1):45-55. DOI: 10.1021/jm030209y. View

15.

Nissink J, Murray C, Hartshorn M, Verdonk M, Cole J, Taylor R . A new test set for validating predictions of protein-ligand interaction. Proteins. 2002; 49(4):457-71. DOI: 10.1002/prot.10232. View

16.

Hooft R, Vriend G, Sander C, Abola E . Errors in protein structures. Nature. 1996; 381(6580):272. DOI: 10.1038/381272a0. View

17.

Hawkins P, Skillman A, Nicholls A . Comparison of shape-matching and docking as virtual screening tools. J Med Chem. 2007; 50(1):74-82. DOI: 10.1021/jm0603365. View

18.

McGaughey G, Sheridan R, Bayly C, Culberson J, Kreatsoulas C, Lindsley S . Comparison of topological, shape, and docking methods in virtual screening. J Chem Inf Model. 2007; 47(4):1504-19. DOI: 10.1021/ci700052x. View

19.

Jones G, Willett P, Glen R, Leach A, Taylor R . Development and validation of a genetic algorithm for flexible docking. J Mol Biol. 1997; 267(3):727-48. DOI: 10.1006/jmbi.1996.0897. View

20.

Friesner R, Murphy R, Repasky M, Frye L, Greenwood J, Halgren T . Extra precision glide: docking and scoring incorporating a model of hydrophobic enclosure for protein-ligand complexes. J Med Chem. 2006; 49(21):6177-96. DOI: 10.1021/jm051256o. View