A Good Check on the Bayes Factor

Overview

Journal Behav Res Methods

Publisher Springer

Specialty Social Sciences

Date 2024 Sep 4

PMID 39231912

Authors

Nikola Sekulovski

Maarten Marsman

Eric-Jan Wagenmakers

Affiliations

Soon will be listed here.

Abstract

Bayes factor hypothesis testing provides a powerful framework for assessing the evidence in favor of competing hypotheses. To obtain Bayes factors, statisticians often require advanced, non-standard tools, making it important to confirm that the methodology is computationally sound. This paper seeks to validate Bayes factor calculations by applying two theorems attributed to Alan Turing and Jack Good. The procedure entails simulating data sets under two hypotheses, calculating Bayes factors, and assessing whether their expected values align with theoretical expectations. We illustrate this method with an ANOVA example and a network psychometrics application, demonstrating its efficacy in detecting calculation errors and confirming the computational correctness of the Bayes factor results. This structured validation approach aims to provide researchers with a tool to enhance the credibility of Bayes factor hypothesis testing, fostering more robust and trustworthy scientific inferences.

References

Wagenmakers E, Lodewyckx T, Kuriyal H, Grasman R . Bayesian hypothesis testing for psychologists: a tutorial on the Savage-Dickey method. Cogn Psychol. 2010; 60(3):158-89. DOI: 10.1016/j.cogpsych.2009.12.001. View

Heck D, Boehm U, Boing-Messing F, Burkner P, Derks K, Dienes Z . A review of applications of the Bayes factor in psychological research. Psychol Methods. 2022; 28(3):558-579. DOI: 10.1037/met0000454. View

Gronau Q, Heathcote A, Matzke D . Computing Bayes factors for evidence-accumulation models using Warp-III bridge sampling. Behav Res Methods. 2019; 52(2):918-937. PMC: 7148283. DOI: 10.3758/s13428-019-01290-6. View

Sanborn A, Hills T . The frequentist implications of optional stopping on Bayesian hypothesis tests. Psychon Bull Rev. 2013; 21(2):283-300. DOI: 10.3758/s13423-013-0518-9. View

Sekulovski N, Keetelaar S, Huth K, Wagenmakers E, van Bork R, den Bergh D . Testing Conditional Independence in Psychometric Networks: An Analysis of Three Bayesian Methods. Multivariate Behav Res. 2024; 59(5):913-933. DOI: 10.1080/00273171.2024.2345915. View

Schad D, Nicenboim B, Burkner P, Betancourt M, Vasishth S . Workflow techniques for the robust use of bayes factors. Psychol Methods. 2022; 28(6):1404-1426. DOI: 10.1037/met0000472. View

Marsman M, Rhemtulla M . Guest Editors' Introduction to The Special Issue "Network Psychometrics in Action": Methodological Innovations Inspired by Empirical Problems. Psychometrika. 2022; 87(1):1-11. PMC: 9021145. DOI: 10.1007/s11336-022-09861-x. View

Gronau Q, Sarafoglou A, Matzke D, Ly A, Boehm U, Marsman M . A tutorial on bridge sampling. J Math Psychol. 2017; 81:80-97. PMC: 5699790. DOI: 10.1016/j.jmp.2017.09.005. View

Marsman M, Maris G, Bechger T, Glas C . Bayesian inference for low-rank Ising networks. Sci Rep. 2015; 5:9050. PMC: 4356966. DOI: 10.1038/srep09050. View

10.

van de Schoot R, Winter S, Ryan O, Zondervan-Zwijnenburg M, Depaoli S . A systematic review of Bayesian articles in psychology: The last 25 years. Psychol Methods. 2017; 22(2):217-239. DOI: 10.1037/met0000100. View

11.

van Doorn J, den Bergh D, Bohm U, Dablander F, Derks K, Draws T . The JASP guidelines for conducting and reporting a Bayesian analysis. Psychon Bull Rev. 2020; 28(3):813-826. PMC: 8219590. DOI: 10.3758/s13423-020-01798-5. View

12.

Lodewyckx T, Tuerlinckx F, Kuppens P, Allen N, Sheeber L . A hierarchical state space approach to affective dynamics. J Math Psychol. 2011; 55(1):68-83. PMC: 3079909. DOI: 10.1016/j.jmp.2010.08.004. View

13.

Williams D . Bayesian Estimation for Gaussian Graphical Models: Structure Learning, Predictability, and Network Comparisons. Multivariate Behav Res. 2021; 56(2):336-352. DOI: 10.1080/00273171.2021.1894412. View

14.

Marsman M, Huth K, Waldorp L, Ntzoufras I . Objective Bayesian Edge Screening and Structure Selection for Ising Networks. Psychometrika. 2022; 87(1):47-82. PMC: 9021150. DOI: 10.1007/s11336-022-09848-8. View

15.

Borsboom D, Cramer A . Network analysis: an integrative approach to the structure of psychopathology. Annu Rev Clin Psychol. 2013; 9:91-121. DOI: 10.1146/annurev-clinpsy-050212-185608. View

16.

Wagenmakers E, Marsman M, Jamil T, Ly A, Verhagen J, Love J . Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychon Bull Rev. 2017; 25(1):35-57. PMC: 5862936. DOI: 10.3758/s13423-017-1343-3. View

17.

Vanpaemel W, Lee M . Using priors to formalize theory: optimal attention and the generalized context model. Psychon Bull Rev. 2012; 19(6):1047-56. DOI: 10.3758/s13423-012-0300-4. View

18.

Wagenmakers E, Love J, Marsman M, Jamil T, Ly A, Verhagen J . Bayesian inference for psychology. Part II: Example applications with JASP. Psychon Bull Rev. 2017; 25(1):58-76. PMC: 5862926. DOI: 10.3758/s13423-017-1323-7. View

19.

Hoijtink H, Mulder J, van Lissa C, Gu X . A tutorial on testing hypotheses using the Bayes factor. Psychol Methods. 2019; 24(5):539-556. DOI: 10.1037/met0000201. View

20.

Benjamin D, Berger J, Johannesson M, Nosek B, Wagenmakers E, Berk R . Redefine statistical significance. Nat Hum Behav. 2019; 2(1):6-10. DOI: 10.1038/s41562-017-0189-z. View