The Psychological Reality of the Learned "p < .05" Boundary

Overview

Journal Cogn Res Princ Implic

Specialty Psychology

Date 2024 May 3

PMID 38700660

Authors

V N Vimal Rao

Jeffrey K Bye

Sashank Varma

Affiliations

Soon will be listed here.

Abstract

The .05 boundary within Null Hypothesis Statistical Testing (NHST) "has made a lot of people very angry and been widely regarded as a bad move" (to quote Douglas Adams). Here, we move past meta-scientific arguments and ask an empirical question: What is the psychological standing of the .05 boundary for statistical significance? We find that graduate students in the psychological sciences show a boundary effect when relating p-values across .05. We propose this psychological boundary is learned through statistical training in NHST and reading a scientific literature replete with "statistical significance". Consistent with this proposal, undergraduates do not show the same sensitivity to the .05 boundary. Additionally, the size of a graduate student's boundary effect is not associated with their explicit endorsement of questionable research practices. These findings suggest that training creates distortions in initial processing of p-values, but these might be dampened through scientific processes operating over longer timescales.

References

Kazak A . Editorial: Journal article reporting standards. Am Psychol. 2018; 73(1):1-2. DOI: 10.1037/amp0000263. View

Dehaene S . The neural basis of the Weber-Fechner law: a logarithmic mental number line. Trends Cogn Sci. 2003; 7(4):145-147. DOI: 10.1016/s1364-6613(03)00055-x. View

Ioannidis J . Why most published research findings are false. PLoS Med. 2005; 2(8):e124. PMC: 1182327. DOI: 10.1371/journal.pmed.0020124. View

Ioannidis J, Trikalinos T . An exploratory test for an excess of significant findings. Clin Trials. 2007; 4(3):245-53. DOI: 10.1177/1740774507079441. View

Simonsohn U, Nelson L, Simmons J . P-curve: a key to the file-drawer. J Exp Psychol Gen. 2013; 143(2):534-47. DOI: 10.1037/a0033242. View

Ashcraft M . Cognitive arithmetic: a review of data and theory. Cognition. 1992; 44(1-2):75-106. DOI: 10.1016/0010-0277(92)90051-i. View

Bishop D . The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. Q J Exp Psychol (Hove). 2019; 73(1):1-19. PMC: 6909195. DOI: 10.1177/1747021819886519. View

Schneider R, Sullivan J, Marusic F, Zaucer R, Biswas P, Mismas P . Do children use language structure to discover the recursive rules of counting?. Cogn Psychol. 2020; 117:101263. DOI: 10.1016/j.cogpsych.2019.101263. View

Rao V, Bye J, Varma S . Categorical Perception of p-Values. Top Cogn Sci. 2021; 14(2):414-425. DOI: 10.1111/tops.12589. View

10.

Cumming G . The new statistics: why and how. Psychol Sci. 2013; 25(1):7-29. DOI: 10.1177/0956797613504966. View

11.

Coulson M, Healey M, Fidler F, Cumming G . Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Front Psychol. 2011; 1:26. PMC: 3095378. DOI: 10.3389/fpsyg.2010.00026. View

12.

Benjamin D, Berger J, Johannesson M, Nosek B, Wagenmakers E, Berk R . Redefine statistical significance. Nat Hum Behav. 2019; 2(1):6-10. DOI: 10.1038/s41562-017-0189-z. View

13.

Ciccione L, Dehaene S . Can humans perform mental regression on a graph? Accuracy and bias in the perception of scatterplots. Cogn Psychol. 2021; 128:101406. DOI: 10.1016/j.cogpsych.2021.101406. View

14.

Barth H, Paladino A . The development of numerical estimation: evidence against a representational shift. Dev Sci. 2010; 14(1):125-35. DOI: 10.1111/j.1467-7687.2010.00962.x. View

15.

Poitevineau J, Lecoutre B . Interpretation of significance levels by psychological researchers: the .05 cliff effect may be overstated. Psychon Bull Rev. 2002; 8(4):847-50. DOI: 10.3758/bf03196227. View

16.

John L, Loewenstein G, Prelec D . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci. 2012; 23(5):524-32. DOI: 10.1177/0956797611430953. View

17.

Varma S, Karl S . Understanding decimal proportions: discrete representations, parallel access, and privileged processing of zero. Cogn Psychol. 2013; 66(3):283-301. DOI: 10.1016/j.cogpsych.2013.01.002. View

18.

Helske J, Helske S, Cooper M, Ynnerman A, Besancon L . Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect. IEEE Trans Vis Comput Graph. 2021; 27(8):3397-3409. DOI: 10.1109/TVCG.2021.3073466. View

19.

Kruschke J, Liddell T . The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychon Bull Rev. 2017; 25(1):178-206. DOI: 10.3758/s13423-016-1221-4. View

20.

Moyer R, Landauer T . Time required for judgements of numerical inequality. Nature. 1967; 215(5109):1519-20. DOI: 10.1038/2151519a0. View