Using Parameter Space Partitioning to Evaluate a Model's Qualitative Fit

Overview

Journal Psychon Bull Rev

Specialty Psychology

Date 2016 Aug 27

PMID 27562764

Authors

Sara Steegen

Francis Tuerlinckx

Wolf Vanpaemel

Affiliations

Soon will be listed here.

Abstract

Parameter space partitioning (PSP) is a versatile tool for model analysis that detects the qualitatively distinctive data patterns a model can generate, and partitions a model's parameter space into regions corresponding to these patterns. In this paper, we propose a PSP fit measure that summarizes the outcome of a PSP analysis into a single number, which can be used for model selection. In contrast to traditional model selection methods, PSP-based model selection focuses on qualitative data. We demonstrate PSP-based model selection by use of application examples in the area of category learning. A large-scale model recovery study reveals excellent recovery properties, suggesting that PSP fit is useful for model selection.

References

Kruschke J . ALCOVE: an exemplar-based connectionist model of category learning. Psychol Rev. 1992; 99(1):22-44. DOI: 10.1037/0033-295x.99.1.22. View

Myung . The Importance of Complexity in Model Selection. J Math Psychol. 2000; 44(1):190-204. DOI: 10.1006/jmps.1999.1283. View

Vrieze S . Model selection and psychological theory: a discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychol Methods. 2012; 17(2):228-43. PMC: 3366160. DOI: 10.1037/a0027127. View

Pitt M, Myung I, Zhang S . Toward a method of selecting among computational models of cognition. Psychol Rev. 2002; 109(3):472-91. DOI: 10.1037/0033-295x.109.3.472. View

McClelland J, Elman J . The TRACE model of speech perception. Cogn Psychol. 1986; 18(1):1-86. DOI: 10.1016/0010-0285(86)90015-0. View

Vanpaemel W, Lee M . The Bayesian evaluation of categorization models: comment on Wills and Pothos (2012). Psychol Bull. 2012; 138(6):1253-8. DOI: 10.1037/a0028551. View

Lee M, Vanpaemel W . Determining informative priors for cognitive models. Psychon Bull Rev. 2017; 25(1):114-127. DOI: 10.3758/s13423-017-1238-3. View

Hawkins G, Brown S, Steyvers M, Wagenmakers E . An optimal adjustment procedure to minimize experiment time in decisions with multiple alternatives. Psychon Bull Rev. 2012; 19(2):339-48. DOI: 10.3758/s13423-012-0216-z. View

Nosofsky R . Attention and learning processes in the identification and categorization of integral stimuli. J Exp Psychol Learn Mem Cogn. 1987; 13(1):87-108. DOI: 10.1037//0278-7393.13.1.87. View

10.

Wills A, Pothos E . On the adequacy of current empirical evaluations of formal models of categorization. Psychol Bull. 2011; 138(1):102-25. DOI: 10.1037/a0025715. View

11.

Roberts S, Pashler H . How persuasive is a good fit? A comment on theory testing. Psychol Rev. 2000; 107(2):358-67. DOI: 10.1037/0033-295x.107.2.358. View

12.

Norris D, McQueen J, Cutler A . Merging information in speech recognition: feedback is never necessary. Behav Brain Sci. 2001; 23(3):299-325; discussion 325-70. DOI: 10.1017/s0140525x00003241. View

13.

Simmons J, Nelson L, Simonsohn U . False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011; 22(11):1359-66. DOI: 10.1177/0956797611417632. View

14.

Vanpaemel W, Storms G . Abstraction and model evaluation in category learning. Behav Res Methods. 2010; 42(2):421-37. DOI: 10.3758/BRM.42.2.421. View

15.

Pitt M, Kim W, Navarro D, Myung J . Global model analysis by parameter space partitioning. Psychol Rev. 2006; 113(1):57-83. DOI: 10.1037/0033-295X.113.1.57. View

16.

Nosofsky R . Attention, similarity, and the identification-categorization relationship. J Exp Psychol Gen. 1986; 115(1):39-61. DOI: 10.1037//0096-3445.115.1.39. View

17.

Navarro D, Pitt M, Myung I . Assessing the distinguishability of models and the informativeness of data. Cogn Psychol. 2004; 49(1):47-84. DOI: 10.1016/j.cogpsych.2003.11.001. View

18.

Vanpaemel W, Lee M . Using priors to formalize theory: optimal attention and the generalized context model. Psychon Bull Rev. 2012; 19(6):1047-56. DOI: 10.3758/s13423-012-0300-4. View

19.

Pitt M, Kim W, Myung I . Flexibility versus generalizability in model selection. Psychon Bull Rev. 2003; 10(1):29-44. DOI: 10.3758/bf03196467. View

20.

Pitt M, Myung I . When a good fit can be bad. Trends Cogn Sci. 2002; 6(10):421-425. DOI: 10.1016/s1364-6613(02)01964-2. View