A New Model of Decision Processing in Instrumental Learning Tasks

Overview

Journal Elife

Specialty Biology

Date 2021 Jan 27

PMID 33501916

Citations 17

Authors

Steven Miletic

Russell J Boag

Anne C Trutti

Niek Stevenson

Birte U Forstmann

Andrew Heathcote

Affiliations

Soon will be listed here.

Abstract

Learning and decision-making are interactive processes, yet cognitive modeling of error-driven learning and decision-making have largely evolved separately. Recently, evidence accumulation models (EAMs) of decision-making and reinforcement learning (RL) models of error-driven learning have been combined into joint RL-EAMs that can in principle address these interactions. However, we show that the most commonly used combination, based on the diffusion decision model (DDM) for binary choice, consistently fails to capture crucial aspects of response times observed during reinforcement learning. We propose a new RL-EAM based on an advantage racing diffusion (ARD) framework for choices among two or more options that not only addresses this problem but captures stimulus difficulty, speed-accuracy trade-off, and stimulus-response-mapping reversal effects. The RL-ARD avoids fundamental limitations imposed by the DDM on addressing effects of absolute values of choices, as well as extensions beyond binary choice, and provides a computationally tractable basis for wider applications.

Citing Articles

Joint modeling of choices and reaction times based on Bayesian contextual behavioral control.

Schwobel S, Markovic D, Smolka M, Kiebel S PLoS Comput Biol. 2024; 20(7):e1012228.

PMID: 38968304 PMC: 11290629. DOI: 10.1371/journal.pcbi.1012228.

How cortico-basal ganglia-thalamic subnetworks can shift decision policies to maximize reward rate.

Bahuguna J, Verstynen T, Rubin J bioRxiv. 2024; .

PMID: 38826315 PMC: 11142098. DOI: 10.1101/2024.05.21.595174.

EZ-CDM: Fast, simple, robust, and accurate estimation of circular diffusion model parameters.

Qarehdaghi H, Rad J Psychon Bull Rev. 2024; 31(5):2058-2091.

PMID: 38587755 DOI: 10.3758/s13423-024-02483-7.

The effects of non-diagnostic information on confidence and decision making.

Kohl A, Sauer J, Palmer M, Brooks J, Heathcote A Mem Cognit. 2024; 52(5):1182-1194.

PMID: 38489145 PMC: 11315710. DOI: 10.3758/s13421-024-01535-6.

Joint Modelling of Latent Cognitive Mechanisms Shared Across Decision-Making Domains.

Stevenson N, Innes R, Boag R, Miletic S, Isherwood S, Trutti A Comput Brain Behav. 2024; 7(1):1-22.

PMID: 38425991 PMC: 10899373. DOI: 10.1007/s42113-023-00192-3.

References

Turner B, Sederberg P . A generalized, likelihood-free method for posterior estimation. Psychon Bull Rev. 2013; 21(2):227-50. PMC: 4143986. DOI: 10.3758/s13423-013-0530-0. View

Trueblood J, Heathcote A, Evans N, Holmes W . Urgency, leakage, and the relative nature of information processing in decision-making. Psychol Rev. 2020; 128(1):160-186. DOI: 10.1037/rev0000255. View

Palminteri S, Wyart V, Koechlin E . The Importance of Falsification in Computational Cognitive Modeling. Trends Cogn Sci. 2017; 21(6):425-433. DOI: 10.1016/j.tics.2017.03.011. View

Collins A, Frank M . Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory. Proc Natl Acad Sci U S A. 2018; 115(10):2502-2507. PMC: 5877949. DOI: 10.1073/pnas.1720963115. View

Pedersen M, Frank M, Biele G . The drift diffusion model as the choice rule in reinforcement learning. Psychon Bull Rev. 2016; 24(4):1234-1251. PMC: 5487295. DOI: 10.3758/s13423-016-1199-y. View

Bogacz R, Larsen T . Integration of reinforcement learning and optimal decision-making theories of the basal ganglia. Neural Comput. 2011; 23(4):817-51. DOI: 10.1162/NECO_a_00103. View

Miletic S, Van Maanen L . Caution in decision-making under time pressure is mediated by timing ability. Cogn Psychol. 2019; 110:16-29. DOI: 10.1016/j.cogpsych.2019.01.002. View

Teodorescu A, Moran R, Usher M . Absolutely relative or relatively absolute: violations of value invariance in human decision making. Psychon Bull Rev. 2015; 23(1):22-38. DOI: 10.3758/s13423-015-0858-8. View

Van Maanen L, Miletic S . The interpretation of behavior-model correlations in unidentified cognitive models. Psychon Bull Rev. 2020; 28(2):374-383. PMC: 8062378. DOI: 10.3758/s13423-020-01783-y. View

10.

Spektor M, Kellen D . The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making : Empirical priors. Psychon Bull Rev. 2018; 25(6):2047-2068. DOI: 10.3758/s13423-018-1446-5. View

11.

Van Maanen L, van der Mijn R, Van Beurden M, Roijendijk L, Kingma B, Miletic S . Core body temperature speeds up temporal processing and choice behavior under deadlines. Sci Rep. 2019; 9(1):10053. PMC: 6624282. DOI: 10.1038/s41598-019-46073-3. View

12.

Reddi B, Carpenter R . The influence of urgency on decision time. Nat Neurosci. 2000; 3(8):827-30. DOI: 10.1038/77739. View

13.

Logan G, Van Zandt T, Verbruggen F, Wagenmakers E . On the ability to inhibit thought and action: general and special theories of an act of control. Psychol Rev. 2014; 121(1):66-95. DOI: 10.1037/a0035230. View

14.

Millner A, Gershman S, Nock M, den Ouden H . Pavlovian Control of Escape and Avoidance. J Cogn Neurosci. 2017; 30(10):1379-1390. DOI: 10.1162/jocn_a_01224. View

15.

Peirce J, Gray J, Simpson S, Macaskill M, Hochenberger R, Sogo H . PsychoPy2: Experiments in behavior made easy. Behav Res Methods. 2019; 51(1):195-203. PMC: 6420413. DOI: 10.3758/s13428-018-01193-y. View

16.

Collins A, Frank M . How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur J Neurosci. 2012; 35(7):1024-35. PMC: 3390186. DOI: 10.1111/j.1460-9568.2011.07980.x. View

17.

Fontanesi L, Gluth S, Spektor M, Rieskamp J . A reinforcement learning diffusion decision model for value-based decisions. Psychon Bull Rev. 2019; 26(4):1099-1121. PMC: 6820465. DOI: 10.3758/s13423-018-1554-2. View

18.

Bowman N, Kording K, Gottfried J . Temporal integration of olfactory perceptual evidence in human orbitofrontal cortex. Neuron. 2012; 75(5):916-27. PMC: 3441053. DOI: 10.1016/j.neuron.2012.06.035. View

19.

Miletic S, Boag R, Forstmann B . Mutual benefits: Combining reinforcement learning with sequential sampling models. Neuropsychologia. 2019; 136:107261. DOI: 10.1016/j.neuropsychologia.2019.107261. View

20.

Anders R, Alario F, Van Maanen L . The shifted Wald distribution for response time data analysis. Psychol Methods. 2016; 21(3):309-27. DOI: 10.1037/met0000066. View