A Reinforcement Learning Mechanism Responsible for the Valuation of Free Choice

Overview

Journal Neuron

Publisher Cell Press

Specialty Neurology

Date 2014 Jul 29

PMID 25066083

Citations 45

Authors

Jeffrey Cockburn

Anne G E Collins

Michael J Frank

Affiliations

Soon will be listed here.

Abstract

Humans exhibit a preference for options they have freely chosen over equally valued options they have not; however, the neural mechanism that drives this bias and its functional significance have yet to be identified. Here, we propose a model in which choice biases arise due to amplified positive reward prediction errors associated with free choice. Using a novel variant of a probabilistic learning task, we show that choice biases are selective to options that are predominantly associated with positive outcomes. A polymorphism in DARPP-32, a gene linked to dopaminergic striatal plasticity and individual differences in reinforcement learning, was found to predict the effect of choice as a function of value. We propose that these choice biases are the behavioral byproduct of a credit assignment mechanism responsible for ensuring the effective delivery of dopaminergic reinforcement learning signals broadcast to the striatum.

Citing Articles

Neural dynamics underlying the illusion of control during reward processing.

Zheng Y, Yang C, Jiang H, Gao B Soc Cogn Affect Neurosci. 2024; 19(1).

PMID: 39300953 PMC: 11466228. DOI: 10.1093/scan/nsae063.

The influence of reward and loss outcomes after free- and forced-tasks on voluntary task choice.

Mittelstadt V, Mackenzie I, Leuthold H Psychol Res. 2024; 88(7):2059-2079.

PMID: 39078508 PMC: 11450031. DOI: 10.1007/s00426-024-02009-9.

Sensitivity to the Instrumental Value of Choice Increases Across Development.

Nussenbaum K, Katzman P, Lu H, Zorowitz S, Hartley C Psychol Sci. 2024; 35(8):933-947.

PMID: 38900963 PMC: 11693699. DOI: 10.1177/09567976241256961.

Adolescents flexibly adapt action selection based on controllability inferences.

Raab H, Goldway N, Foord C, Hartley C Learn Mem. 2024; 31(3).

PMID: 38527752 PMC: 11000582. DOI: 10.1101/lm.053901.123.

Prefrontal signals precede striatal signals for biased credit assignment in motivational learning biases.

Algermissen J, Swart J, Scheeringa R, Cools R, den Ouden H Nat Commun. 2024; 15(1):19.

PMID: 38168089 PMC: 10762147. DOI: 10.1038/s41467-023-44632-x.

References

Collins A, Frank M . Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol Rev. 2014; 121(3):337-66. DOI: 10.1037/a0037015. View

Egan L, Santos L, Bloom P . The origins of cognitive dissonance: evidence from children and monkeys. Psychol Sci. 2007; 18(11):978-83. DOI: 10.1111/j.1467-9280.2007.02012.x. View

Lee C, Abercrombie E, Tepper J . Pallidal control of substantia nigra dopaminergic neuron firing pattern and its relation to extracellular neostriatal dopamine levels. Neuroscience. 2004; 129(2):481-9. DOI: 10.1016/j.neuroscience.2004.07.034. View

Stipanovich A, Valjent E, Matamales M, Nishi A, Ahn J, Maroteaux M . A phosphatase cascade by which rewarding stimuli control nucleosomal response. Nature. 2008; 453(7197):879-84. PMC: 2796210. DOI: 10.1038/nature06994. View

Leotti L, Delgado M . The value of exercising control over monetary gains and losses. Psychol Sci. 2014; 25(2):596-604. PMC: 3970926. DOI: 10.1177/0956797613514589. View

St Onge J, Ahn S, Phillips A, Floresco S . Dynamic fluctuations in dopamine efflux in the prefrontal cortex and nucleus accumbens during risk-based decision making. J Neurosci. 2012; 32(47):16880-91. PMC: 6621758. DOI: 10.1523/JNEUROSCI.3807-12.2012. View

Matsumoto M, Weickert C, Akil M, Lipska B, Hyde T, Herman M . Catechol O-methyltransferase mRNA expression in human and rat brain: evidence for a role in cortical neuronal function. Neuroscience. 2003; 116(1):127-37. DOI: 10.1016/s0306-4522(02)00556-0. View

Lieberman M, Ochsner K, Gilbert D, Schacter D . Do amnesics exhibit cognitive dissonance reduction? The role of explicit memory and attention in attitude change. Psychol Sci. 2001; 12(2):135-40. DOI: 10.1111/1467-9280.00323. View

Frank M, Doll B, Oas-Terpstra J, Moreno F . Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci. 2009; 12(8):1062-8. PMC: 3062477. DOI: 10.1038/nn.2342. View

10.

Sharot T, Velasquez C, Dolan R . Do decisions shape preference? Evidence from blind choice. Psychol Sci. 2010; 21(9):1231-5. PMC: 3196841. DOI: 10.1177/0956797610379235. View

11.

Huotari M, Gogos J, Karayiorgou M, Koponen O, Forsberg M, Raasmaja A . Brain catecholamine metabolism in catechol-O-methyltransferase (COMT)-deficient mice. Eur J Neurosci. 2002; 15(2):246-56. DOI: 10.1046/j.0953-816x.2001.01856.x. View

12.

Maunsell J . Neuronal representations of cognitive state: reward or attention?. Trends Cogn Sci. 2004; 8(6):261-5. DOI: 10.1016/j.tics.2004.04.003. View

13.

Lobb C, Troyer T, Wilson C, Paladini C . Disinhibition bursting of dopaminergic neurons. Front Syst Neurosci. 2011; 5:25. PMC: 3095811. DOI: 10.3389/fnsys.2011.00025. View

14.

Doll B, Hutchison K, Frank M . Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J Neurosci. 2011; 31(16):6188-98. PMC: 3098533. DOI: 10.1523/JNEUROSCI.6486-10.2011. View

15.

ODoherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan R . Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science. 2004; 304(5669):452-4. DOI: 10.1126/science.1094285. View

16.

Ashby F, Ennis J, Spiering B . A neurobiological theory of automaticity in perceptual categorization. Psychol Rev. 2007; 114(3):632-56. DOI: 10.1037/0033-295X.114.3.632. View

17.

Frank M, Moustafa A, Haughey H, Curran T, Hutchison K . Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci U S A. 2007; 104(41):16311-6. PMC: 2042203. DOI: 10.1073/pnas.0706111104. View

18.

Stephan K, Penny W, Daunizeau J, Moran R, Friston K . Bayesian model selection for group studies. Neuroimage. 2009; 46(4):1004-17. PMC: 2703732. DOI: 10.1016/j.neuroimage.2009.03.025. View

19.

Joel D, Weiner I . The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience. 2000; 96(3):451-74. DOI: 10.1016/s0306-4522(99)00575-8. View

20.

Frank M . Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci. 2005; 17(1):51-72. DOI: 10.1162/0898929052880093. View