Dopamine Role in Learning and Action Inference

Overview

Journal Elife

Specialty Biology

Date 2020 Jul 8

PMID 32633715

Citations 18

Authors

Rafal Bogacz

Affiliations

Soon will be listed here.

Abstract

This paper describes a framework for modelling dopamine function in the mammalian brain. It proposes that both learning and action planning involve processes minimizing prediction errors encoded by dopaminergic neurons. In this framework, dopaminergic neurons projecting to different parts of the striatum encode errors in predictions made by the corresponding systems within the basal ganglia. The dopaminergic neurons encode differences between rewards and expectations in the goal-directed system, and differences between the chosen and habitual actions in the habit system. These prediction errors trigger learning about rewards and habit formation, respectively. Additionally, dopaminergic neurons in the goal-directed system play a key role in action planning: They compute the difference between a desired reward and the reward expected from the current motor plan, and they facilitate action planning until this difference diminishes. Presented models account for dopaminergic responses during movements, effects of dopamine depletion on behaviour, and make several experimental predictions.

Citing Articles

Noradrenergic and Dopaminergic modulation of meta-cognition and meta-control.

Ershadmanesh S, Rajabi S, Rostami R, Moran R, Dayan P PLoS Comput Biol. 2025; 21(2):e1012675.

PMID: 40009609 PMC: 11903042. DOI: 10.1371/journal.pcbi.1012675.

Hippocampal neuronal activity is aligned with action plans.

Zutshi I, Apostolelli A, Yang W, Zheng Z, Dohi T, Balzani E Nature. 2025; 639(8053):153-161.

PMID: 39779866 DOI: 10.1038/s41586-024-08397-7.

Substantial Heritability Underlies Fairness Norm Adaptation Capability and its Neural Basis.

Jin Y, Zheng D, Gu R, Fan Q, Dietz M, Wang C Adv Sci (Weinh). 2024; 12(9):e2411070.

PMID: 39679781 PMC: 11884581. DOI: 10.1002/advs.202411070.

Iron deficiency negatively affects behavioral measures of learning, indirect neural measures of dopamine, and neural efficiency.

Rhoten S, Wenger M, De Stefano L Cogn Affect Behav Neurosci. 2024; 25(1):89-113.

PMID: 39638921 DOI: 10.3758/s13415-024-01241-5.

Reward Bases: A simple mechanism for adaptive acquisition of multiple reward types.

Millidge B, Song Y, Lak A, Walton M, Bogacz R PLoS Comput Biol. 2024; 20(11):e1012580.

PMID: 39561186 PMC: 11614280. DOI: 10.1371/journal.pcbi.1012580.

References

Bogacz R . A tutorial on the free-energy framework for modelling perception and learning. J Math Psychol. 2017; 76(Pt B):198-211. PMC: 5341759. DOI: 10.1016/j.jmp.2015.11.003. View

Hardwick R, Forrence A, Krakauer J, Haith A . Time-dependent competition between goal-directed and habitual response preparation. Nat Hum Behav. 2019; 3(12):1252-1262. DOI: 10.1038/s41562-019-0725-0. View

de Wit S, Barker R, Dickinson A, Cools R . Habitual versus goal-directed action control in Parkinson disease. J Cogn Neurosci. 2010; 23(5):1218-29. DOI: 10.1162/jocn.2010.21514. View

Eshel N, Tian J, Bukwich M, Uchida N . Dopamine neurons share common response function for reward prediction error. Nat Neurosci. 2016; 19(3):479-86. PMC: 4767554. DOI: 10.1038/nn.4239. View

Huntley G, Morrison J, Prikhozhan A, Sealfon S . Localization of multiple dopamine receptor subtype mRNAs in human and monkey motor cortex and striatum. Brain Res Mol Brain Res. 1992; 15(3-4):181-8. DOI: 10.1016/0169-328x(92)90107-m. View

Stephan K, Manjaly Z, Mathys C, Weber L, Paliwal S, Gard T . Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression. Front Hum Neurosci. 2016; 10:550. PMC: 5108808. DOI: 10.3389/fnhum.2016.00550. View

Burton A, Nakamura K, Roesch M . From ventral-medial to dorsal-lateral striatum: neural correlates of reward-guided decision-making. Neurobiol Learn Mem. 2014; 117:51-9. PMC: 4240773. DOI: 10.1016/j.nlm.2014.05.003. View

FitzGerald T, Dolan R, Friston K . Model averaging, optimal inference, and habit formation. Front Hum Neurosci. 2014; 8:457. PMC: 4071291. DOI: 10.3389/fnhum.2014.00457. View

Bogacz R, Gurney K . The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Comput. 2007; 19(2):442-77. DOI: 10.1162/neco.2007.19.2.442. View

10.

Yin H, Ostlund S, Knowlton B, Balleine B . The role of the dorsomedial striatum in instrumental conditioning. Eur J Neurosci. 2005; 22(2):513-23. DOI: 10.1111/j.1460-9568.2005.04218.x. View

11.

Rochester L, Hetherington V, Jones D, Nieuwboer A, Willems A, Kwakkel G . The effect of external rhythmic cues (auditory and visual) on walking during a functional task in homes of people with Parkinson's disease. Arch Phys Med Rehabil. 2005; 86(5):999-1006. DOI: 10.1016/j.apmr.2004.10.040. View

12.

Gershman S . Dopamine, Inference, and Uncertainty. Neural Comput. 2017; 29(12):3311-3326. DOI: 10.1162/neco_a_01023. View

13.

Roelfsema P, van Ooyen A . Attention-gated reinforcement learning of internal representations for classification. Neural Comput. 2005; 17(10):2176-214. DOI: 10.1162/0899766054615699. View

14.

Keramati M, Gutkin B . Homeostatic reinforcement learning for integrating reward collection and physiological stability. Elife. 2014; 3. PMC: 4270100. DOI: 10.7554/eLife.04811. View

15.

Howe M, Dombeck D . Rapid signalling in distinct dopaminergic axons during locomotion and reward. Nature. 2016; 535(7613):505-10. PMC: 4970879. DOI: 10.1038/nature18942. View

16.

Yin H, Knowlton B, Balleine B . Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci. 2004; 19(1):181-9. DOI: 10.1111/j.1460-9568.2004.03095.x. View

17.

Kravitz A, Freeze B, Parker P, Kay K, Thwin M, Deisseroth K . Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature. 2010; 466(7306):622-6. PMC: 3552484. DOI: 10.1038/nature09159. View

18.

Dodson P, Dreyer J, Jennings K, Syed E, Wade-Martins R, Cragg S . Representation of spontaneous movement by dopaminergic neurons is cell-type selective and disrupted in parkinsonism. Proc Natl Acad Sci U S A. 2016; 113(15):E2180-8. PMC: 4839395. DOI: 10.1073/pnas.1515941113. View

19.

Collins A, Frank M . Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol Rev. 2014; 121(3):337-66. DOI: 10.1037/a0037015. View

20.

Watabe-Uchida M, Zhu L, Ogawa S, Vamanrao A, Uchida N . Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron. 2012; 74(5):858-73. DOI: 10.1016/j.neuron.2012.03.017. View