Neural Circuitry of Reward Prediction Error

Overview

Journal Annu Rev Neurosci

Specialty Neurology

Date 2017 Apr 26

PMID 28441114

Citations 137

Authors

Mitsuko Watabe-Uchida

Neir Eshel

Naoshige Uchida

Affiliations

Soon will be listed here.

Abstract

Dopamine neurons facilitate learning by calculating reward prediction error, or the difference between expected and actual reward. Despite two decades of research, it remains unclear how dopamine neurons make this calculation. Here we review studies that tackle this problem from a diverse set of approaches, from anatomy to electrophysiology to computational modeling and behavior. Several patterns emerge from this synthesis: that dopamine neurons themselves calculate reward prediction error, rather than inherit it passively from upstream regions; that they combine multiple separate and redundant inputs, which are themselves interconnected in a dense recurrent network; and that despite the complexity of inputs, the output from dopamine neurons is remarkably homogeneous and robust. The more we study this simple arithmetic computation, the knottier it appears to be, suggesting a daunting (but stimulating) path ahead for neuroscience more generally.

Citing Articles

Natural behaviour is learned through dopamine-mediated reinforcement.

Kasdin J, Duffy A, Nadler N, Raha A, Fairhall A, Stachenfeld K Nature. 2025; .

PMID: 40074908 DOI: 10.1038/s41586-025-08729-1.

Dopamine activity encodes the changing valence of the same stimulus in conditioned taste aversion paradigms.

Loh M, Hurh S, Bazzino P, Donka R, Keinath A, Roitman J Elife. 2025; 13.

PMID: 40042246 PMC: 11882140. DOI: 10.7554/eLife.103260.

Omissions of threat trigger subjective relief and prediction error-like signaling in the human reward and salience systems.

Willems A, Van Oudenhove L, Vervliet B Elife. 2025; 12.

PMID: 40008871 PMC: 11875134. DOI: 10.7554/eLife.91400.

Contextual cues facilitate dynamic value encoding in the mesolimbic dopamine system.

Fraser K, Collins V, Wolff A, Ottenheimer D, Bornhoft K, Pat F Curr Biol. 2025; 35(4):746-760.e5.

PMID: 39855205 PMC: 11859769. DOI: 10.1016/j.cub.2024.12.031.

Distributed representations of temporally accumulated reward prediction errors in the mouse cortex.

Makino H, Suhaimi A Sci Adv. 2025; 11(4):eadi4782.

PMID: 39841828 PMC: 11753378. DOI: 10.1126/sciadv.adi4782.

References

Rao R, Ballard D . Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci. 1999; 2(1):79-87. DOI: 10.1038/4580. View

Contreras-Vidal J, Schultz W . A predictive reinforcement model of dopamine neurons for learning approach behavior. J Comput Neurosci. 1999; 6(3):191-214. DOI: 10.1023/a:1008862904946. View

Brown J, Bullock D, Grossberg S . How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J Neurosci. 1999; 19(23):10502-11. PMC: 6782432. View

Horvitz J . Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience. 2000; 96(4):651-6. DOI: 10.1016/s0306-4522(00)00019-1. View

Martin S, Grimwood P, Morris R . Synaptic plasticity and memory: an evaluation of the hypothesis. Annu Rev Neurosci. 2000; 23:649-711. DOI: 10.1146/annurev.neuro.23.1.649. View

Waelti P, Dickinson A, Schultz W . Dopamine responses comply with basic assumptions of formal learning theory. Nature. 2001; 412(6842):43-8. DOI: 10.1038/35083500. View

Neuhoff H, Neu A, Liss B, Roeper J . I(h) channels contribute to the different functional properties of identified dopaminergic subpopulations in the midbrain. J Neurosci. 2002; 22(4):1290-302. PMC: 6757558. View

Chance F, Abbott L, Reyes A . Gain modulation from background synaptic input. Neuron. 2002; 35(4):773-82. DOI: 10.1016/s0896-6273(02)00820-6. View

Joel D, Niv Y, Ruppin E . Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 2002; 15(4-6):535-47. DOI: 10.1016/s0893-6080(02)00047-3. View

10.

Fiorillo C, Tobler P, Schultz W . Discrete coding of reward probability and uncertainty by dopamine neurons. Science. 2003; 299(5614):1898-902. DOI: 10.1126/science.1077349. View

11.

Murphy B, Miller K . Multiplicative gain changes are induced by excitation or inhibition alone. J Neurosci. 2003; 23(31):10040-51. PMC: 6740852. View

12.

Shu Y, Hasenstaub A, Badoual M, Bal T, McCormick D . Barrages of synaptic activity control the gain and sensitivity of cortical neurons. J Neurosci. 2003; 23(32):10388-401. PMC: 6741011. View

13.

BUSH R, Mosteller F . A mathematical model for simple learning. Psychol Rev. 1951; 58(5):313-23. DOI: 10.1037/h0054388. View

14.

Wise R . Dopamine, learning and motivation. Nat Rev Neurosci. 2004; 5(6):483-94. DOI: 10.1038/nrn1406. View

15.

Morris G, Arkadir D, Nevet A, Vaadia E, Bergman H . Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron. 2004; 43(1):133-43. DOI: 10.1016/j.neuron.2004.06.012. View

16.

Vandecasteele M, Glowinski J, Venance L . Electrical synapses between dopaminergic neurons of the substantia nigra pars compacta. J Neurosci. 2005; 25(2):291-8. PMC: 6725490. DOI: 10.1523/JNEUROSCI.4167-04.2005. View

17.

Tobler P, Fiorillo C, Schultz W . Adaptive coding of reward value by dopamine neurons. Science. 2005; 307(5715):1642-5. DOI: 10.1126/science.1105370. View

18.

Bayer H, Glimcher P . Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron. 2005; 47(1):129-41. PMC: 1564381. DOI: 10.1016/j.neuron.2005.05.020. View

19.

Geisler S, Zahm D . Afferents of the ventral tegmental area in the rat-anatomical substratum for integrative functions. J Comp Neurol. 2005; 490(3):270-94. DOI: 10.1002/cne.20668. View

20.

Boyden E, Zhang F, Bamberg E, Nagel G, Deisseroth K . Millisecond-timescale, genetically targeted optical control of neural activity. Nat Neurosci. 2005; 8(9):1263-8. DOI: 10.1038/nn1525. View