Reinforcement Learning: Computational Theory and Biological Mechanisms

Overview

Journal HFSP J

Specialty Biology

Date 2009 May 1

PMID 19404458

Citations 48

Authors

Kenji Doya

Affiliations

Soon will be listed here.

Abstract

Reinforcement learning is a computational framework for an active agent to learn behaviors on the basis of a scalar reward signal. The agent can be an animal, a human, or an artificial system such as a robot or a computer program. The reward can be food, water, money, or whatever measure of the performance of the agent. The theory of reinforcement learning, which was developed in an artificial intelligence community with intuitions from animal learning theory, is now giving a coherent account on the function of the basal ganglia. It now serves as the "common language" in which biologists, engineers, and social scientists can exchange their problems and findings. This article reviews the basic theoretical framework of reinforcement learning and discusses its recent and future contributions toward the understanding of animal behaviors and human decision making.

Citing Articles

Distributed representations of temporally accumulated reward prediction errors in the mouse cortex.

Makino H, Suhaimi A Sci Adv. 2025; 11(4):eadi4782.

PMID: 39841828 PMC: 11753378. DOI: 10.1126/sciadv.adi4782.

Motor synergy and energy efficiency emerge in whole-body locomotion learning.

Li G, Hayashibe M Sci Rep. 2025; 15(1):712.

PMID: 39753645 PMC: 11698959. DOI: 10.1038/s41598-024-82472-x.

Dopamine transients encode reward prediction errors independent of learning rates.

Mah A, Golden C, Constantinople C Cell Rep. 2024; 43(10):114840.

PMID: 39395170 PMC: 11571066. DOI: 10.1016/j.celrep.2024.114840.

Dopamine transients encode reward prediction errors independent of learning rates.

Mah A, Golden C, Constantinople C bioRxiv. 2024; .

PMID: 38659861 PMC: 11042285. DOI: 10.1101/2024.04.18.590090.

An opponent striatal circuit for distributional reinforcement learning.

Lowet A, Zheng Q, Meng M, Matias S, Drugowitsch J, Uchida N bioRxiv. 2024; .

PMID: 38260354 PMC: 10802299. DOI: 10.1101/2024.01.02.573966.

References

Dorris M, Glimcher P . Activity in posterior parietal cortex is correlated with the relative subjective desirability of action. Neuron. 2004; 44(2):365-78. DOI: 10.1016/j.neuron.2004.09.009. View

Lee D, McGreevy B, Barraclough D . Learning and decision making in monkeys during a rock-paper-scissors game. Brain Res Cogn Brain Res. 2005; 25(2):416-30. DOI: 10.1016/j.cogbrainres.2005.07.003. View

Yoshida W, Ishii S . Resolution of uncertainty in prefrontal cortex. Neuron. 2006; 50(5):781-9. DOI: 10.1016/j.neuron.2006.05.006. View

Redgrave P, Gurney K . The short-latency dopamine signal: a role in discovering novel actions?. Nat Rev Neurosci. 2006; 7(12):967-75. DOI: 10.1038/nrn2022. View

Kawagoe R, Takikawa Y, Hikosaka O . Reward-predicting activity of dopamine and caudate neurons--a possible mechanism of motivational control of saccadic eye movement. J Neurophysiol. 2003; 91(2):1013-24. DOI: 10.1152/jn.00721.2003. View

Waelti P, Dickinson A, Schultz W . Dopamine responses comply with basic assumptions of formal learning theory. Nature. 2001; 412(6842):43-8. DOI: 10.1038/35083500. View

Reynolds J, Wickens J . Substantia nigra dopamine regulates synaptic plasticity and membrane potential fluctuations in the rat neostriatum, in vivo. Neuroscience. 2000; 99(2):199-203. DOI: 10.1016/s0306-4522(00)00273-6. View

Deiber M, Ibanez V, Honda M, Sadato N, Raman R, Hallett M . Cerebral processes related to visuomotor imagery and generation of simple finger movements studied with positron emission tomography. Neuroimage. 1998; 7(2):73-85. DOI: 10.1006/nimg.1997.0314. View

Contreras-Vidal J, Schultz W . A predictive reinforcement model of dopamine neurons for learning approach behavior. J Comput Neurosci. 1999; 6(3):191-214. DOI: 10.1023/a:1008862904946. View

10.

Balleine B, Killcross S . Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci. 2006; 29(5):272-9. DOI: 10.1016/j.tins.2006.03.002. View

11.

Marder E, Thirumalai V . Cellular, synaptic and network effects of neuromodulation. Neural Netw. 2002; 15(4-6):479-93. DOI: 10.1016/s0893-6080(02)00043-6. View

12.

Daw N, Doya K . The computational neurobiology of learning and reward. Curr Opin Neurobiol. 2006; 16(2):199-204. DOI: 10.1016/j.conb.2006.03.006. View

13.

Kawagoe R, Takikawa Y, Hikosaka O . Expectation of reward modulates cognitive signals in the basal ganglia. Nat Neurosci. 1999; 1(5):411-6. DOI: 10.1038/1625. View

14.

Schultz W, Tremblay L, Hollerman J . Reward processing in primate orbitofrontal cortex and basal ganglia. Cereb Cortex. 2000; 10(3):272-84. DOI: 10.1093/cercor/10.3.272. View

15.

Lee D, Conroy M, McGreevy B, Barraclough D . Reinforcement learning and decision making in monkeys during a competitive game. Brain Res Cogn Brain Res. 2004; 22(1):45-58. DOI: 10.1016/j.cogbrainres.2004.07.007. View

16.

Nakahara H, Itoh H, Kawagoe R, Takikawa Y, Hikosaka O . Dopamine neurons can represent context-dependent prediction error. Neuron. 2004; 41(2):269-80. DOI: 10.1016/s0896-6273(03)00869-9. View

17.

Schultz W . Predictive reward signal of dopamine neurons. J Neurophysiol. 1998; 80(1):1-27. DOI: 10.1152/jn.1998.80.1.1. View

18.

Kawato M . Internal models for motor control and trajectory planning. Curr Opin Neurobiol. 1999; 9(6):718-27. DOI: 10.1016/s0959-4388(99)00028-8. View

19.

Tanaka S, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S . Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci. 2004; 7(8):887-93. DOI: 10.1038/nn1279. View

20.

Houk J, Wise S . Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: their role in planning and controlling action. Cereb Cortex. 1995; 5(2):95-110. DOI: 10.1093/cercor/5.2.95. View