The Serial Blocking Effect: a Testbed for the Neural Mechanisms of Temporal-difference Learning

Overview

Journal Sci Rep

Specialty Science

Date 2019 Apr 14

PMID 30979910

Citations 5

Authors

Ashraf Mahmud

Petio Petrov

Guillem R Esber

Mihaela D Iordanova

Affiliations

Soon will be listed here.

Abstract

Temporal-difference (TD) learning models afford the neuroscientist a theory-driven roadmap in the quest for the neural mechanisms of reinforcement learning. The application of these models to understanding the role of phasic midbrain dopaminergic responses in reward prediction learning constitutes one of the greatest success stories in behavioural and cognitive neuroscience. Critically, the classic learning paradigms associated with TD are poorly suited to cast light on its neural implementation, thus hampering progress. Here, we present a serial blocking paradigm in rodents that overcomes these limitations and allows for the simultaneous investigation of two cardinal TD tenets; namely, that learning depends on the computation of a prediction error, and that reinforcing value, whether intrinsic or acquired, propagates back to the onset of the earliest reliable predictor. The implications of this paradigm for the neural exploration of TD mechanisms are highlighted.

Citing Articles

Understanding Associative Learning Through Higher-Order Conditioning.

Gostolupce D, Lay B, Maes E, Iordanova M Front Behav Neurosci. 2022; 16:845616.

PMID: 35517574 PMC: 9062293. DOI: 10.3389/fnbeh.2022.845616.

Neural substrates of appetitive and aversive prediction error.

Iordanova M, Yau J, McDannald M, Corbit L Neurosci Biobehav Rev. 2021; 123:337-351.

PMID: 33453307 PMC: 7933120. DOI: 10.1016/j.neubiorev.2020.10.029.

Reward foraging task and model-based analysis reveal how fruit flies learn value of available options.

Seidenbecher S, Sanders J, von Philipsborn A, Kvitsiani D PLoS One. 2020; 15(10):e0239616.

PMID: 33007023 PMC: 7531776. DOI: 10.1371/journal.pone.0239616.

Different methods of fear reduction are supported by distinct cortical substrates.

Lay B, Pitaru A, Boulianne N, Esber G, Iordanova M Elife. 2020; 9.

PMID: 32589138 PMC: 7343386. DOI: 10.7554/eLife.55294.

Causal evidence supporting the proposal that dopamine transients function as temporal difference prediction errors.

Maes E, Sharpe M, Usypchuk A, Lozzi M, Chang C, Gardner M Nat Neurosci. 2020; 23(2):176-178.

PMID: 31959935 PMC: 7007380. DOI: 10.1038/s41593-019-0574-1.

References

Lauzon N, Bishop S, Laviolette S . Dopamine D1 versus D4 receptors differentially modulate the encoding of salient versus nonsalient emotional information in the medial prefrontal cortex. J Neurosci. 2009; 29(15):4836-45. PMC: 6665325. DOI: 10.1523/JNEUROSCI.0178-09.2009. View

Sharpe M, Chang C, Liu M, Batchelor H, Mueller L, Jones J . Dopamine transients are sufficient and necessary for acquisition of model-based associations. Nat Neurosci. 2017; 20(5):735-742. PMC: 5413864. DOI: 10.1038/nn.4538. View

Rasmussen K, Strecker R, Jacobs B . Single unit response of noradrenergic, serotonergic and dopaminergic neurons in freely moving cats to simple sensory stimuli. Brain Res. 1986; 369(1-2):336-40. DOI: 10.1016/0006-8993(86)90546-9. View

Iordanova M, Westbrook R, Killcross A . Dopamine activity in the nucleus accumbens modulates blocking in fear conditioning. Eur J Neurosci. 2006; 24(11):3265-70. DOI: 10.1111/j.1460-9568.2006.05195.x. View

Jennings D, Kirkpatrick K . Interval duration effects on blocking in appetitive conditioning. Behav Processes. 2005; 71(2-3):318-29. DOI: 10.1016/j.beproc.2005.11.007. View

McDannald M, Esber G, Wegener M, Wied H, Liu T, Stalnaker T . Orbitofrontal neurons acquire responses to 'valueless' Pavlovian cues during unblocking. Elife. 2014; 3:e02653. PMC: 4132288. DOI: 10.7554/eLife.02653. View

Yin H, Barnet R, Miller R . Second-order conditioning and Pavlovian conditioned inhibition: operational similarities and differences. J Exp Psychol Anim Behav Process. 1994; 20(4):419-28. View

Steinberg E, Keiflin R, Boivin J, Witten I, Deisseroth K, Janak P . A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci. 2013; 16(7):966-73. PMC: 3705924. DOI: 10.1038/nn.3413. View

Waelti P, Dickinson A, Schultz W . Dopamine responses comply with basic assumptions of formal learning theory. Nature. 2001; 412(6842):43-8. DOI: 10.1038/35083500. View

10.

Dieu Y, Seillier A, Majchrzak M, Marchand A, Di Scala G . Systemic or intra-accumbens injection of D-amphetamine delays habituation to a tone stimulus in rats. Behav Pharmacol. 2005; 16(1):35-42. DOI: 10.1097/00008877-200502000-00004. View

11.

Pearce J, Hall G . A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev. 1980; 87(6):532-52. View

12.

Cole S, McNally G . Opioid receptors mediate direct predictive fear learning: evidence from one-trial blocking. Learn Mem. 2007; 14(4):229-35. PMC: 2219694. DOI: 10.1101/lm.489507. View

13.

Quirk G, Armony J, LeDoux J . Fear conditioning enhances different temporal components of tone-evoked spike trains in auditory cortex and lateral amygdala. Neuron. 1997; 19(3):613-24. DOI: 10.1016/s0896-6273(00)80375-x. View

14.

Young A, Moran P, Joseph M . The role of dopamine in conditioning and latent inhibition: what, when, where and how?. Neurosci Biobehav Rev. 2005; 29(6):963-76. DOI: 10.1016/j.neubiorev.2005.02.004. View

15.

Blanchard R, Blanchard D . Crouching as an index of fear. J Comp Physiol Psychol. 1969; 67(3):370-5. DOI: 10.1037/h0026779. View

16.

Schreurs B, Westbrook R . The effects of changes in the CS-US interval during compound conditioning upon an otherwise blocked element. Q J Exp Psychol B. 1982; 34 (Pt 1):19-30. DOI: 10.1080/14640748208400887. View

17.

Barnet R, Grahame N, Miller R . Temporal encoding as a determinant of blocking. J Exp Psychol Anim Behav Process. 1993; 19(4):327-41. DOI: 10.1037//0097-7403.19.4.327. View

18.

Takahashi Y, Roesch M, Stalnaker T, Haney R, Calu D, Taylor A . The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron. 2009; 62(2):269-80. PMC: 2693075. DOI: 10.1016/j.neuron.2009.03.005. View

19.

Sierra-Mercado D, Padilla-Coreano N, Quirk G . Dissociable roles of prelimbic and infralimbic cortices, ventral hippocampus, and basolateral amygdala in the expression and extinction of conditioned fear. Neuropsychopharmacology. 2010; 36(2):529-38. PMC: 3005957. DOI: 10.1038/npp.2010.184. View

20.

Takahashi Y, Roesch M, Wilson R, Toreson K, ODonnell P, Niv Y . Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci. 2011; 14(12):1590-7. PMC: 3225718. DOI: 10.1038/nn.2957. View