Executive Function Assigns Value to Novel Goal-Congruent Outcomes

Overview

Journal Cereb Cortex

Publisher Oxford University Press

Specialty Neurology

Date 2021 Jul 7

PMID 34231854

Citations 5

Authors

Samuel D McDougle

Ian C Ballard

Beth Baribault

Sonia J Bishop

Anne G E Collins

Affiliations

Soon will be listed here.

Abstract

People often learn from the outcomes of their actions, even when these outcomes do not involve material rewards or punishments. How does our brain provide this flexibility? We combined behavior, computational modeling, and functional neuroimaging to probe whether learning from abstract novel outcomes harnesses the same circuitry that supports learning from familiar secondary reinforcers. Behavior and neuroimaging revealed that novel images can act as a substitute for rewards during instrumental learning, producing reliable reward-like signals in dopaminergic circuits. Moreover, we found evidence that prefrontal correlates of executive control may play a role in shaping flexible responses in reward circuits. These results suggest that learning from novel outcomes is supported by an interplay between high-level representations in prefrontal cortex and low-level responses in subcortical reward circuits. This interaction may allow for human reinforcement learning over arbitrarily abstract reward functions.

Citing Articles

Insight in obsessive-compulsive disorder: conception, clinical characteristics, neuroimaging, and treatment.

Huang Y, Weng Y, Lan L, Zhu C, Shen T, Tang W Psychoradiology. 2024; 3:kkad025.

PMID: 38666121 PMC: 10917385. DOI: 10.1093/psyrad/kkad025.

Reward Reinforcement Creates Enduring Facilitation of Goal-directed Behavior.

Ballard I, Waskom M, Nix K, DEsposito M J Cogn Neurosci. 2024; 36(12):2847-2862.

PMID: 38579249 PMC: 11602007. DOI: 10.1162/jocn_a_02150.

Intrinsic rewards explain context-sensitive valuation in reinforcement learning.

Molinaro G, Collins A PLoS Biol. 2023; 21(7):e3002201.

PMID: 37459394 PMC: 10374061. DOI: 10.1371/journal.pbio.3002201.

Sensorimotor feedback loops are selectively sensitive to reward.

Codol O, Kashefi M, Forgaard C, Galea J, Pruszynski J, Gribble P Elife. 2023; 12.

PMID: 36637162 PMC: 9910828. DOI: 10.7554/eLife.81325.

Advances in modeling learning and decision-making in neuroscience.

Collins A, Shenhav A Neuropsychopharmacology. 2021; 47(1):104-118.

PMID: 34453117 PMC: 8617262. DOI: 10.1038/s41386-021-01126-y.

References

Esteban O, Markiewicz C, Blair R, Moodie C, Isik A, Erramuzpe A . fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat Methods. 2018; 16(1):111-116. PMC: 6319393. DOI: 10.1038/s41592-018-0235-4. View

Howard J, Gottfried J, Tobler P, Kahnt T . Identity-specific coding of future rewards in the human orbitofrontal cortex. Proc Natl Acad Sci U S A. 2015; 112(16):5195-200. PMC: 4413264. DOI: 10.1073/pnas.1503550112. View

McDougle S, Butcher P, Parvin D, Mushtaq F, Niv Y, Ivry R . Neural Signatures of Prediction Errors in a Decision-Making Task Are Modulated by Action Execution Failures. Curr Biol. 2019; 29(10):1606-1613.e5. PMC: 6535105. DOI: 10.1016/j.cub.2019.04.011. View

Collins A, Brown J, Gold J, Waltz J, Frank M . Working memory contributions to reinforcement learning impairments in schizophrenia. J Neurosci. 2014; 34(41):13747-56. PMC: 4188972. DOI: 10.1523/JNEUROSCI.0989-14.2014. View

Fromer R, Dean Wolf C, Shenhav A . Goal congruency dominates reward value in accounting for behavioral and neural correlates of value-based decision-making. Nat Commun. 2019; 10(1):4926. PMC: 6820735. DOI: 10.1038/s41467-019-12931-x. View

Piray P, Dezfouli A, Heskes T, Frank M, Daw N . Hierarchical Bayesian inference for concurrent model fitting and comparison for group studies. PLoS Comput Biol. 2019; 15(6):e1007043. PMC: 6581260. DOI: 10.1371/journal.pcbi.1007043. View

McClure S, Berns G, Montague P . Temporal prediction errors in a passive learning task activate human striatum. Neuron. 2003; 38(2):339-46. DOI: 10.1016/s0896-6273(03)00154-5. View

Davidow J, Foerde K, Galvan A, Shohamy D . An Upside to Reward Sensitivity: The Hippocampus Supports Enhanced Reinforcement Learning in Adolescence. Neuron. 2016; 92(1):93-99. DOI: 10.1016/j.neuron.2016.08.031. View

Langdon A, Sharpe M, Schoenbaum G, Niv Y . Model-based predictions for dopamine. Curr Opin Neurobiol. 2017; 49:1-7. PMC: 6034703. DOI: 10.1016/j.conb.2017.10.006. View

10.

Collins A, Cockburn J . Beyond dichotomies in reinforcement learning. Nat Rev Neurosci. 2020; 21(10):576-586. PMC: 7800310. DOI: 10.1038/s41583-020-0355-6. View

11.

Yeo B, Krienen F, Eickhoff S, Yaakub S, Fox P, Buckner R . Functional Specialization and Flexibility in Human Association Cortex. Cereb Cortex. 2014; 25(10):3654-72. PMC: 4598819. DOI: 10.1093/cercor/bhu217. View

12.

Pashler H . Dual-task interference in simple tasks: data and theory. Psychol Bull. 1994; 116(2):220-44. DOI: 10.1037/0033-2909.116.2.220. View

13.

Wilson R, Takahashi Y, Schoenbaum G, Niv Y . Orbitofrontal cortex as a cognitive map of task space. Neuron. 2014; 81(2):267-279. PMC: 4001869. DOI: 10.1016/j.neuron.2013.11.005. View

14.

Manoach D, Greve D, Lindgren K, Dale A . Identifying regional activity associated with temporally separated components of working memory using event-related functional MRI. Neuroimage. 2003; 20(3):1670-84. DOI: 10.1016/j.neuroimage.2003.08.002. View

15.

Emrich S, Riggall A, LaRocque J, Postle B . Distributed patterns of activity in sensory cortex reflect the precision of multiple items maintained in visual short-term memory. J Neurosci. 2013; 33(15):6516-23. PMC: 3664518. DOI: 10.1523/JNEUROSCI.5732-12.2013. View

16.

Doll B, Jacobs W, Sanfey A, Frank M . Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res. 2009; 1299:74-94. PMC: 3050481. DOI: 10.1016/j.brainres.2009.07.007. View

17.

Doll B, Simon D, Daw N . The ubiquity of model-based reinforcement learning. Curr Opin Neurobiol. 2012; 22(6):1075-81. PMC: 3513648. DOI: 10.1016/j.conb.2012.08.003. View

18.

Collins A, Ciullo B, Frank M, Badre D . Working Memory Load Strengthens Reward Prediction Errors. J Neurosci. 2017; 37(16):4332-4342. PMC: 5413179. DOI: 10.1523/JNEUROSCI.2700-16.2017. View

19.

Keramati M, Dezfouli A, Piray P . Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol. 2011; 7(5):e1002055. PMC: 3102758. DOI: 10.1371/journal.pcbi.1002055. View

20.

Han S, Huettel S, Raposo A, Adcock R, Dobbins I . Functional significance of striatal responses during episodic decisions: recovery or goal attainment?. J Neurosci. 2010; 30(13):4767-75. PMC: 3433049. DOI: 10.1523/JNEUROSCI.3077-09.2010. View