Adaptive Integration of Habits into Depth-limited Planning Defines a Habitual-goal-directed Spectrum

Overview

Journal Proc Natl Acad Sci U S A

Specialty Science

Date 2016 Oct 30

PMID 27791110

Citations 59

Authors

Mehdi Keramati

Peter Smittenaar

Raymond J Dolan

Peter Dayan

Affiliations

Soon will be listed here.

Abstract

Behavioral and neural evidence reveal a prospective goal-directed decision process that relies on mental simulation of the environment, and a retrospective habitual process that caches returns previously garnered from available choices. Artificial systems combine the two by simulating the environment up to some depth and then exploiting habitual values as proxies for consequences that may arise in the further future. Using a three-step task, we provide evidence that human subjects use such a normative plan-until-habit strategy, implying a spectrum of approaches that interpolates between habitual and goal-directed responding. We found that increasing time pressure led to shallower goal-directed planning, suggesting that a speed-accuracy tradeoff controls the depth of planning with deeper search leading to more accurate evaluation, at the cost of slower decision-making. We conclude that subjects integrate habit-based cached values directly into goal-directed evaluations in a normative manner.

Citing Articles

Noradrenergic and Dopaminergic modulation of meta-cognition and meta-control.

Ershadmanesh S, Rajabi S, Rostami R, Moran R, Dayan P PLoS Comput Biol. 2025; 21(2):e1012675.

PMID: 40009609 PMC: 11903042. DOI: 10.1371/journal.pcbi.1012675.

Exploring replay.

Antonov G, Dayan P Nat Commun. 2025; 16(1):1657.

PMID: 39955280 PMC: 11829958. DOI: 10.1038/s41467-025-56731-y.

Expert navigators deploy rational complexity-based decision precaching for large-scale real-world planning.

Fernandez Velasco P, Griesbauer E, Brunec I, Morley J, Manley E, McNamee D Proc Natl Acad Sci U S A. 2025; 122(4):e2407814122.

PMID: 39847337 PMC: 11789086. DOI: 10.1073/pnas.2407814122.

Choosing the right frame: how context preferences facilitate subsequent decisions.

Treiman L, Kool W Sci Rep. 2024; 14(1):31607.

PMID: 39738252 PMC: 11685499. DOI: 10.1038/s41598-024-79510-z.

Approximate planning in spatial search.

Kryven M, Yu S, Kleiman-Weiner M, Ullman T, Tenenbaum J PLoS Comput Biol. 2024; 20(11):e1012582.

PMID: 39531486 PMC: 11584085. DOI: 10.1371/journal.pcbi.1012582.

References

Schultz W, Dayan P, Montague P . A neural substrate of prediction and reward. Science. 1997; 275(5306):1593-9. DOI: 10.1126/science.275.5306.1593. View

Pezzulo G, Rigoli F, Chersi F . The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front Psychol. 2013; 4:92. PMC: 3586710. DOI: 10.3389/fpsyg.2013.00092. View

Dolan R, Dayan P . Goals and habits in the brain. Neuron. 2013; 80(2):312-25. PMC: 3807793. DOI: 10.1016/j.neuron.2013.09.007. View

Smittenaar P, FitzGerald T, Romei V, Wright N, Dolan R . Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron. 2013; 80(4):914-9. PMC: 3893454. DOI: 10.1016/j.neuron.2013.08.009. View

Lee S, Shimojo S, ODoherty J . Neural computations underlying arbitration between model-based and model-free learning. Neuron. 2014; 81(3):687-99. PMC: 3968946. DOI: 10.1016/j.neuron.2013.11.028. View

Doya K . What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?. Neural Netw. 2003; 12(7-8):961-974. DOI: 10.1016/s0893-6080(99)00046-5. View

Everitt B, Robbins T . Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat Neurosci. 2005; 8(11):1481-9. DOI: 10.1038/nn1579. View

Dayan P . How to set the switches on this thing. Curr Opin Neurobiol. 2012; 22(6):1068-74. DOI: 10.1016/j.conb.2012.05.011. View

Glascher J, Daw N, Dayan P, ODoherty J . States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron. 2010; 66(4):585-95. PMC: 2895323. DOI: 10.1016/j.neuron.2010.04.016. View

10.

Doll B, Duncan K, Simon D, Shohamy D, Daw N . Model-based choices involve prospective neural activity. Nat Neurosci. 2015; 18(5):767-72. PMC: 4414826. DOI: 10.1038/nn.3981. View

11.

Yin H, Knowlton B, Balleine B . Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci. 2004; 19(1):181-9. DOI: 10.1111/j.1460-9568.2004.03095.x. View

12.

Huys Q, Eshel N, ONions E, Sheridan L, Dayan P, Roiser J . Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput Biol. 2012; 8(3):e1002410. PMC: 3297555. DOI: 10.1371/journal.pcbi.1002410. View

13.

Akam T, Costa R, Dayan P . Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task. PLoS Comput Biol. 2015; 11(12):e1004648. PMC: 4686094. DOI: 10.1371/journal.pcbi.1004648. View

14.

Radenbach C, Reiter A, Engert V, Sjoerds Z, Villringer A, Heinze H . The interaction of acute and chronic stress impairs model-based behavioral control. Psychoneuroendocrinology. 2015; 53:268-80. DOI: 10.1016/j.psyneuen.2014.12.017. View

15.

Dezfouli A, Balleine B . Habits, action sequences and reinforcement learning. Eur J Neurosci. 2012; 35(7):1036-51. PMC: 3325518. DOI: 10.1111/j.1460-9568.2012.08050.x. View

16.

Gershman S, Markman A, Otto A . Retrospective revaluation in sequential decision making: a tale of two systems. J Exp Psychol Gen. 2012; 143(1):182-94. DOI: 10.1037/a0030844. View

17.

Balleine B, ODoherty J . Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology. 2009; 35(1):48-69. PMC: 3055420. DOI: 10.1038/npp.2009.131. View

18.

Valentin V, Dickinson A, ODoherty J . Determining the neural substrates of goal-directed learning in the human brain. J Neurosci. 2007; 27(15):4019-26. PMC: 6672546. DOI: 10.1523/JNEUROSCI.0564-07.2007. View

19.

Daw N, Gershman S, Seymour B, Dayan P, Dolan R . Model-based influences on humans' choices and striatal prediction errors. Neuron. 2011; 69(6):1204-15. PMC: 3077926. DOI: 10.1016/j.neuron.2011.02.027. View

20.

Gillan C, Kosinski M, Whelan R, Phelps E, Daw N . Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. Elife. 2016; 5. PMC: 4786435. DOI: 10.7554/eLife.11305. View