Do Not Bet on the Unknown Versus Try to Find Out More: Estimation Uncertainty and "Unexpected Uncertainty" Both Modulate Exploration

Overview

Journal Front Neurosci

Date 2012 Oct 23

PMID 23087606

Citations 23

Authors

Elise Payzan-LeNestour

Peter Bossaerts

Affiliations

Soon will be listed here.

Abstract

Little is known about how humans solve the exploitation/exploration trade-off. In particular, the evidence for uncertainty-driven exploration is mixed. The current study proposes a novel hypothesis of exploration that helps reconcile prior findings that may seem contradictory at first. According to this hypothesis, uncertainty-driven exploration involves a dilemma between two motives: (i) to speed up learning about the unknown, which may beget novel reward opportunities; (ii) to avoid the unknown because it is potentially dangerous. We provide evidence for our hypothesis using both behavioral and simulated data, and briefly point to recent evidence that the brain differentiates between these two motives.

Citing Articles

Losses resulting from deliberate exploration trigger beta oscillations in frontal cortex.

Chernyshev B, Pultsina K, Tretyakova V, Miasnikova A, Prokofyev A, Kozunova G Front Neurosci. 2023; 17:1152926.

PMID: 37250414 PMC: 10211346. DOI: 10.3389/fnins.2023.1152926.

Neurons in human pre-supplementary motor area encode key computations for value-based choice.

Aquino T, Cockburn J, Mamelak A, Rutishauser U, ODoherty J Nat Hum Behav. 2023; 7(6):970-985.

PMID: 36959327 PMC: 10330469. DOI: 10.1038/s41562-023-01548-2.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy.

Brown V, Price R, Dombrovski A Cogn Affect Behav Neurosci. 2023; 23(3):844-868.

PMID: 36869259 PMC: 10475148. DOI: 10.3758/s13415-023-01080-w.

The effects of time horizon and guided choices on explore-exploit decisions in rodents.

Wang S, Gerken B, Wieland J, Wilson R, Fellous J Behav Neurosci. 2023; 137(2):127-142.

PMID: 36633987 PMC: 10787949. DOI: 10.1037/bne0000549.

Humans adaptively resolve the explore-exploit dilemma under cognitive constraints: Evidence from a multi-armed bandit task.

Brown V, Hallquist M, Frank M, Dombrovski A Cognition. 2022; 229:105233.

PMID: 35917612 PMC: 9530017. DOI: 10.1016/j.cognition.2022.105233.

References

Hirayama J, Yoshimoto J, Ishii S . Bayesian representation learning in the cortex regulated by acetylcholine. Neural Netw. 2004; 17(10):1391-400. DOI: 10.1016/j.neunet.2004.06.006. View

Yu A, Cohen J . Sequential effects: Superstition or rational behavior?. Adv Neural Inf Process Syst. 2015; 21:1873-1880. PMC: 4580342. View

Jepma M, Nieuwenhuis S . Pupil diameter predicts changes in the exploration-exploitation trade-off: evidence for the adaptive gain theory. J Cogn Neurosci. 2010; 23(7):1587-96. DOI: 10.1162/jocn.2010.21548. View

Payzan-LeNestour E, Bossaerts P . Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput Biol. 2011; 7(1):e1001048. PMC: 3024253. DOI: 10.1371/journal.pcbi.1001048. View

Cavanagh J, Figueroa C, Cohen M, Frank M . Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb Cortex. 2011; 22(11):2575-86. PMC: 4296208. DOI: 10.1093/cercor/bhr332. View

Hertwig R, Barron G, Weber E, Erev I . Decisions from experience and the effect of rare events in risky choice. Psychol Sci. 2004; 15(8):534-9. DOI: 10.1111/j.0956-7976.2004.00715.x. View

Yoshida W, Ishii S . Resolution of uncertainty in prefrontal cortex. Neuron. 2006; 50(5):781-9. DOI: 10.1016/j.neuron.2006.05.006. View

Rutishauser U, Mamelak A, Schuman E . Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complex. Neuron. 2006; 49(6):805-13. DOI: 10.1016/j.neuron.2006.02.015. View

Behrens T, Woolrich M, Walton M, Rushworth M . Learning the value of information in an uncertain world. Nat Neurosci. 2007; 10(9):1214-21. DOI: 10.1038/nn1954. View

10.

Yu A, Dayan P . Uncertainty, neuromodulation, and attention. Neuron. 2005; 46(4):681-92. DOI: 10.1016/j.neuron.2005.04.026. View

11.

Hsu M, Bhatt M, Adolphs R, Tranel D, Camerer C . Neural systems responding to degrees of uncertainty in human decision-making. Science. 2005; 310(5754):1680-3. DOI: 10.1126/science.1115327. View

12.

Sugrue L, Corrado G, Newsome W . Matching behavior and the representation of value in the parietal cortex. Science. 2004; 304(5678):1782-7. DOI: 10.1126/science.1094765. View

13.

Daw N, ODoherty J, Dayan P, Seymour B, Dolan R . Cortical substrates for exploratory decisions in humans. Nature. 2006; 441(7095):876-9. PMC: 2635947. DOI: 10.1038/nature04766. View

14.

Trommershauser J, Maloney L, Landy M . Decision making, movement planning and statistical decision theory. Trends Cogn Sci. 2008; 12(8):291-7. PMC: 2678412. DOI: 10.1016/j.tics.2008.04.010. View

15.

Levy I, Snell J, Nelson A, Rustichini A, Glimcher P . Neural representation of subjective value under risk and ambiguity. J Neurophysiol. 2009; 103(2):1036-47. DOI: 10.1152/jn.00853.2009. View

16.

Huettel S, Stowe C, Gordon E, Warner B, Platt M . Neural signatures of economic preferences for risk and ambiguity. Neuron. 2006; 49(5):765-75. DOI: 10.1016/j.neuron.2006.01.024. View

17.

Badre D, Doll B, Long N, Frank M . Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron. 2012; 73(3):595-607. PMC: 3285405. DOI: 10.1016/j.neuron.2011.12.025. View

18.

Doya K . Modulators of decision making. Nat Neurosci. 2008; 11(4):410-6. DOI: 10.1038/nn2077. View

19.

Kakade S, Dayan P . Dopamine: generalization and bonuses. Neural Netw. 2002; 15(4-6):549-59. DOI: 10.1016/s0893-6080(02)00048-5. View

20.

Frank M, Doll B, Oas-Terpstra J, Moreno F . Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci. 2009; 12(8):1062-8. PMC: 3062477. DOI: 10.1038/nn.2342. View