Computational Mechanisms Underlying Latent Value Updating of Unchosen Actions

Overview

Journal Sci Adv

Specialties Biology
Science

Date 2023 Oct 20

PMID 37862419

Authors

Ido Ben-Artzi

Yoav Kessler

Bruno Nicenboim

Nitzan Shahar

Affiliations

Soon will be listed here.

Abstract

Current studies suggest that individuals estimate the value of their choices based on observed feedback. Here, we ask whether individuals also update the value of their unchosen actions, even when the associated feedback remains unknown. One hundred seventy-eight individuals completed a multi-armed bandit task, making choices to gain rewards. We found robust evidence suggesting latent value updating of unchosen actions based on the chosen action's outcome. Computational modeling results suggested that this effect is mainly explained by a value updating mechanism whereby individuals integrate the outcome history for choosing an option with that of rejecting the alternative. Properties of the deliberation (i.e., duration/difficulty) did not moderate the latent value updating of unchosen actions, suggesting that memory traces generated during deliberation might take a smaller role in this specific phenomenon than previously thought. We discuss the mechanisms facilitating credit assignment to unchosen actions and their implications for human decision-making.

Citing Articles

The more random people's preference judgments are, the more they explore in gambling tasks.

Zhu J, Katahira K, Hirakawa M, Nakao T BMC Psychol. 2024; 12(1):766.

PMID: 39707509 PMC: 11662857. DOI: 10.1186/s40359-024-02252-0.

Cognitive psychology: computing the value of the choices we do not make.

Brazil I Commun Psychol. 2024; 2(1):17.

PMID: 39242742 PMC: 11332109. DOI: 10.1038/s44271-024-00064-x.

From Tripping and Falling to Ruminating and Worrying: A Meta-Control Account of Repetitive Negative Thinking.

Hitchcock P, Frank M Curr Opin Behav Sci. 2024; 56.

PMID: 39130377 PMC: 11314892. DOI: 10.1016/j.cobeha.2024.101356.

Flexible gating between subspaces in a neural network model of internally guided task switching.

Liu Y, Wang X Nat Commun. 2024; 15(1):6497.

PMID: 39090084 PMC: 11294624. DOI: 10.1038/s41467-024-50501-y.

Value Estimation versus Effort Mobilization: A General Dissociation between Ventromedial and Dorsomedial Prefrontal Cortex.

Clairis N, Pessiglione M J Neurosci. 2024; 44(17).

PMID: 38514180 PMC: 11044108. DOI: 10.1523/JNEUROSCI.1176-23.2024.

References

Bavard S, Palminteri S . Why unchosen options linger in our minds. Commun Biol. 2021; 4(1):1271. PMC: 8575880. DOI: 10.1038/s42003-021-02803-w. View

Stine G, Trautmann E, Jeurissen D, Shadlen M . A neural mechanism for terminating decisions. Neuron. 2023; 111(16):2601-2613.e5. PMC: 10565788. DOI: 10.1016/j.neuron.2023.05.028. View

Chambon V, Thero H, Vidal M, Vandendriessche H, Haggard P, Palminteri S . Information about action outcomes differentially affects learning from self-determined versus imposed choices. Nat Hum Behav. 2020; 4(10):1067-1079. DOI: 10.1038/s41562-020-0919-5. View

Gronau Q, Wagenmakers E . Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection. Comput Brain Behav. 2019; 2(1):1-11. PMC: 6400414. DOI: 10.1007/s42113-018-0011-7. View

Palminteri S, Lebreton M . The computational roots of positivity and confirmation biases in reinforcement learning. Trends Cogn Sci. 2022; 26(7):607-621. DOI: 10.1016/j.tics.2022.04.005. View

Gluth S, Kern N, Kortmann M, Vitali C . Value-based attention but not divisive normalization influences decisions with multiple alternatives. Nat Hum Behav. 2020; 4(6):634-645. PMC: 7306407. DOI: 10.1038/s41562-020-0822-0. View

Bayer H, Glimcher P . Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron. 2005; 47(1):129-41. PMC: 1564381. DOI: 10.1016/j.neuron.2005.05.020. View

Troudart Y, Shahar N . Formation of non-veridical action-outcome associations following exposure to threat-related cues. Emotion. 2023; 23(7):2094-2099. DOI: 10.1037/emo0001212. View

Biderman N, Gershman S, Shohamy D . The role of memory in counterfactual valuation. J Exp Psychol Gen. 2023; 152(6):1754-1767. PMC: 10272005. DOI: 10.1037/xge0001364. View

10.

Roese N . Counterfactual thinking. Psychol Bull. 1997; 121(1):133-48. DOI: 10.1037/0033-2909.121.1.133. View

11.

Louie K, Grattan L, Glimcher P . Reward value-based gain control: divisive normalization in parietal cortex. J Neurosci. 2011; 31(29):10627-39. PMC: 3285508. DOI: 10.1523/JNEUROSCI.1237-11.2011. View

12.

Kishida K, Saez I, Lohrenz T, Witcher M, Laxton A, Tatter S . Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward. Proc Natl Acad Sci U S A. 2015; 113(1):200-5. PMC: 4711839. DOI: 10.1073/pnas.1513619112. View

13.

Collins A, Frank M . Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol Rev. 2014; 121(3):337-66. DOI: 10.1037/a0037015. View

14.

Wang S, Feng S, Bornstein A . Mixing memory and desire: How memory reactivation supports deliberative decision-making. Wiley Interdiscip Rev Cogn Sci. 2021; 13(2):e1581. DOI: 10.1002/wcs.1581. View

15.

Maoz U, Yaffe G, Koch C, Mudrik L . Neural precursors of decisions that matter-an ERP study of deliberate and arbitrary choice. Elife. 2019; 8. PMC: 6809608. DOI: 10.7554/eLife.39787. View

16.

Wimmer G, Shohamy D . Preference by association: how memory mechanisms in the hippocampus bias decisions. Science. 2012; 338(6104):270-3. DOI: 10.1126/science.1223252. View

17.

Heitz R . The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Front Neurosci. 2014; 8:150. PMC: 4052662. DOI: 10.3389/fnins.2014.00150. View

18.

Lind M, Visentini M, Mantyla T, Missier F . Choice-Supportive Misremembering: A New Taxonomy and Review. Front Psychol. 2017; 8:2062. PMC: 5723021. DOI: 10.3389/fpsyg.2017.02062. View

19.

Voigt K, Murawski C, Bode S . Endogenous formation of preferences: Choices systematically change willingness-to-pay for goods. J Exp Psychol Learn Mem Cogn. 2017; 43(12):1872-1882. DOI: 10.1037/xlm0000415. View

20.

Wright G, Ayton P . Decision time, subjective probability, and task difficulty. Mem Cognit. 1988; 16(2):176-85. DOI: 10.3758/bf03213487. View