Modeling Short Visual Events Through the BOLD Moments Video FMRI Dataset and Metadata

Overview

Journal Nat Commun

Specialty Biology

Date 2024 Jul 24

PMID 39048577

Authors

Benjamin Lahner

Kshitij Dwivedi

Polina Iamshchinina

Monika Graumann

Alex Lascelles

Gemma Roig

Alessandro Thomas Gifford

Bowen Pan

SouYoung Jin

N Apurva Ratan Murty

Kendrick Kay

Aude Oliva

Radoslaw Cichy

Affiliations

Soon will be listed here.

Abstract

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.

Citing Articles

An fMRI dataset in response to large-scale short natural dynamic facial expression videos.

Chen P, Zhang C, Li B, Tong L, Wang L, Ma S Sci Data. 2024; 11(1):1247.

PMID: 39562568 PMC: 11576863. DOI: 10.1038/s41597-024-04088-0.

Principles of intensive human neuroimaging.

Kupers E, Knapen T, Merriam E, Kay K Trends Neurosci. 2024; 47(11):856-864.

PMID: 39455343 PMC: 11563852. DOI: 10.1016/j.tins.2024.09.011.

Hierarchical organization of social action features along the lateral visual pathway.

McMahon E, Bonner M, Isik L Curr Biol. 2023; 33(23):5035-5047.e8.

PMID: 37918399 PMC: 10841461. DOI: 10.1016/j.cub.2023.10.015.

References

Hardwick R, Caspers S, Eickhoff S, Swinnen S . Neural correlates of action: Comparing meta-analyses of imagery, observation, and execution. Neurosci Biobehav Rev. 2018; 94:31-44. DOI: 10.1016/j.neubiorev.2018.08.003. View

Senior C, Barnes J, Giampietro V, Simmons A, Bullmore E, Brammer M . The functional neuroanatomy of implicit-motion perception or representational momentum. Curr Biol. 2000; 10(1):16-22. DOI: 10.1016/s0960-9822(99)00259-6. View

Murray J, Bernacchia A, Freedman D, Romo R, Wallis J, Cai X . A hierarchy of intrinsic timescales across primate cortex. Nat Neurosci. 2014; 17(12):1661-3. PMC: 4241138. DOI: 10.1038/nn.3862. View

Schrimpf M, Kubilius J, Lee M, Ratan Murty N, Ajemian R, DiCarlo J . Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence. Neuron. 2020; 108(3):413-423. DOI: 10.1016/j.neuron.2020.07.040. View

Schneider W . Selective visual processing across competition episodes: a theory of task-driven visual attention and working memory. Philos Trans R Soc Lond B Biol Sci. 2013; 368(1628):20130060. PMC: 3758203. DOI: 10.1098/rstb.2013.0060. View

Haxby J . Multivariate pattern analysis of fMRI: the early beginnings. Neuroimage. 2012; 62(2):852-5. PMC: 3389290. DOI: 10.1016/j.neuroimage.2012.03.016. View

Wurm M, Caramazza A, Lingnau A . Action Categories in Lateral Occipitotemporal Cortex Are Organized Along Sociality and Transitivity. J Neurosci. 2017; 37(3):562-575. PMC: 6596756. DOI: 10.1523/JNEUROSCI.1717-16.2016. View

Bonner M, Epstein R . Object representations in the human brain reflect the co-occurrence statistics of vision and language. Nat Commun. 2021; 12(1):4081. PMC: 8253839. DOI: 10.1038/s41467-021-24368-2. View

Han J, Chen C, Shao L, Hu X, Han J, Liu T . Learning Computational Models of Video Memorability from fMRI Brain Imaging. IEEE Trans Cybern. 2014; 45(8):1692-703. DOI: 10.1109/TCYB.2014.2358647. View

10.

Grill-Spector K, Kourtzi Z, Kanwisher N . The lateral occipital complex and its role in object recognition. Vision Res. 2001; 41(10-11):1409-22. DOI: 10.1016/s0042-6989(01)00073-6. View

11.

Kriegeskorte N, Mur M, Bandettini P . Representational similarity analysis - connecting the branches of systems neuroscience. Front Syst Neurosci. 2008; 2:4. PMC: 2605405. DOI: 10.3389/neuro.06.004.2008. View

12.

Wurm M, Caramazza A . Two 'what' pathways for action and object recognition. Trends Cogn Sci. 2021; 26(2):103-116. DOI: 10.1016/j.tics.2021.10.003. View

13.

Rajalingham R, Issa E, Bashivan P, Kar K, Schmidt K, DiCarlo J . Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks. J Neurosci. 2018; 38(33):7255-7269. PMC: 6096043. DOI: 10.1523/JNEUROSCI.0388-18.2018. View

14.

Pascual-Leone A, Walsh V . Fast backprojections from the motion to the primary visual area necessary for visual awareness. Science. 2001; 292(5516):510-2. DOI: 10.1126/science.1057099. View

15.

Khosla M, Ratan Murty N, Kanwisher N . A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Curr Biol. 2022; 32(19):4159-4171.e9. PMC: 9561032. DOI: 10.1016/j.cub.2022.08.009. View

16.

Martin A, Chao L . Semantic memory and the brain: structure and processes. Curr Opin Neurobiol. 2001; 11(2):194-201. DOI: 10.1016/s0959-4388(00)00196-3. View

17.

Cohen J, Perlstein W, Braver T, Nystrom L, Noll D, Jonides J . Temporal dynamics of brain activation during a working memory task. Nature. 1997; 386(6625):604-8. DOI: 10.1038/386604a0. View

18.

Julian J, Fedorenko E, Webster J, Kanwisher N . An algorithmic method for functionally defining regions of interest in the ventral visual pathway. Neuroimage. 2012; 60(4):2357-64. DOI: 10.1016/j.neuroimage.2012.02.055. View

19.

Wittkuhn L, Schuck N . Dynamics of fMRI patterns reflect sub-second activation sequences and reveal replay in human visual cortex. Nat Commun. 2021; 12(1):1795. PMC: 7979874. DOI: 10.1038/s41467-021-21970-2. View

20.

Chang N, Pyles J, Marcus A, Gupta A, Tarr M, Aminoff E . BOLD5000, a public fMRI dataset while viewing 5000 visual images. Sci Data. 2019; 6(1):49. PMC: 6502931. DOI: 10.1038/s41597-019-0052-3. View