What is Changing When: Decoding Visual Information in Movies from Human Intracranial Recordings

Overview

Journal Neuroimage

Specialty Radiology

Date 2017 Aug 22

PMID 28823828

Citations 6

Authors

Leyla Isik

Jedediah Singer

Joseph R Madsen

Nancy Kanwisher

Gabriel Kreiman

Affiliations

Soon will be listed here.

Abstract

The majority of visual recognition studies have focused on the neural responses to repeated presentations of static stimuli with abrupt and well-defined onset and offset times. In contrast, natural vision involves unique renderings of visual inputs that are continuously changing without explicitly defined temporal transitions. Here we considered commercial movies as a coarse proxy to natural vision. We recorded intracranial field potential signals from 1,284 electrodes implanted in 15 patients with epilepsy while the subjects passively viewed commercial movies. We could rapidly detect large changes in the visual inputs within approximately 100 ms of their occurrence, using exclusively field potential signals from ventral visual cortical areas including the inferior temporal gyrus and inferior occipital gyrus. Furthermore, we could decode the content of those visual changes even in a single movie presentation, generalizing across the wide range of transformations present in a movie. These results present a methodological framework for studying cognition during dynamic and natural vision.

Citing Articles

Semantic novelty modulates neural responses to visual change across the human brain.

Nentwich M, Leszczynski M, Russ B, Hirsch L, Markowitz N, Sapru K Nat Commun. 2023; 14(1):2910.

PMID: 37217478 PMC: 10203305. DOI: 10.1038/s41467-023-38576-5.

Modulation of Spectral Representation and Connectivity Patterns in Response to Visual Narrative in the Human Brain.

Sabra Z, Alawieh A, Bonilha L, Naselaris T, AuYong N Front Hum Neurosci. 2022; 16:886938.

PMID: 36277048 PMC: 9582122. DOI: 10.3389/fnhum.2022.886938.

Neurons detect cognitive boundaries to structure episodic memories in humans.

Zheng J, Schjetnan A, Yebra M, Gomes B, Mosher C, Kalia S Nat Neurosci. 2022; 25(3):358-368.

PMID: 35260859 PMC: 8966433. DOI: 10.1038/s41593-022-01020-w.

Growing evidence for separate neural mechanisms for attention and consciousness.

Maier A, Tsuchiya N Atten Percept Psychophys. 2020; 83(2):558-576.

PMID: 33034851 PMC: 7886945. DOI: 10.3758/s13414-020-02146-4.

Human stereoEEG recordings reveal network dynamics of decision-making in a rule-switching task.

Ter Wal M, Platonov A, Cardellicchio P, Pelliccia V, LoRusso G, Sartori I Nat Commun. 2020; 11(1):3075.

PMID: 32555174 PMC: 7300004. DOI: 10.1038/s41467-020-16854-w.

References

Destrieux C, Fischl B, Dale A, Halgren E . Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage. 2010; 53(1):1-15. PMC: 2937159. DOI: 10.1016/j.neuroimage.2010.06.010. View

Rust N, Movshon J . In praise of artifice. Nat Neurosci. 2005; 8(12):1647-50. DOI: 10.1038/nn1606. View

Grill-Spector K, Malach R . The human visual cortex. Annu Rev Neurosci. 2004; 27:649-77. DOI: 10.1146/annurev.neuro.27.070203.144220. View

Conroy B, Singer B, Guntupalli J, Ramadge P, Haxby J . Inter-subject alignment of human cortical anatomy using functional connectivity. Neuroimage. 2013; 81:400-411. PMC: 3729877. DOI: 10.1016/j.neuroimage.2013.05.009. View

Hung C, Kreiman G, Poggio T, DiCarlo J . Fast readout of object identity from macaque inferior temporal cortex. Science. 2005; 310(5749):863-6. DOI: 10.1126/science.1117593. View

Fiser J, Chiu C, Weliky M . Small modulation of ongoing cortical dynamics by sensory input during natural vision. Nature. 2004; 431(7008):573-8. DOI: 10.1038/nature02907. View

Logothetis N, Sheinberg D . Visual object recognition. Annu Rev Neurosci. 1996; 19:577-621. DOI: 10.1146/annurev.ne.19.030196.003045. View

Bansal A, Singer J, Anderson W, Golby A, Madsen J, Kreiman G . Temporal stability of visually selective responses in intracranial field potentials recorded from human occipital and temporal lobes. J Neurophysiol. 2012; 108(11):3073-86. PMC: 3544863. DOI: 10.1152/jn.00458.2012. View

Felsen G, Dan Y . A natural approach to studying vision. Nat Neurosci. 2005; 8(12):1643-6. DOI: 10.1038/nn1608. View

10.

Privman E, Fisch L, Neufeld M, Kramer U, Kipervasser S, Andelman F . Antagonistic relationship between gamma power and visual evoked potentials revealed in human visual cortex. Cereb Cortex. 2010; 21(3):616-24. DOI: 10.1093/cercor/bhq128. View

11.

Vinje W, Gallant J . Sparse coding and decorrelation in primary visual cortex during natural vision. Science. 2000; 287(5456):1273-6. DOI: 10.1126/science.287.5456.1273. View

12.

Huth A, Nishimoto S, Vu A, Gallant J . A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron. 2012; 76(6):1210-24. PMC: 3556488. DOI: 10.1016/j.neuron.2012.10.014. View

13.

Montemurro M, Rasch M, Murayama Y, Logothetis N, Panzeri S . Phase-of-firing coding of natural visual stimuli in primary visual cortex. Curr Biol. 2008; 18(5):375-80. DOI: 10.1016/j.cub.2008.02.023. View

14.

Keysers C, Xiao D, Foldiak P, Perrett D . The speed of sight. J Cogn Neurosci. 2001; 13(1):90-101. DOI: 10.1162/089892901564199. View

15.

Ojemann G . Treatment of temporal lobe epilepsy. Annu Rev Med. 1997; 48:317-28. DOI: 10.1146/annurev.med.48.1.317. View

16.

Lei Y, Sun N, Wilson F, Wang X, Chen N, Yang J . Telemetric recordings of single neuron activity and visual scenes in monkeys walking in an open field. J Neurosci Methods. 2004; 135(1-2):35-41. DOI: 10.1016/j.jneumeth.2003.11.014. View

17.

Nishimoto S, Vu A, Naselaris T, Benjamini Y, Yu B, Gallant J . Reconstructing visual experiences from brain activity evoked by natural movies. Curr Biol. 2011; 21(19):1641-6. PMC: 3326357. DOI: 10.1016/j.cub.2011.08.031. View

18.

Tanaka K . Inferotemporal cortex and object vision. Annu Rev Neurosci. 1996; 19:109-39. DOI: 10.1146/annurev.ne.19.030196.000545. View

19.

Booth M, Rolls E . View-invariant representations of familiar objects by neurons in the inferior temporal visual cortex. Cereb Cortex. 1998; 8(6):510-23. DOI: 10.1093/cercor/8.6.510. View

20.

Bartels A, Zeki S . Brain dynamics during natural viewing conditions--a new guide for mapping connectivity in vivo. Neuroimage. 2005; 24(2):339-49. DOI: 10.1016/j.neuroimage.2004.08.044. View