Identifying Natural Images from Human Brain Activity

Overview

Journal Nature

Specialty Science

Date 2008 Mar 7

PMID 18322462

Citations 439

Authors

Kendrick N Kay

Thomas Naselaris

Ryan J Prenger

Jack L Gallant

Affiliations

Soon will be listed here.

Abstract

A challenging goal in neuroscience is to be able to read out, or decode, mental content from brain activity. Recent functional magnetic resonance imaging (fMRI) studies have decoded orientation, position and object category from activity in visual cortex. However, these studies typically used relatively simple stimuli (for example, gratings) or images drawn from fixed categories (for example, faces, houses), and decoding was based on previous measurements of brain activity evoked by those same stimuli or categories. To overcome these limitations, here we develop a decoding method based on quantitative receptive-field models that characterize the relationship between visual stimuli and fMRI activity in early visual areas. These models describe the tuning of individual voxels for space, orientation and spatial frequency, and are estimated directly from responses evoked by natural images. We show that these receptive-field models make it possible to identify, from a large set of completely novel natural images, which specific image was seen by an observer. Identification is not a mere consequence of the retinotopic organization of visual areas; simpler receptive-field models that describe only spatial tuning yield much poorer identification performance. Our results suggest that it may soon be possible to reconstruct a picture of a person's visual experience from measurements of brain activity alone.

Citing Articles

Decoding auditory working memory content from EEG aftereffects of auditory-cortical TMS.

Uluc I, Daneshzand M, Jas M, Kotlarz P, Lankinen K, Fiedler J bioRxiv. 2025; .

PMID: 39975364 PMC: 11838191. DOI: 10.1101/2024.03.04.583379.

Compression-enabled interpretability of voxelwise encoding models.

Kamali F, Suratgar A, Menhaj M, Abbasi-Asl R PLoS Comput Biol. 2025; 21(2):e1012822.

PMID: 39970189 PMC: 11867343. DOI: 10.1371/journal.pcbi.1012822.

Improved image reconstruction from brain activity through automatic image captioning.

Kalantari F, Faez K, Amindavar H, Nazari S Sci Rep. 2025; 15(1):4907.

PMID: 39930076 PMC: 11811215. DOI: 10.1038/s41598-025-89242-3.

Ultra high density imaging arrays in diffuse optical tomography for human brain mapping improve image quality and decoding performance.

Markow Z, Trobaugh J, Richter E, Tripathy K, Rafferty S, Svoboda A Sci Rep. 2025; 15(1):3175.

PMID: 39863633 PMC: 11762274. DOI: 10.1038/s41598-025-85858-7.

Multisensory naturalistic decoding with high-density diffuse optical tomography.

Tripathy K, Markow Z, Fogarty M, Schroeder M, Svoboda A, Eggebrecht A Neurophotonics. 2025; 12(1):015002.

PMID: 39850351 PMC: 11755382. DOI: 10.1117/1.NPh.12.1.015002.

References

Hansen K, David S, Gallant J . Parametric reverse correlation reveals spatial linearity of retinotopic human V1 BOLD response. Neuroimage. 2004; 23(1):233-41. DOI: 10.1016/j.neuroimage.2004.05.012. View

Heeger D, Ress D . What does fMRI tell us about neuronal activity?. Nat Rev Neurosci. 2002; 3(2):142-51. DOI: 10.1038/nrn730. View

Haynes J, Rees G . Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat Neurosci. 2005; 8(5):686-91. DOI: 10.1038/nn1445. View

Olman C, Ugurbil K, Schrater P, Kersten D . BOLD fMRI and psychophysical measurements of contrast response to broadband images. Vision Res. 2004; 44(7):669-83. DOI: 10.1016/j.visres.2003.10.022. View

Smith A, Singh K, Williams A, Greenlee M . Estimating receptive field size from fMRI data in human striate and extrastriate visual cortex. Cereb Cortex. 2001; 11(12):1182-90. DOI: 10.1093/cercor/11.12.1182. View

Daugman J . Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. J Opt Soc Am A. 1985; 2(7):1160-9. DOI: 10.1364/josaa.2.001160. View

Engel S, Rumelhart D, Wandell B, Lee A, Glover G, Chichilnisky E . fMRI of human visual cortex. Nature. 1994; 369(6481):525. DOI: 10.1038/369525a0. View

Dumoulin S, Wandell B . Population receptive field estimates in human visual cortex. Neuroimage. 2007; 39(2):647-60. PMC: 3073038. DOI: 10.1016/j.neuroimage.2007.09.034. View

Haynes J, Rees G . Predicting the stream of consciousness from activity in human visual cortex. Curr Biol. 2005; 15(14):1301-7. DOI: 10.1016/j.cub.2005.06.026. View

10.

Simoncelli E, Olshausen B . Natural image statistics and neural representation. Annu Rev Neurosci. 2001; 24:1193-216. DOI: 10.1146/annurev.neuro.24.1.1193. View

11.

Haxby J, Gobbini M, Furey M, Ishai A, SCHOUTEN J, Pietrini P . Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 2001; 293(5539):2425-30. DOI: 10.1126/science.1063736. View

12.

Stanley G, Li F, Dan Y . Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. J Neurosci. 1999; 19(18):8036-42. PMC: 6782475. View

13.

Tsao D, Freiwald W, Tootell R, Livingstone M . A cortical region consisting entirely of face-selective cells. Science. 2006; 311(5761):670-4. PMC: 2678572. DOI: 10.1126/science.1119983. View

14.

Wu M, David S, Gallant J . Complete functional characterization of sensory neurons by system identification. Annu Rev Neurosci. 2006; 29:477-505. DOI: 10.1146/annurev.neuro.29.051605.113024. View

15.

Cox D, Savoy R . Functional magnetic resonance imaging (fMRI) "brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage. 2003; 19(2 Pt 1):261-70. DOI: 10.1016/s1053-8119(03)00049-1. View

16.

Zhang K, Ginzburg I, McNaughton B, Sejnowski T . Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells. J Neurophysiol. 1998; 79(2):1017-44. DOI: 10.1152/jn.1998.79.2.1017. View

17.

Haynes J, Lotto R, Rees G . Responses of human visual cortex to uniform surfaces. Proc Natl Acad Sci U S A. 2004; 101(12):4286-91. PMC: 384733. DOI: 10.1073/pnas.0307948101. View

18.

Sereno M, Dale A, Reppas J, Kwong K, Belliveau J, Brady T . Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science. 1995; 268(5212):889-93. DOI: 10.1126/science.7754376. View

19.

Kamitani Y, Tong F . Decoding the visual and subjective contents of the human brain. Nat Neurosci. 2005; 8(5):679-85. PMC: 1808230. DOI: 10.1038/nn1444. View

20.

Haynes J, Rees G . Decoding mental states from brain activity in humans. Nat Rev Neurosci. 2006; 7(7):523-34. DOI: 10.1038/nrn1931. View