Correlation Detection As a General Mechanism for Multisensory Integration

Overview

Journal Nat Commun

Specialty Biology

Date 2016 Jun 7

PMID 27265526

Citations 55

Authors

Cesare V Parise

Marc O Ernst

Affiliations

Soon will be listed here.

Abstract

The brain efficiently processes multisensory information by selectively combining related signals across the continuous stream of multisensory inputs. To do so, it needs to detect correlation, lag and synchrony across the senses; optimally integrate related information; and dynamically adapt to spatiotemporal conflicts across the senses. Here we show that all these aspects of multisensory perception can be jointly explained by postulating an elementary processing unit akin to the Hassenstein-Reichardt detector-a model originally developed for visual motion perception. This unit, termed the multisensory correlation detector (MCD), integrates related multisensory signals through a set of temporal filters followed by linear combination. Our model can tightly replicate human perception as measured in a series of empirical studies, both novel and previously published. MCDs provide a unified general theory of multisensory processing, which simultaneously explains a wide spectrum of phenomena with a simple, yet physiologically plausible model.

Citing Articles

Multisensory integration operates on correlated input from unimodal transient channels.

Parise C, Ernst M Elife. 2025; 12.

PMID: 39841060 PMC: 11753778. DOI: 10.7554/eLife.90841.

During haptic communication, the central nervous system compensates distinctly for delay and noise.

Eden J, Ivanova E, Burdet E PLoS Comput Biol. 2024; 20(11):e1012037.

PMID: 39504311 PMC: 11573204. DOI: 10.1371/journal.pcbi.1012037.

Multisensory perceptual and causal inference is largely preserved in medicated post-acute individuals with schizophrenia.

Rohe T, Hesse K, Ehlis A, Noppeney U PLoS Biol. 2024; 22(9):e3002790.

PMID: 39255328 PMC: 11466413. DOI: 10.1371/journal.pbio.3002790.

Independent-channels models of temporal-order judgment revisited: A model comparison.

Kelber P, Ulrich R Atten Percept Psychophys. 2024; 86(6):2187-2209.

PMID: 39107652 PMC: 11410913. DOI: 10.3758/s13414-024-02915-5.

Multi-timescale neural dynamics for multisensory integration.

Senkowski D, Engel A Nat Rev Neurosci. 2024; 25(9):625-642.

PMID: 39090214 DOI: 10.1038/s41583-024-00845-7.

References

Spence C . Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule. Ann N Y Acad Sci. 2013; 1296:31-49. DOI: 10.1111/nyas.12121. View

Ernst M, Bulthoff H . Merging the senses into a robust percept. Trends Cogn Sci. 2004; 8(4):162-9. DOI: 10.1016/j.tics.2004.02.002. View

Shams L, Beierholm U . Causal inference in perception. Trends Cogn Sci. 2010; 14(9):425-32. DOI: 10.1016/j.tics.2010.07.001. View

Cai M, Stetson C, Eagleman D . A neural model for temporal order judgments and their active recalibration: a common mechanism for space and time?. Front Psychol. 2012; 3:470. PMC: 3487422. DOI: 10.3389/fpsyg.2012.00470. View

Carandini M, Heeger D . Normalization as a canonical neural computation. Nat Rev Neurosci. 2011; 13(1):51-62. PMC: 3273486. DOI: 10.1038/nrn3136. View

Denison R, Driver J, Ruff C . Temporal structure and complexity affect audio-visual correspondence detection. Front Psychol. 2013; 3:619. PMC: 3550803. DOI: 10.3389/fpsyg.2012.00619. View

Ma W, Beck J, Latham P, Pouget A . Bayesian inference with probabilistic population codes. Nat Neurosci. 2006; 9(11):1432-8. DOI: 10.1038/nn1790. View

Ohshiro T, Angelaki D, DeAngelis G . A normalization model of multisensory integration. Nat Neurosci. 2011; 14(6):775-82. PMC: 3102778. DOI: 10.1038/nn.2815. View

Behnia R, Clark D, Carter A, Clandinin T, Desplan C . Processing properties of ON and OFF pathways for Drosophila motion detection. Nature. 2014; 512(7515):427-30. PMC: 4243710. DOI: 10.1038/nature13427. View

10.

Stein B, Stanford T . Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci. 2008; 9(4):255-66. DOI: 10.1038/nrn2331. View

11.

Adelson E, Bergen J . Spatiotemporal energy models for the perception of motion. J Opt Soc Am A. 1985; 2(2):284-99. DOI: 10.1364/josaa.2.000284. View

12.

Knill D, Pouget A . The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 2004; 27(12):712-9. DOI: 10.1016/j.tins.2004.10.007. View

13.

Burr D, Silva O, Cicchini G, Banks M, Morrone M . Temporal mechanisms of multimodal binding. Proc Biol Sci. 2009; 276(1663):1761-9. PMC: 2674495. DOI: 10.1098/rspb.2008.1899. View

14.

Fujisaki W, Nishida S . Feature-based processing of audio-visual synchrony perception revealed by random pulse trains. Vision Res. 2007; 47(8):1075-93. DOI: 10.1016/j.visres.2007.01.021. View

15.

Chen L, Vroomen J . Intersensory binding across space and time: a tutorial review. Atten Percept Psychophys. 2013; 75(5):790-811. DOI: 10.3758/s13414-013-0475-4. View

16.

Slutsky D, Recanzone G . Temporal and spatial dependency of the ventriloquism effect. Neuroreport. 2001; 12(1):7-10. DOI: 10.1097/00001756-200101220-00009. View

17.

Fetsch C, DeAngelis G, Angelaki D . Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat Rev Neurosci. 2013; 14(6):429-42. PMC: 3820118. DOI: 10.1038/nrn3503. View

18.

Lewis P, Miall R . Distinct systems for automatic and cognitively controlled time measurement: evidence from neuroimaging. Curr Opin Neurobiol. 2003; 13(2):250-5. DOI: 10.1016/s0959-4388(03)00036-9. View

19.

Raposo D, Sheppard J, Schrater P, Churchland A . Multisensory decision-making in rats and humans. J Neurosci. 2012; 32(11):3726-35. PMC: 3335889. DOI: 10.1523/JNEUROSCI.4998-11.2012. View

20.

JEFFRESS L . A place theory of sound localization. J Comp Physiol Psychol. 1948; 41(1):35-9. DOI: 10.1037/h0061495. View