Natural Cross-modal Mappings Between Visual and Auditory Features

Overview

Journal J Vis

Specialty Ophthalmology

Date 2010 Feb 11

PMID 20143899

Citations 109

Authors

Karla K Evans

Anne Treisman

Affiliations

Soon will be listed here.

Abstract

The brain may combine information from different sense modalities to enhance the speed and accuracy of detection of objects and events, and the choice of appropriate responses. There is mounting evidence that perceptual experiences that appear to be modality-specific are also influenced by activity from other sensory modalities, even in the absence of awareness of this interaction. In a series of speeded classification tasks, we found spontaneous mappings between the auditory feature of pitch and the visual features of vertical location, size, and spatial frequency but not contrast. By dissociating the task variables from the features that were cross-modally related, we find that the interactions happen in an automatic fashion and are possibly located at the perceptual level.

Citing Articles

Development of color learning protocol based on music-color association for people with visual impairment.

Park H Ann Med. 2025; 57(1):2476728.

PMID: 40059778 PMC: 11894748. DOI: 10.1080/07853890.2025.2476728.

Auditory and tactile frequency mapping for visual distance perception: A step forward in sensory substitution and augmentation.

Jiang P, Rossiter J, Kent C PLoS One. 2025; 20(3):e0318354.

PMID: 40029888 PMC: 11875370. DOI: 10.1371/journal.pone.0318354.

Invited reply: Comment on 'Left-right asymmetry of the Maxwell spot centroids in adults without and with dyslexia'.

Le Floch A, Ropars G Proc Biol Sci. 2025; 292(2040):20243036.

PMID: 39936338 PMC: 11815583. DOI: 10.1098/rspb.2024.3036.

Color and tone color: audiovisual crossmodal correspondences with musical instrument timbre.

Reymore L, Lindsey D Front Psychol. 2025; 15():1520131.

PMID: 39839933 PMC: 11747214. DOI: 10.3389/fpsyg.2024.1520131.

Acoustic features of instrumental movie soundtracks elicit distinct and mostly non-overlapping extra-musical meanings in the mind of the listener.

Groves K, Farbood M, Carone B, Ripolles P, Zuanazzi A Sci Rep. 2025; 15(1):2327.

PMID: 39825090 PMC: 11748619. DOI: 10.1038/s41598-025-86089-6.

References

Rusconi E, Kwan B, Giordano B, Umilta C, Butterworth B . Spatial representation of pitch height: the SMARC effect. Cognition. 2005; 99(2):113-29. DOI: 10.1016/j.cognition.2005.01.004. View

Patching G, Quinlan P . Garner and congruence effects in the speeded classification of bimodal signals. J Exp Psychol Hum Percept Perform. 2002; 28(4):755-75. View

Gallace A, Spence C . Multisensory synesthetic interactions in the speeded classification of visual size. Percept Psychophys. 2007; 68(7):1191-203. DOI: 10.3758/bf03193720. View

Ben-Artzi E, Marks L . Visual-auditory interaction in speeded classification: role of stimulus difference. Percept Psychophys. 1995; 57(8):1151-62. DOI: 10.3758/bf03208371. View

Marks L, Hammeal R, Bornstein M . Perceiving similarity and comprehending metaphor. Monogr Soc Res Child Dev. 1987; 52(1):1-102. View

Pelli D . The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis. 1997; 10(4):437-42. View

Mondloch C, Maurer D . Do small white balls squeak? Pitch-object correspondences in young children. Cogn Affect Behav Neurosci. 2004; 4(2):133-6. DOI: 10.3758/cabn.4.2.133. View

Martino G, Marks L . Perceptual and linguistic interactions in speeded classification: tests of the semantic coding hypothesis. Perception. 2000; 28(7):903-23. DOI: 10.1068/p2866. View

Marks L . On cross-modal similarity: auditory-visual interactions in speeded discrimination. J Exp Psychol Hum Percept Perform. 1987; 13(3):384-94. DOI: 10.1037//0096-1523.13.3.384. View

10.

Long J . Division of attention between simultaneous and successive nonverbal signals varying in discriminability. Q J Exp Psychol. 1977; 29(1):7-30. DOI: 10.1080/00335557743000008. View

11.

Smith L, Sera M . A developmental analysis of the polar structure of dimensions. Cogn Psychol. 1992; 24(1):99-142. DOI: 10.1016/0010-0285(92)90004-l. View

12.

Melara R . Dimensional interaction between color and pitch. J Exp Psychol Hum Percept Perform. 1989; 15(1):69-79. DOI: 10.1037//0096-1523.15.1.69. View

13.

Marks L . On associations of light and sound: the mediation of brightness, pitch, and loudness. Am J Psychol. 1974; 87(1-2):173-88. View

14.

GREENWALD A, McGhee D, Schwartz J . Measuring individual differences in implicit cognition: the implicit association test. J Pers Soc Psychol. 1998; 74(6):1464-80. DOI: 10.1037//0022-3514.74.6.1464. View

15.

Melara R, Marks L . Processes underlying dimensional interactions: correspondences between linguistic and nonlinguistic dimensions. Mem Cognit. 1990; 18(5):477-95. DOI: 10.3758/bf03198481. View

16.

Lu C, Proctor R . The influence of irrelevant location information on performance: A review of the Simon and spatial Stroop effects. Psychon Bull Rev. 2013; 2(2):174-207. DOI: 10.3758/BF03210959. View

17.

Ramachandran V, Hubbard E . Psychophysical investigations into the neural basis of synaesthesia. Proc Biol Sci. 2001; 268(1470):979-83. PMC: 1088697. DOI: 10.1098/rspb.2000.1576. View

18.

Brainard D . The Psychophysics Toolbox. Spat Vis. 1997; 10(4):433-6. View

19.

Marks L . On cross-modal similarity: the perceptual structure of pitch, loudness, and brightness. J Exp Psychol Hum Percept Perform. 1989; 15(3):586-602. DOI: 10.1037//0096-1523.15.3.586. View

20.

Bernstein I, Edelstein B . Effects of some variations in auditory input upon visual choice reaction time. J Exp Psychol. 1971; 87(2):241-7. DOI: 10.1037/h0030524. View