Violins are Green, Pianos are Blue: Cross-modal Sound-to-Sight Associations with Timbre in Synesthetes & Non-Synesthetes
56.427, Tuesday, May 14, 2:45 - 6:45 pm, Orchid Ballroom
William Griscom1, Stephen Palmer1; 1University of California, Berkeley
Previous research has shown that auditory-visual cross-modal associations for complex music in non-synesthetes appear to be mediated by the emotional connotations of the stimuli (Palmer & Schloss, in review), rather than (or in addition to) exhibiting a direct sensory correspondence, as has been found for more basic auditory features such as pitch or loudness (Marks, 1978). In this study, we investigated a class of musical stimuli of intermediate complexity: namely, instrumental timbres. In a series of experiments, participants indicated what colors, edge contrasts, and onset dynamics in visual displays "went best" with the sound of 17 different common musical instruments (e.g., violin, piano, marimba). We found evidence for emotional mediation in non-synesthetes color choices, in that participants tended to choose colors whose emotional quality was similar to the rated emotional quality of the sound (e.g., "happy" looking colors, such as saturated red, were chosen for "happy" sounding instruments, such as a harpsichord). Systematic sound-to-sight correspondences were also evident for timbral attack and decay time in non-synesthetes choices of "best fitting" edge contrasts (low to high spatial frequencies) and dynamic onsets (temporal dynamics of contrast) of circular shapes. In contrast, timbre-color synesthetes, who chose the colors most similar to the colors they experienced to the sounds, reported color experiences qualitatively different from those made by non-synesthetes. In particular synesthetes showed consistently weaker effects of emotional mediation and greater effects of low-level perceptual features. These results indicate that, although both groups give consistent patterns of responses, synesthetic color experiences to instrumental sounds appear to be driven more strongly by sensory-perceptual features and less strongly by semantic and emotional factors than is typical of non-synesthetic color associations to those same sounds.