 |
       |
|
Cross-modal associations from musical timbres and intervals to visual textures
Previous research has provided evidence that cross-modal music-to-color associations
are mediated by emotion, both for classical orchestral music (Palmer et al., 2013) and
for a wide range of genres, from heavy-metal to Hindustani-sitar (Whiteford et al., VSS-
2013). Similar results suggesting emotional mediation have been found using lowerlevel
musical stimuli, including musical melodies (Palmer et al., VSS-2011) and twonote
intervals and instrumental timbres (Griscom & Palmer, VSS-2012). Here we
extended this line of investigation by studying musical associations to another salient
visual domain: line-based geometrical textures. Using analogous experimental
methods, we examined cross-modal associations from instrumental timbres and twonote
musical intervals to visual textures consisting of many similar elements constructed
from lines and/or curves. While listening to one of 16 instrumental timbres (flute,
harpsichord, trombone, violin, etc.) or one of 12 two-note musical intervals (the tonic
paired with the other 12 semitones in a full chromatic octave) participants chose the
three most consistent and three least consistent visual textures from a 7x4 matrix of 28
black-and-white textures. Each subject later rated each timbre, interval, and texture
individually on 13 dimensions: 5 emotional (e.g., happy/sad, angry/not-angry) and 8
geometric (e.g., sharp/smooth, simple/complex). For each dimension (e.g., happy/sad,
sharp/smooth) we computed an index of timbre-texture associations (TTAs) and intervaltexture
associations (ITAs) as a weighted average of the (happy/sad) ratings of the 6
textures chosen as going best and worst with each timbre and each interval. We found
strong correlations between ratings of the timbres and the TTAs of the textures chosen
to go with them for some geometric and emotional dimensions (e.g., sharp/
smooth=+.85, angry/not-angry=+.75). The same was true for two-note intervals for
some dimensions (e.g., simple/complex=+.83; happy/sad=+.83). The results thus
suggest that associations from low-level musical stimuli to visual textures may be
mediated by corresponding dimensions in particular geometric and emotional qualities.
Visual Texture, Music, and Emotion
Previous research indicates that cross-modal music-to-color associations are systematic
in non-synesthetes and are mediated by emotion (e.g., Palmer et al., 2013; Langlois,
VSS-2013; Whiteford et al., VSS-2013). The present research asks whether similarly
systematic associations are evident from music to line-based geometric visual textures,
and, if so, whether they are mediated by emotional and/or geometric effects. We created
28 black-and-white line-based textures that differed on multiple dimensions (e.g., curved/
straight, granular/fibrous, simple/complex, While listening to 34 widely different musical
selections that varied from heavy-metal to Hindustani-sitar, 27 non-synesthetic participants
picked the 3 most-consistent (and later the 3 least consistent) textures for each musical
selection from a 7x4 array of black-and-white textures. Afterward, they rated each musical
selection and each visual texture along 5 emotion dimensions (Happy/Sad, Angry/Not-
Angry, Agitated/Calm, Weak/Strong, Harmonious/Disharmonious), and 8 geometric
dimensions (e.g., Simple/Complex, Sharp/Smooth, Granular/Fibrous). For each dimension
(e.g., Calm/Agitated) we computed an index of music-texture associations (MTAs) as a
weighted average of the (e.g., Calm/Agitated) ratings of the 6 textures chosen as going
best and worst with each musical selection. The results suggest that cross-modal music-
to-texture associations are mediated in part by emotion, because the emotional ratings of
the music were so strongly correlated with the emotional MTAs of the textures that were
chosen as going best with the music (.92 for Angry/Not-angry, .94 for Calm/Agitated, .82
for Active/Passive, .86 for Harmonious/Disharmonious). Unlike music-to-color associations,
however, Happy/Sad was not relevant in music-to-texture associations (r=.07), primarily
because the textures we used did not vary reliably in this emotional dimension. Music-to-
texture associations were also mediated by shared geometric features (e.g., .91 for Simple/
Complex and .87 for Sharp/Smooth), suggesting that these cross-modal mappings may
result from both shared emotional associations and shared non-emotional associations.
Shape-to-Color Associations in Non-synesthetes: Evidence for Emotional Mediation
There is growing evidence that cross-modal music-to-color associations are mediated by emotion in non-synesthetes
(Palmer, Schloss, Xu, Prado-Leon, 2013; Whiteford, Schloss, Palmer, 2013). Here we investigated whether emotion might
also mediate cross-dimensional shape-to-color associations in non-synesthetes (Albertazzi et al., 2012). Experiment 1
tested shape-to-color associations with 44 line stimuli that differed in the number of line segments (2/3/8), kind of edges
(curved/angular), level of closure (open/semi-closed/intersecting-once/intersecting>1) and symmetry (asymmetric/symmetric).
While viewing each stimulus, participants picked the three most consistent (and the three least consistent) among 37 colors.
Later, they also rated each color and each line on 7 bipolar emotional dimensions (sad/happy, calm/agitated, not-angry/angry,
passive/active, weak/strong, safe/harmful, and unpleasant/pleasant). The colors chosen to go with a line were well predicted by
specific perceptual features of the line. In particular, more saturated colors were associated with more closed, angular,
intersecting lines; darker colors were associated with more angular, intersecting lines; redder colors were associated with
more angular, closed lines; and yellower colors were associated with more closed, angular, asymmetric lines. Consistent with
the emotional mediation hypothesis, participants reliably associated colors with lines having similar emotional content for 6
of the 7 emotional dimensions, with correlations ranging from .84 for safe/harmful to .69 for unpleasant/pleasant. Preference
(liked/disliked) also seemed to be related to color choices (r=.58). Principal Components Analysis of the dimensions showed
that 91% of the variance could be explained by 2 components that roughly corresponded to not-angry/angry and sad/happy.
Experiment 2 investigated similar shape-to-color associations for 45 closed geometric shapes that differed in the number
of lines (3/4/9), kind of edges (curved/angular/pointy), concavity (0/1/>1 concavities), and symmetry (0/1/>1 symmetry axes).
Results were similar in that the safe-harmful emotional dimension produced the highest shape-to-color correlations.
Color, Music, Emotion, and Synesthesia Research on cross-modal mappings between music and color in non-synesthetes revealed systematic cross-modal associations and strong evidence for the role of emotion as a mediating factor in these associations. We investigated whether synesthetes (music-color synesthetes as determined by the Eagleman Synesthesia Battery) have similar cross-modal associations between dimensions of music and dimensions of color, and if emotion mediates their cross-modal associations. Synesthetes picked the three colors that were the most similar (and later the least similar) to their synesthetic colors while listening to thirty-two controlled single-line melodies that varied in tonality, tempo, note-density, register, and timbre. They also rated each melody and each color along six emotion dimensions (Happy-Sad, Angry-Not Angry, Agitated-Calm, Active-Passive, and Harmonious-Disharmonious). Results indicated that synesthetes’ cross-modal associations were mediated by emotion to some degree, because of correlations between the emotional ratings of the music and the emotional ratings of the colors the synesthetes chose to match their synesthetic colors on some dimensions (.58 for Happy-Sad, .22 for Active-Passive). In addition, synesthetes tended to choose lighter, yellower, and more saturated colors when matching their synesthetic responses to the melodies in the major mode, and darker, bluer, and less saturated colors when matching their synesthetic responses to the melodies in the minor mode. These cross-modal associations are similar to those of non-synesthetes. Also, synesthetes tended to choose darker, more saturated colors when matching their synesthetic responses to the cello melodies, when compared to the colors that they chose when matching their synesthetic responses to the piano melodies. Finally, when asked to select the colors that were most emotionally compatible to the music, correlations between the emotional ratings of the music and the emotional ratings of the colors indicated that synesthetes could match colors to music with similar emotional content (.81 for Happy-Sad, .83 for Active-Passive, .61 for Strong-Weak). Cross-Modal Sound to Sight Associations with Musical Timbre in Non-Synesthetes Previously our lab has found that both complex classical music and two-note musical intervals have consistent cross-modal mappings to dimensions of color, and that these associations seem to be mediated by the shared semantic and emotional meaning of colors and music (Palmer & Schloss, 2013; Griscom & Palmer, VSS 2012). In a new set of experiments, we asked participants to indicate what colors, edge contrasts, and onset dynamics "went best" with the sound of 17 different common musical instruments (violin, piano, marimba). We found evidence for emotional mediation in particiant color choices, such that they tended to choose colors whose emotional quality was similar to the rated emotional quality of the sound (e.g., "happy" looking colors, such as saturated yellow, were chosen for "happy" sounding instruments, such as a harpsichord). Systematic sound-to-sight correspondences were also evident in non-synesthetes' choices of "best fitting" edge contrasts (low to high spatial frequencies) and dynamic onsets (temporal dynamics of contrast) for circular shapes, but these were better explained by correspondences among low-level perceptual features such as attack time and spectral brightness than by any kind of emotional mediation. Additionally, we tested a set of stimuli with multiple timbres and found that the visual matches with these audio were well predicted by a simple combination of the visual matches for their component timbres. The Color of Musical Sounds: Color Associates of Harmony and Timbre in Non-Synesthetes Previous studies have shown that the tempo and harmonic mode (major vs. minor) of different pieces of classical music produce consistent cross-modal associations with dimensions of individual colors and that these associations are largely mediated by the shared emotional content of the colors and music (Schloss & Palmer, VSS-2008; Xu et al., VSS-2011; Palmer et al., VSS-2011). In the present study we extended this line of research by investigating cross-modal associations between two-color displays and musical sounds that varied in instrumental timbre and the harmonic structure of 2-note intervals and 3-note chords. The color stimuli consisted of color pairs chosen from the 37 colors of the Berkeley Color Project and the sound stimuli consisted of synthesized notes, two-note intervals, and three-note chords, using instrumental timbres derived from Grey's (1977) timbre space. Participants were asked to perform a 2AFC task to indicate which of two color pairs they felt was more strongly associated with the presented musical sound. They also rated the sound and the color pairs separately on each of five emotional dimensions (happy/sad, angry/clam, active/passive, strong/weak, and harmonious/disharmonious). The probability of choosing a given color pair was found to be strongly related to the fit between the degree of harmony of the musical sound and the difference in the degree of harmony between the two color pairs presented on that trial (i.e., harmonious color pairs were chosen to go with harmonious sounds and disharmonious color pairs to go with disharmonious sounds). Instrumental timbre also showed systematic effects on participants' cross-modal choices of visual displays that differed in both color and spatial characteristics. Color, Music, and Emotion Arnheim (1986) speculated that different aesthetic domains (e.g., color and music) might be related to each other through common emotional associations. We investigated this hypothesis by having participants pick from among an array of 37 colors the five colors that went best (and later the five that went worst) with each of a set of musical selections that varied in composer, tempo, and mode (major/minor). They also rated each musical selection and each color for its emotional associations (happy-sad, lively-dreary, strong-weak, angry-calm). For both orchestral music and solo piano music, systematic mappings were found between the dimensions of color and music: faster music and major mode were associated with lighter, more saturated, yellower colors, whereas slower music and minor mode were associated with darker, desaturated, bluer colors. These mappings appear to be mediated by common emotional associations, because the correlation between emotional ratings of the musical selections and emotional ratings of the colors chosen to go with them were extremely high (.90 to .98) for all emotional dimensions studied (e.g., people picked happy colors to go with happy music and dreary colors to go with dreary music). Further studies using better-controlled musical stimuli (unaccompanied theme-and-variation melodies by Mozart) dissociated effects due to instrumental timbre (piano/cello), register (high/low pitch), and note density (quarter-note theme vs. eighth-note variation), as well as tempo and mode from the specific influences of different melodic and harmonic structure in the earlier studies. The mediating role of emotion was established by obtaining analogous effects when people picked the colors that went best (and worst) with faces and body poses that expressed emotions (happy-sad and angry-calm). Similarly high correlations were obtained when the emotional ratings of the faces/gestures were compared with corresponding emotional ratings of the colors chosen to go with them. The Color of Music We investigated the relations among color, music, and emotion for the 37 colors of the Berkeley Color Project: saturated, desaturated, light, and dark shades of red, orange, yellow, yellow-green, green, blue-green, blue, and purple, plus white, black, and 3 grays. To study color/music relations, participants viewed all 37 colors while listening to 18 orchestral selections in major and minor keys by Bach, Mozart, and Brahms that had slow, moderate, and fast tempos. For each selection, participants chose the five most-consistent and the five least-consistent colors. Across composers, faster tempos and major keys were both associated with brighter, warmer, more saturated colors than slower tempos and minor keys. To determine whether affective responses might mediate these color/emotion associations, we also studied the relation between the color samples and emotion words and between the musical selections and emotion words. Participants produced color/emotion associations by rating the consistency between 16 emotion words and each of the 37 colors. Strong associations were found between many emotion words and colorimetric dimensions: e.g., happy, lively, and enthusiastic with light-warm-saturated colors; sad, dreary, and unenthusiastic with dark-cool-desaturated colors; strong and aggressive with warm-saturated colors; and weak and shy with cool-desaturated colors. Participants also produced music/emotion associations by rating the consistency between each emotion word and each of the 18 musical selections. We found a strong link between the affective response to musical selections and the affective response to the corresponding chosen colors. Specifically, there was a strong positive correlation between the ratings of emotional associations to the 18 musical selections and the ratings of emotional association to the colors people chose as most/least consistent with the same musical selection. This finding suggests that affective response may mediate the relations we found between color and music. All of the above research is funded in part by NSF Grant BCS-0745820 to Stephen E. Palmer and by a gift from Google, Inc. |
   |