Smelly colors

10 min readSep 12, 2022

My favorite fruit is a peach in the month of August. The perfect August peach is ripened to a softness and juiciness that I can only eat while leaning aggressively over the kitchen sink, allowing the tub to catch the sweet fruity juice running down my elbows. Naturally, for a long time, my favorite color was also peach. I associated seeing that color with the flavor, scent, and feel of an August peach.

This is one somewhat mild instance of a crossmodal correspondence. In this article, I will give a brief review of this sensory concept through the lens of the smell-color correspondence. I offer some of the vocabulary and tools to continue thinking about crossmodal correspondences in your own experiences.

In this article, I will walk through:

A common framework within which to understand crossmodal correspondences. What are they and where do they come from?
Some of the ideas, studies, and findings of existing works in odor-color crossmodal correspondences.
Synesthesia as a special case of semantic crossmodal correspondence.
An open question about whether pre-trained machine learning models capture common cultural crossmodal correspondences.

Crossmodal correspondences are defined as the broader family of psychological associations in which the experience (or imagination) of one sensory stimuli is accompanied by experiencing or expecting an associated feature in another sensory modality. For example, hearing a high-pitched noise and picturing a small far-away object. Or smelling something grassy and visualizing a green landscape. Or the famed bouba/kiki effect.

Crossmodal correspondences might arise due to any one or combination of the following three forms (following Charles Spence’s 2011 ‘Crossmodal correspondences: a tutorial review’):

structural — Structural correspondences are inherent in the structure of neural interplays in our brain.
statistical — Statistical correspondences are learned from some singular or repeated experience of multiple simultaneous sensory stimuli in the environment.
semantic — Semantic correspondences are due to matching identity / meaning of different stimuli. They can be informed by language itself.

The third form, semantic crossmodal correspondence, warrants further explanation. The concept of semantically-mediated crossmodal correspondence has also been referred to as ideasthesia, the “sensing of concepts.”

This may be best illustrated with an example: One study demonstrated that accompanying a speeded visual size discrimination task with a task-irrelevant sound can significantly influence participants’ performance on the task. The experimenters presented two disks, one after another, and asked the participant to identify whether the second disk was larger or smaller than the first. The second disk was presented with either no sound or a relatively high or low pitch. The participants responded significantly more quickly (and somewhat more accurately) when a high-frequency sound was presented with a smaller second disk and a low-frequency sound was presented with a larger second disk.

This finding is compelling… but couldn’t this be statistical crossmodal correspondence? Maybe we are used to hearing a high-pitched buzzing and seeing a fly dart across our vision.

So then the same experimenters presented the participants with the spoken words “high” and “low” while doing the same size discrimination activity. They found that the words had a similar effect as the pitched sounds on participants’ performance: participants identified the relative size of the second disk better when presented with the word “high” when the second disk was smaller and “low” when the second disk was larger. These findings support the hypothesis that there may be a semantic connection with language as the mental intermediary between “high”/”low” to high-/low- pitch to smaller/larger objects. The semantic connection between the word “high” and high-frequency sounds, then the statistical association between high-frequency sounds and smaller objects suggest that there is some identity association in our brains between these different sensory stimuli.

Color-odor correspondence

Association between color and odor is a crossmodal correspondence between the visual and olfactory senses. Visual features play a strong influence on odor perception. In fact, some color-smell associations are so cogent that when presented with an orange-colored cherry-flavored drink, people perceived the beverage as orange-flavored.

In a number of fascinating publications, color-odor associations have been related to culture (statistical correspondence), linguistic cues (semantic correspondence), and fMRI brain scans (structural correspondence). Below I highlight a few:

Culture (statistical, semantic):

Color-odor associations have been found to be relatively consistent within a culture but tend to vary across cultures, which is in line with the notion that many associations are learned through experience rather than a universal association embedded in the neural structures.

Odor perception and color perception individually vary by culture. For example, in the US, anise, wintergreen, and cinnamon have been associated with sweets while in France, they tend to be associated with florals and traditional medicine. People in different cultures also tend to judge foreign odors as more intense.

The variance in access to different colors and color descriptors across languages also affects peoples perceptions of color. That is not to say that two people from different cultures would physically view a single color shade differently, but they may categorize that color under different labels. For example, Korean and Russian have two separate color names for what English refers to as one color: “blue.”

Color-odor associations are relatively consistent within a culture but vary by culture. The following image shows results from a study that had participants from different populations smell odor pens of various scents (fruity, flower, candy, etc.) and select colors that they felt were congruous with the scent.

Most frequently associated colors (block colors) with scents (horizontal axis), according to population culture (vertical axis).

Some associations are consistent — for example, “musty” was matched to brown, green, yellow hues in all cultures. Some others are quite different! Most of interest to me was the “candy” column. In the US, the scent of candy was matched most to pink and blue hues. (I’m thinking of Skittles, M&Ms, rainbow ropes…) But in China, candy was most matched to black, then blue and pink hues.

My mom and dad grew up in China then immigrated to America in their 20’s, so I asked them about this. Here is a quote from our family text conversation:

Mom: It is black sugar or dark brown sugar. You can get it from the Chinese market. You can drink it in warm water… it makes the blood flow smoother somehow. 黑糖…
Dad: I don’t think Chinese associate black with sweet in general though.
Mom: I would say pink if you ask me to associate with sweet.

Brain scans (structural):

fMRI machines have been used to examine neural activity in response to certain stimuli. One study reports enhanced neural activity in learning and memory sections of the human brain when subjects are presented with compatible picture-odor combinations (e.g. picture of bus, smell of diesel).

In a small study of 9 participants, subjects were scanned under an fMRI machine while exposed to either odors or colors in isolation or color-odor combinations varying in rating of how well they were perceived to match (yellow, red, turquoise, brown : lemon, strawberry, spearmint, caramel). Brain activity in the orbitofrontal (decision-making) and insular (sensory, consciousness, emotion) cortexes increased with perceived congruency of the odor-color pairs.

Linguistic cues (semantic):

Linguistic cues have also shown to play a role in smell perception.

In one study, participants were presented odors with a positive, neutral, or negative name. They found that the same smell, when presented with a pleasant name such as “pine needles,” was identified as a more positive scent than when presented with a neutral “thirty-one” or negative name “old turpentine”. Additionally, the study found that odors were rated as more intense when presented with negative names.

Another study was inspired by the observation that Dutch speakers tend to describe odors by referring to the source (e.g. “smells like banana”) while Maniq and Thai speakers tend to describe odors with an abstract dedicated scent vocabulary (e.g. “musty”). In this study, experimenters placed objects in a squeezy bottle (as illustrated for the title image of this blog post), squeezed and wafted the scents toward the participants, and had participants select a color from a color card. Among other findings that certain objects and their scents varied by culture in familiarity and thus identifiability, their investigation of the role of language in odor-color associations yielded findings suggesting that when people use abstract smell terms to describe odors, they are less likely to choose the color match of the original object, while when they used a source-based term, the color choices were more accurate. It should be additionally noted that this “success” metric may be biased for western cultures; previous studies have shown the role of urbanization and structured education on brain organization.

This communication between seemingly disparate sensory regions in the brain is especially pronounced in individuals with synesthesia.

Synesthesia

Synesthesia is a neural phenomenon in which some crossmodal correspondences are particularly strong and involuntary. There are a number of different forms of synesthesia, as it can occur in any combination of sensory and cognitive pathways. One common form of synesthesia is associating letters and/or numbers with colors. These types of synesthetes evoke certain colors upon processing certain letter or number characters.

There have been a couple potential explanations for synesthesia.

Some support the semantic vacuum hypothesis, which works with ideasthesia to explain how synesthesia develops in early childhood education. Ideasthesia, in line with semantic crossmodal correspondence, suggests that people learn by actively assigning some meaning to stimulus in their world. As a child begins to learn their first abstract concepts such as numbers and letters and time, that child has to build a semantic network in order to assign meaning to stimuli. The hypothesis suggests that a child, in the absence of a large semantic network or any semantic network at all, is faced with the “semantic vacuum.” A synesthetic child may turn to synesthesia to facilitate their building of their semantic network, thus associating any combination of concepts learned at the time, including but not limited to letters, numbers, dates, and colors.

Synesthesia may also be accounted for by increased cross-talk or disinhibited feedback between the regions of the brain that are specialized for different sensory and cognitive functions.

Color-odor synesthesia

Synesthesia can affect the way that odors are perceived and concepts are shaped. One might theorize that individuals with color-odor synesthesia might perform spectacularly well in the color-identification-by-scent experiments discussed in the previous section.

One study examined exactly that. Their findings:

synesthetes outperform controls on ability to discriminate odors
synesthetes outperform controls on ability to discriminate colors, especially in the yellow, green-yellow, green, red, and red-purple hue regions
no difference between synesthetes and control participants on ability to pick up odors where the scent is less perceptible
no difference between synesthetes and controls in olfactory imagery ability

The findings, however, were not unexpected for the scientists. After all, previous studies have shown an impressive ability of nonsynesthetes to match colors to odors, especially where the source object can be named. The odor-color associations in synesthetes are also not always in accordance with real-world odor-color associations. See below an image of color associations by one synesthete in the study.

Example of odor-color associations by one synesthetic participant in the study.

The colors associated with leather are pink and purple, which is not a common color of leather in the real world. This dichotomy between real-world associations and synesthetic associations might actually negatively affect the ability of synesthetes to identify source object colors to smells.

However, still synesthetes outperformed control participants on odor discrimination and color discrimination tasks. This finding is consistent with models of synesthesia that identify the presence (or lack of inhibition) of additional neural connections in synesthetic individuals.

Machine learning and crossmodal correspondences

As a student of machine learning, particularly natural language processing, I wondered: if crossmodal correspondences can arise from statistical co-occurrences of concepts, can pre-trained word embedding or other machine learning models learn the same crossmodal correspondences as humans do?

Word embedding models map real words and phrases to real-valued vectors in the embedding space. This tutorial from CMU explains the idea superbly. They also released this Word Embedding Demo, an interactive word2vec model visualization. You can add words, dimensions, and see each entry’s most similar words.

Here’s a image from me playing around with it:

Based on my cursory play-around, I’m not convinced that word2vec is learning human-like crossmodal correspondences…. but I’m also not convinced it’s not.

So I now present an open question:

What do you think? Would we expect machine learned models to be able to learn crossmodal correspondences, if they are learned from only text?

If they do learn crossmodal correspondences, we would expect them to learn different correspondences depending on the language / culture of the text from which the model learns. → Try out another language: word2vec trained on Finnish words, hosted by the Turku NLP group

If they do not, why not? What if they are learned from multiple modalities?