Does ‘Stadium Arcadium’ Taste Like Spicy Food?

What we can learn from synesthesia and how to fake it

When we talk about building natural interfaces — tools that intuitively communicate with our minds — we’re talking about figuring out how we process information, specifically sensory information, because that’s how everything comes into our brains (at least at a primary, primitive level). Synesthesia is literally a malfunction in the processing of information, an opportune blending of the senses that has long been associated with greater creativity and artistic ability (Nabokov, Mozart, Van Gogh, Feyman, Tesla, Jimi Hendrix, Lady Gaga are all proclaimed synesthetes). This makes perfect sense, because after all, what is creativity and where does it come from? (And why is it so hard to teach, making it a huge challenge for AI?)

Creativity is linking together disparate concepts and coming up with crosstalky solutions, pulling info from one unique experience and applying it to another, putting a distant 2 and 2 together to get fish, and having it make sense. Novel ideas come from interdiscplinariness, and synesthesia is by definition an interdisciplinary human experience.

Famed neuroscientist V.S. Ramachandran found that grapheme-color synesthetes have an unusually high number of connections between certain brain areas, such as the fusiform gyrus (responsible for processing number shapes) and visual area V4 (responsible for processing colors). He suggested that these connections promote cross-communication that is responsible for associations such as yellow 8s and magenta Qs.

What if we could mimic these extra connections and all become synesthetes (in a controlled way, of course)? Besides potentially being more creative, would we learn better/differently? Would we have more vivid memories, because more of our senses would automatically be engaged with the information?

Some scientists certainly believe in the synesthetic advantage. According to Dr. Nicolas Rothen at the University of Sussex,

“There is evidence that grapheme-colour synaesthesia in particular leads to enhanced memory functions” (src)

Some apps (e.g. Color Code) actually try to teach synesthesia by conditioning the brain to associate certain symbols and colors — which is technically valid, as demonstrated by Bor & Rothen in 2014. Associative memory can be powerful; if you see blue with the number 2 enough times, soon you’ll be making that association everywhere (similar to how we universally envision grass as green, apples as red). However, even if this approach works, this association will be “in the mind’s eye.” In contrast, many synesthetes actually project colors into the real world — they actually SEE them, activating their visual and color processing systems in a very real way.

Associative learning can have quite an effect on synesthesia though — if you are already synesthetic. A recent study found that many grapheme-color synesthetes (who perceive colors when they read letters/numbers) had associations that could be traced back to Fisher-Price’s alphabet fridge magnets! A follow-up study tested over 6000 synesthetes, and found that ~6% of them matched the alphabet set. For kids born during the magnets’ peak popularity (1975–80), the rate was an astonishing 15%! The fridge magnets didn’t cause synesthesia, but certainly shaped it during development. (Witthoft, Winawer, Eagleman, 2015)

It’s clear that synesthesia has its benefits, and that it’s hard, if not impossible, to acquire. But is there a way for us to artificially simulate these associations? How can we cross-activate our senses and layer information in unconventional ways, e.g. visualizing music, sonifying color, etc?

We can use augmented reality to feel some of the effects — and benefits — of synesthesia.

AR is built for this, to convey these sorts of experiences. AR = digital information layered on top of the physical world. It literally means the augmentation, the enhancement of our realities. If all we see is numbers, AR is the path to seeing colors blended with those numbers — or let’s go one step further, even savant-esque visions of mathematical relationships in the real world. AR is how we artificially layer more information on the world and perceive, through our senses, a more complete picture.

In fact, AR can go further than synesthesia by allowing us to control our sensory projections, interact with them. Synesthesia is a very personal condition; each manifestation is unique. Maybe one day, we can have synesthesia views tailored for our individual brains, with inputs and outputs for taste and smell, as well as vision and hearing. Who wouldn’t want to be “ambisensous”?(trademarking that)

So far, I haven’t been able to find a quality synesthesia simulator for AR — something I’m sure will come very soon. Perception will be forever changed by the person who creates an experience that imparts honed, useful, and personalized synesthesia for intuitive, multimodal processing of complex information.

Would we learn faster, remember more? What sort of creative potential would we unlock?

Let’s take a meaningful step toward discovering the benefits of sensory cooperation. By conceptualizing data with multiple senses, we can recreate the associations that are instinctively called on by true synesthetes…until we uncover a way to really control/manipulate/reprogram our senses for some truly next-gen cross-chatter in the brain and the ability to unlock brain experiences only people tripping on acid have been able to describe.

How Far Can Perception Go?

Our imaginations are infinitely powerful; kids are a testament to that. Think back to when you were a kid and used to play make-believe. Even when they know better, kids can play pretend and actually believe it. And therein lies the magic — they can choose when to let go and believe their imaginations, and when it’s time to come down to reality. Is there a way to return to that state of pure imagination, to harness that ability? Imagine being able to actually believe — just for a moment — that you’re Spaceman Spiff again.

As we grow older, our games of make-believe become less and less convincing. We think it’s because we become more mature and understand how the world really works. But is it possible that kids have such an easy time imagining things because, like synesthetes, they have more communication overlap in their brains? It has been widely demonstrated that children have far more connections in the brain than do adults. As they develop, brain areas become more specialized and synapses are ‘pruned’ away. There’s actually a theory that we’re all born synesthetic, but lose the ability during the pruning process. Synesthesia has also been shown to be more common in children. Because young kids still have those extra synapses, could it be that they’re actually seeing and hearing what they’re believing, at least on some level? Because kids’ brains are so much more flexible/plastic, and their brain regions are fluidly communicating with each other, they presumably have significant cross-activation in their brains, meaning that they might — in some small part — be experiencing what they are imagining.

This ability isn’t totally lost in adults. We might have fewer connections and less plasticity, but the potential is still there; LSD gives evidence of that. Many people who have taken LSD or other hallucinogens have related experiences of synesthesia (usually auditory-visual), such as the woman in the clip to the right who starts to see air and sound: “I can see everything in color…I’ve never seen such infinite beauty in my life…This is reality.” The name “Purple Haze” practically speaks for itself. These accounts show that the capacity exists — we just need to come up with better, more controlled ways to unlock it. (Read a comprehensive research review on chemically induced synesthesia here)

It is unclear how, or even whether, we will someday be able to selectively revert our brains to some plastic state and continuously shape our brains. Nevertheless, it does seem clear that the way toward this worthy goal is to build natural interfaces — intuitive tools that engage more of our sensory spectrum and enhance creative connections. By doing so, perhaps we can uncover increasingly effective ways to learn.

For more info on grapheme-color synesthesia, see this paper I wrote while studying neuroanatomy at Brown: https://1drv.ms/b/s!AqHnZbti30LAga9kTUYdPpfrJoctCQ