In part 2 of our 6 part series on Neil Harbisson, the world’s first cyborg, we interview him about how the technology that translates color into sound works and how blending the senses of sight and hearing into a whole new experience.

Munsell met up with Neil Harbisson in New York City in April of 2014 for the interview. He told us he felt very comfortable in New York City because he didn’t have to feel as self-conscious as there were so many different types of people there that no one pays attention or says anything about his antenna. However, upon meeting him, his brightly colored choice of clothing made him stand out in the crowd more than the antenna attached to his head. Although his pants were black, he was wearing a bright turquoise shirt, fluorescent orange jacket and two different color socks.

The software running in the chip embedded in his skull maps the colors the camera perceives into a sound. Harbisson hears the sound in his head, but because he hears it through bone induction, those around him cannot hear any sounds. Multiple colors can blend together sounds, like a chord. But as we learned, mapping the frequency of visible light to the frequency of notes on a scale has far more granularity than most people would be able to differentiate.

To get an idea of what colors sound like to Neil Harbisson, you can hear his sonochromotic music scale in this video:

Hearing Color Balance Vs. Seeing Color Balance

Munsell: If you have a color palette that’s really well balanced versus clashing colors, does it translate color to sound in a way that works the same way? For example, if you have a harmonious color, does it sound more like a pleasant chord, or does it not map to musical notes like that?

NH: I don’t think it will match cause when it comes to sound, then the harmonies are different. Do you mean if there’s a relationship between color harmony and music harmony?

Munsell: Yes, did they think about that when they mapped the colors to sounds?

NH: Oh no, they didn’t think about it, no, because it’s direct transposition of light frequencies to sound frequencies, so there is no connection between the cultural element of color or the visual harmony of color and the frequencies I receive, so it doesn’t really match, no.

Munsell: So if you’re looking at a color palette that goes really well together, it could actually sound bad to you.

NH: Exactly, yeah. But then, this is also very subjective because for some people it might sound bad and for some other people it might sound good. And this happens a lot, and it happens the same visually, that some people might find a color combination really, really good and others don’t. So yes, maybe in general, if there was a big study of harmony maybe they would be, I think there would be disagreement between the relationship between color harmony and sound harmony.

What Neil Harbisson wore during the Munsell interview.

Munsell: I notice you’re wearing complementary colors today. Do you find that you often wear harmonious color combinations?

NH: Well, yeah. I guess today I’m wearing silent trousers [black], which is unusual, but then to compensate the lack of sound I am wearing two different colored socks. One is A and the other is G. Just to compensate the silence.

Munsell: So do complementary colors sound harmonious?

HM: Yeah, more or less.

Munsell: And what about colors that sound clashing?

NH: For example, someone would dress in different shades of green and people would say that it looks fine, but it sounds terrible!

Munsell: Because if it’s not a one to one mapping with musical notes, then I guess sounding good and looking good aren’t necessarily the same.

NH: No, so someone might be dressed in shades of blue for example, for example my shoes and my shirt don’t sound the same. They are the same color but they sound slightly different and they wouldn’t really sound good together. But I’m wearing it, so… [laughs]. Same color family but there is different hue. This one is a bit lower and this one’s a bit higher so when you hear this musically, it creates a vibration that, for most people, it would be uncomfortable to hear, but to me, I’m so used to hearing this combo that I like it. But musically it’s out of tune to wear this.

Wearing colors that “sound good”. (photo credit: Lars Norgaard)

How Hearing Colors Affects Music

Munsell: You were a musician, right?

NH: Yes, this actually destroyed all my musical training, because hearing color is much more complex than hearing music.

I used to play the piano, which had twelve notes, and color has 360 notes in an octave, so to me the piano now is just too simple. There isn’t enough in an octave.

Munsell: So does that mean the entire visible spectrum of colors lies within an octave? That’s a lot of colors!

NH: A bit less than that, yeah.

Munsell: So there’s no repeat?

NH: No repeat, so when you scale down light frequency to sound frequency, then all the visual colors fit within less than one octave. So it’s a bit less than an octave.

Harbisson’s Sonochromatic Music Scale is a microtonal and logarithmic scale with 360 notes in an octave. Each note corresponds to a specific degree of the color wheel. (source: wikipedia)

Munsell: Do you think it would have been possible when they created the software to map colors to sounds in a way that would harmonize musically when the colors harmonize visually?

NH: Yeah, but I didn’t want to perceive color like this. I wanted to perceive color as pure as possible so I wanted to perceive the frequency of the light and then we could hear red we would hear a note between F and F sharp, which is for most people, this is already annoying because you can’t play it with a piano, because it’s between notes, so it’s a microtone. But that’s the actually frequency of red is between those.

Munsell: So it’s much more granular than the notes on a scale?

NH: Yes. So it’s when you listen to the frequencies of color, you need to, basically, forget about music. Because music only has 12 colors, it’s just twelve notes that keep repeating themselves. So it’s the same 12 notes in different octaves.

Munsell: Did that change your perception of music? If you hear music that is a little off key, are you more sensitive to it because you can better differentiate notes?

NH: Yes, color perception has given me a perception of sound that I didn’t have before. So thanks to color, now I can detect and I have perfect pitch within this octave which I didn’t have before, and also, it’s changed the way I perceive untuned music. I actually like hearing music that is untuned because it gives colors that are not usually heard in well tuned music.

Munsell: So it didn’t make you more sensitive, it made you appreciate it!

NH: Yeah, exactly. So, for example, like, schools, children’s schools, when they play [out of tune] it’s perfect! It’s so much more colorful than a perfectly tuned orchestra. That orchestra will usually play the same 12 colors whereas the school will play maybe 60 more colors.

When you hear color it’s extremely microtonal. In my case I have 360 notes in an octave, so, there are many more chances of creating chords that don’t sound good.

Our interview is continued with Part 3: What Color Sounds Like. For the first post of this series, as well as links to future posts about Harbisson, read Neil Harbisson – Part 1: Rearranging the Senses of Color Sight and Sound