Comments on: Audio as Visual

Audio as Visual

youarenothere — Wed, 09 Apr 2008 13:50:08 -0800

The intersect of data visualization and aural phenomena is a fascinating space, from simple chartings of the history of sampling to mapping the entire world of music (or even just electronica). Pop songs become sketches, iTunes libraries become twisted geometric forms, and last.fm listening behaviors form coloured orbs and waves. The collaborative networks of comtemporary rappers, jazz musicians, and classical composers are revealing of specific and meaningful community structures. Explore the algorithmic music of Stephan Wolfram's computational universe, listen to pi or e or the Mona Lisa or the weather or the temperature in New York City, discover the shape of sound, or just, you know, see music. Use the Echo Nest to visualize your own music (example), tag your music collection with colours, or just wade through the plethora of ways to map connections between artists and genres. (several previously)

By: googly

googly — Wed, 09 Apr 2008 13:57:13 -0800

From the "contemporary rappers" link: Figure 3. The Wu-Tang Clan and their neighbours. Plotted with the Kamada-Kawai graphing algorithm. This is so many different kinds of geeky cool that I just had to breathe into a paper bag for a few moments upon reading it. Neat post. thanks.

By: elmono

elmono — Wed, 09 Apr 2008 14:12:11 -0800

Fantastic post. Tks.

By: eritain

eritain — Wed, 09 Apr 2008 14:58:45 -0800

The visualizations from flyingpudding.com are pretty cool. Anyone want to speculate with me about the details of the timbre -> color matching? First guess: It's based on the overall power spectrum at that time slice (i.e., they're not decomposing it into notes again, a la Melodyne, which is why you don't see warm and cool colors in the same column), the brightness is simply the energy in band (from FFT), and the high frequencies correspond to the red end of the color spectrum. I'd really like to see it done after a Melodyne note-extraction analysis, actually. Then you could analyze timbres relative to the note's fundamental, and—heck, take them all the way through a cepstral analysis, and you could make saturation correspond to harmonicity and value correspond to spectral density. Ooh, and then you could shorten the timescale so that percussive attacks would show up as a white front edge ... mmm, tasty. yep, I'm a nerd

By: demiurge

demiurge — Wed, 09 Apr 2008 14:59:54 -0800

listen to pi or e or the Mona Lisa or the weather or the temperature in New York City This is a waste of time. There is no understanding to be gained from converting these sorts of signals into audio, and it doesn't even sound good.

By: brianwhitman

brianwhitman — Wed, 09 Apr 2008 15:11:41 -0800

(nb: I am a cofounder of the echo nest) The visualizations from flyingpudding.com are pretty cool. Anyone want to speculate with me about the details of the timbre -> color matching? First guess: It's based on the overall power spectrum at that time slice (i.e., they're not decomposing it into notes again, As far as I know Anita is using some function of timbre and pitch data from the Echo Nest Analyze XML per segment. It may just be the first 3 bins of the timbre data mapped to RGB. You can see a pretty good description of the data in the XML on our developer site (where you can make your own analyses.) We also have parsers in Python, Flash, Processing and Objective C. I put up a Mefi project for the launch of the analyze API and This Is My Jam, a site that makes audio use of the API to make mini-beatmatched mixes.

By: brianwhitman

brianwhitman — Wed, 09 Apr 2008 15:15:12 -0800

oh PS we have a page full of examples of different stuff (audio and visual) made using the Analyze API)

By: flapjax at midnite

flapjax at midnite — Wed, 09 Apr 2008 16:30:17 -0800

...listen to pi or e or the Mona Lisa or the weather or the temperature in New York City... There is no understanding to be gained from converting these sorts of signals into audio... Understanding shmunderstanding. You just put a beat under it and dance. Dance the night away!

By: anthill

anthill — Wed, 09 Apr 2008 17:33:13 -0800

Very good point. To get understanding, data alone isn't enough. Transforming sound data into information, real comprehension, is something humans can only do in certain ways - cat pi > /dev/dsp isn't one of them. Good design of sonifications requires designers to consider not only what fundamental information that needs to be communicated, but also the abilities and limitations of the human aural system. This scientific journal article shows some very promising up and coming research in how to design effective medical and aerospace alarms. More here and here.

By: anthill

anthill — Wed, 09 Apr 2008 17:34:15 -0800

I really wish Penny Sanderson (the main researcher in the above work) would post some sound samples online - her work is obviously better heard than seen.

By: avoision

avoision — Wed, 09 Apr 2008 19:48:09 -0800

I'm loathe to self-link in general... but given the context of this post, I think I'm still staying on-topic here. Last July, I created a project entitled "Astronaut" that takes a friend's song and visualizes the lyrics based on keywords via Flickr. I was listening obsessively to this song, and happened to be digging around the Flickr API at the same time. If anyone's interested, there's a bit more backstory/context in this blog post. Also - as a brief aside - the pi10k project originated from my early days learning Flash/Actionscript. It was mostly born out of me wanting to learn how to better manipulate strings/arrays. To this day, I'm in awe that something which originated as a personal exercise is still making the rounds.

By: klangklangston

klangklangston — Wed, 09 Apr 2008 23:10:52 -0800

Yeah, that fucking waves thing for last.fm? I put in my request three weeks ago when this was last posted and it still hasn't rendered.

By: jonp72

jonp72 — Thu, 10 Apr 2008 08:28:18 -0800

"A collection of adjacent k=3 k-cliques centering on the rapper RZA found using the clique percolation method after the weighted edge disparity algorithm is run for X = 50." Coolest. Graph. Ever.