Algorithmic sampling
April 21, 2016 12:57 AM Subscribe
Music Mappr will take any uploaded mp3 or soundcloud link, chop the linked piece of music into short fragments, reorder the fragments based on sonic similarity and let you play back clusters of these fragments independently.
The sorting process looked amazing, sadly the fragments of my mp3 were unlistenable because of the harsh clicks. Suggestion: mask splits by adding 10ms fades to the beginnings and ends of fragments.
posted by yoHighness at 3:03 AM on April 21, 2016 [1 favorite]
posted by yoHighness at 3:03 AM on April 21, 2016 [1 favorite]
Infinite Call Me Maybe anyone?
posted by Nanukthedog at 3:11 AM on April 21, 2016
posted by Nanukthedog at 3:11 AM on April 21, 2016
Does anyone know what measure of similarity it is using? Because it seems to be a very different one from the one my ears and brain use. The visualisation looks like it may be doing something like Sammon mapping to visualise the points corresponding to snippets into 3D space, but it'd be interesting to know what the high dimensional vector being generated for each snippet is made up of before it is mapped into 3D.
posted by drnick at 4:06 AM on April 21, 2016 [1 favorite]
posted by drnick at 4:06 AM on April 21, 2016 [1 favorite]
Oh, found it on the github front page for it:
... it extracts raw samples from a song. Then, it divides those samples into chunks and performs a fourier transform on each chunk to extract frequency data (dividing the work into multiple WebWorkers to speed up the process). Then, MusicMappr uses the T-SNE algorithm to visualize differences between the high dimensional fourier features in 2-d space. Finally, it runs K-means clustering to show the clusters that arise from T-SNE.
So it looks like its working on ampitude in frequency bins/ranges to create a high D vector to compare snippets and then map into 3D. It'd be nice to try different measures or combinations of them to try and better match aural perception.
posted by drnick at 4:18 AM on April 21, 2016 [1 favorite]
... it extracts raw samples from a song. Then, it divides those samples into chunks and performs a fourier transform on each chunk to extract frequency data (dividing the work into multiple WebWorkers to speed up the process). Then, MusicMappr uses the T-SNE algorithm to visualize differences between the high dimensional fourier features in 2-d space. Finally, it runs K-means clustering to show the clusters that arise from T-SNE.
So it looks like its working on ampitude in frequency bins/ranges to create a high D vector to compare snippets and then map into 3D. It'd be nice to try different measures or combinations of them to try and better match aural perception.
posted by drnick at 4:18 AM on April 21, 2016 [1 favorite]
Yeah this is useless for creating music but delightful for taking something like, say, Alan Jackson's It's 5 O Clock Somewhere and turning into nightmarish chaos!
posted by Potomac Avenue at 5:12 AM on April 21, 2016
posted by Potomac Avenue at 5:12 AM on April 21, 2016
I just fed it Norma Tanega's "You're Dead" -- a simple 60s folk rock song -- and it actually picked out and clustered repeated or rhyming words in the lyrics. "Dead," "down," "stay/say/pray" "you're" and so on. That's pretty spiff. I tried it on "Jolene" by Dolly Parton and I got all the low "Jolenes" on one key and the high "Jolenes" on another. On more complicated music like Eleanor Rigby it seems to pay as much attention to background instrumentation as to lyrics though.
posted by edheil at 6:01 AM on April 21, 2016 [2 favorites]
posted by edheil at 6:01 AM on April 21, 2016 [2 favorites]
Interesting. I was thinking of doing something similar, only as a way of searching a large music library, most of whose contents I haven't heard for ages, for similarities. My idea was to go through all the files, divide each one into a set of sounds (by chopping up and clustering), and then globally calculate the closest connections between segments across files. I.e., if there are two songs which share, somewhere, a particular timbre or texture, they'd show up in the search.
(I haven't gotten around to doing this yet, but intend to do so at some point.)
As for the metric, I heard that Mel-frequency cepstral coefficients are a good proxy for timbre, at the cost of having 5 or more dimensions.
posted by acb at 6:26 AM on April 21, 2016 [1 favorite]
(I haven't gotten around to doing this yet, but intend to do so at some point.)
As for the metric, I heard that Mel-frequency cepstral coefficients are a good proxy for timbre, at the cost of having 5 or more dimensions.
posted by acb at 6:26 AM on April 21, 2016 [1 favorite]
Fascinating. I just tried this on Tom's Diner and on Bohemian Rhapsody.
It really didn't know what to do with Tom's Diner, which is not surprising, because (algorithmically, at least), the whole song sounds the same. So there are a bunch of tiny sound snippets clustered more or less at random.
Bohemian Rhapsody, of course, is the exact opposite: it's like 5 songs in one, each very different. It got broken into clusters that are nearly intact passages of the original song.
posted by adamrice at 8:12 AM on April 21, 2016 [1 favorite]
It really didn't know what to do with Tom's Diner, which is not surprising, because (algorithmically, at least), the whole song sounds the same. So there are a bunch of tiny sound snippets clustered more or less at random.
Bohemian Rhapsody, of course, is the exact opposite: it's like 5 songs in one, each very different. It got broken into clusters that are nearly intact passages of the original song.
posted by adamrice at 8:12 AM on April 21, 2016 [1 favorite]
I fed it Yes's "Going for the One"; there were a few nice out-of-context chord changes that are decent stealable ideas for other songs. It seems you get good results if the chopped-up bits of song are not all contiguous (e.g. first four bars of verse 2) but instead pull in portions spread throughout the song.
posted by kurumi at 10:04 AM on April 21, 2016
posted by kurumi at 10:04 AM on April 21, 2016
1. Feed it this
2. Randomly roll your mouse over all the clusters, the faster the better
3. ???
4. Madness!
posted by Hal Mumkin at 4:07 PM on April 21, 2016
2. Randomly roll your mouse over all the clusters, the faster the better
3. ???
4. Madness!
posted by Hal Mumkin at 4:07 PM on April 21, 2016
I wonder if there's an easy way to, like, somehow feed this output into a VST? That would definitely make it go from interesting to awesome.
posted by threeants at 8:37 PM on April 21, 2016
posted by threeants at 8:37 PM on April 21, 2016
This is implemented in JavaScript, right? Perhaps it's time for a VST containing an entire browser environment, and routing the inputs and outputs appropriately...Get to it, Lazyweb...
posted by acb at 1:04 AM on April 22, 2016
posted by acb at 1:04 AM on April 22, 2016
This is promising but I'd still rather do it by ear/hand: source, product.
posted by STFUDonnie at 6:34 AM on April 22, 2016 [1 favorite]
posted by STFUDonnie at 6:34 AM on April 22, 2016 [1 favorite]
« Older treated static | If cities are sexed, then Cork is a male place. Newer »
This thread has been archived and is closed to new comments
posted by lozierj at 2:01 AM on April 21, 2016