Comments on: Speaking Piano

Speaking Piano

Burhanistan — Wed, 07 Oct 2009 11:32:01 -0800

The Speaking Piano, and Transforming Audio to MIDI - Austrian Composer Peter Ablinger has transformed a child speaking so that it can be played as MIDI events on a mechanically-controlled piano, making the piano a kind of speech speaker.

By: Think_Long

Think_Long — Wed, 07 Oct 2009 11:35:25 -0800

so is the original played back over the piano? It seems like that's the case; if so, what's the point of the piano? It's still pretty cool

By: phrontist

phrontist — Wed, 07 Oct 2009 11:37:20 -0800

so is the original played back over the piano? It seems like that's the case; if so, what's the point of the piano? You and I have nothing in common.

By: jedicus

jedicus — Wed, 07 Oct 2009 11:41:21 -0800

Ha. Peter Frampton, eat your heart out.

By: CynicalKnight

CynicalKnight — Wed, 07 Oct 2009 11:43:46 -0800

transformed a child speaking so that it can be played as MIDI events That kid's gonna be too cool for school with that 5 pin DIN plug in the back of his head...

By: Think_Long

Think_Long — Wed, 07 Oct 2009 11:45:03 -0800

You and I have nothing in common. ouch. I was just wondering, I still think it's an awesome project

By: pb

pb — Wed, 07 Oct 2009 12:00:15 -0800

The piano voice sounds eerily like the child voice in The Color of Fire by Boards of Canada. They can finally perform it live!

By: phrontist

phrontist — Wed, 07 Oct 2009 12:00:30 -0800

Think_Long: That was a bit tounge and cheek, and in any case there was no intended value judgement.

By: fatbaq

fatbaq — Wed, 07 Oct 2009 12:03:15 -0800

So the article suggests that the childs voice is most likely mixed in with the piano during playback...? I'm guessing it would just sound like a cat running across the keys without the vocals played alongside it? But why not just use a MIDI capable keyboard instead of wiring up that piano, that looks like alot of work. cool video.

By: nanojath

nanojath — Wed, 07 Oct 2009 12:05:26 -0800

Try to understand any of it without looking at the subtitles though - and not something you've already seen with the subtitles once, something totally new. I'm not saying the noises it produces are irrelevant to the words they represent - they clearly aren't - but listening to something while watching subtitles of what is supposedly being said creates an illusion that the speech is far clearer than it actually is as your brain "fills in". I couldn't make out a word of it without the subtitles.

By: tommasz

tommasz — Wed, 07 Oct 2009 12:11:50 -0800

Couldn't the Mellotron have done this too?

By: nanojath

nanojath — Wed, 07 Oct 2009 12:14:42 -0800

So the article suggests that the childs voice is most likely mixed in with the piano during playback...? I'm pretty sure the audio you hear in the video is purely the piano. As far as why he's using a real piano, come on, it's art. You could just have a computer generate a vague synthesis of speech out of a limited range of tones too but nobody would find that particularly interesting I think, "look I've created a poor speech synthesizer that requires input from a live actor to say anything!"

By: adipocere

adipocere — Wed, 07 Oct 2009 12:16:53 -0800

This is going to be creepy, isn't it?

By: MrMoonPie

MrMoonPie — Wed, 07 Oct 2009 12:18:43 -0800

Reminds me of Herbie Hancock (and Tatyana Ali!) on Sesame Street. But that was straight midi, I guess.

By: raygirvan

raygirvan — Wed, 07 Oct 2009 12:20:16 -0800

"Sparky, oh, Sparky ... It is I, your piano..."

By: idiopath

idiopath — Wed, 07 Oct 2009 12:28:11 -0800

This is not just playing the sound into the piano, it is converting the sound into a flurry of notes, basically a cross between fft resynthesis and granular synthesis via midi controlled servo motors. If I am not mistaken the original input sound is not being replayed at all. If he used OSC rather than midi he could actually get a much more accurate recreation of the sound, because midi is limited to a very low clock speed.

By: ardgedee

ardgedee — Wed, 07 Oct 2009 12:34:00 -0800

> As far as why he's using a real piano, come on, it's art. You could just have a computer generate a vague synthesis of speech out of a limited range of tones too... We're conditioned to believe any sound can come out of a speaker. Electronics have been capable of human speech synthesis for decades. There's nothing to see, and it'd be difficult to make a new approach seem impressive at all. The piano makes it amazing because it's an overt demonstration of how close you can get to approximating human speech and voice by striking piano keys the right way.

By: phrontist

phrontist — Wed, 07 Oct 2009 13:04:13 -0800

The piano makes it amazing because it's an overt demonstration of how close you can get to approximating human speech and voice by striking piano keys the right way. Moreover it's a great way of demonstrating to someone that sounds can be decomposed, like colors.

By: The Bellman

The Bellman — Wed, 07 Oct 2009 13:06:20 -0800

Reminds me of Herbie Hancock (and Tatyana Ali!) on Sesame Street . But that was straight midi, I guess. Probably not MIDI at all. Just a then-state-of-the-art sampler. Possibly a Fairlight CMI, judging from the guy playing with the waveform on the big CRT in the back.

By: The Bellman

The Bellman — Wed, 07 Oct 2009 13:07:41 -0800

On looking at the first part of the clip, yes. A Fairlight CMI.

By: The White Hat

The White Hat — Wed, 07 Oct 2009 13:12:21 -0800

It's slurring its words-- the piano has been drinking.

By: b1tr0t

b1tr0t — Wed, 07 Oct 2009 13:13:39 -0800

This project won't be complete until a score is printed, and a classroom full of 9 year old musical prodigies (no, not this kind) perform it on a stadium full of pianos.

By: empath

empath — Wed, 07 Oct 2009 13:43:31 -0800

i think it might be better with a big pipe organ than a piano.

By: Hartham's Hugging Robots

Hartham's Hugging Robots — Wed, 07 Oct 2009 13:45:17 -0800

So the article suggests that the childs voice is most likely mixed in with the piano during playback...? I'm pretty sure the audio you hear in the video is purely the piano. 1) In the body of the article there's an edit that says: "Listening again, the short answer to how you can hear so much of the voice through the piano seems to be, you can't; the original is almost certainly mixed in." FWIW, I'd say that's absolutely right. 2) If y'all read on down to the comments, you'll see someone saying: "Here's a piano only video http://vimeo.com/1483630. Not the same text and with additional notes. Very amazing." The sound in that video is plainly different to the video in the linked article, further suggesting the sound you hear in the linked article there does have the original mixed in. Having said all of this: I think it's an awesome piece of work, lovely, fascinating, thanks Burhanistan...

By: Hartham's Hugging Robots

Hartham's Hugging Robots — Wed, 07 Oct 2009 13:46:24 -0800

See also the link posted within that Vimeo video for a full explanation of the work by the artist.

By: RobotVoodooPower

RobotVoodooPower — Wed, 07 Oct 2009 13:53:55 -0800

There's no way you could generate the siblants (s, f, etc) with just piano keys -- thus the overdubbing. What you have here is the world's most complicated spring reverb unit.

By: phrontist

phrontist — Wed, 07 Oct 2009 14:17:00 -0800

Bah, if there is overdubbing this is pointless.

By: Hartham's Hugging Robots

Hartham's Hugging Robots — Wed, 07 Oct 2009 14:18:55 -0800

There's no way you could generate the siblants (s, f, etc) with just piano keys You're right, but what's interesting is the way the temporal resolution, specifically, makes for... a very unique type of sound from a mechanical instrument.

By: No-sword

No-sword — Wed, 07 Oct 2009 14:55:03 -0800

Well, now I know how robots are going to speak in my steampunk novel.

By: koeselitz

koeselitz — Wed, 07 Oct 2009 15:04:30 -0800

[♦] Peter Ablinger's web site [♦] Peter Ablinger performing a piece by Michael Pinter-Koschell [♦] A brief interview with Peter Ablinger from five years ago, mostly about his cycle Words and Music [♦♦♦♦] Mark Knoop performs Peter Ablinger: Voices and Piano 1, 2; Instrumente und Rauschen; Akkordeon und Rauschen

By: koeselitz

koeselitz — Wed, 07 Oct 2009 15:07:11 -0800

Sorry; [♦] Mark Knoop plays Peter Ablinger's Voices and Piano 1. My favorite is Ablinger's rendition of the piece by Pinter-Koschell.

By: mandymanwasregistered

mandymanwasregistered — Wed, 07 Oct 2009 15:47:25 -0800

Heeey, speaking of Peter Ablinger: Voices and Piano...metafilter's own speicus will be performing 3 selections from that work in South Pasadena on the 17th. Details are here. I heard "Angela Davis" the other night, and it's pretty sweet.

By: chococat

chococat — Wed, 07 Oct 2009 16:10:29 -0800

Terrifying. Soon to be appropriated by the makers of the next Friday the 13th movie.

By: The Great Big Mulp

The Great Big Mulp — Wed, 07 Oct 2009 16:21:05 -0800

Man, I've had an idea similar to this for quite some time. It's good to see someone's done it.

By: speicus

speicus — Wed, 07 Oct 2009 16:30:01 -0800

Most of you are misinformed in this thread -- the original sound is NOT mixed in with the piano. It's true that some sounds can't be duplicated by the piano -- which is why the words are not audible unless you are reading the text. Your brain fills in the missing information. It's also important to note that this would not be possible with a live performer. The player piano has a "resolution" of about 16 attacks per second, far faster than any human being could perform continuously and accurately. This is necessary because the sound has to at least approach the rate of digital synthesis. Here is a little more background on this phenomenon of making musical instruments "speak," which he calls "phonorealism." Basically, as I understand it, he's using software to break down the sound into its component parts, i.e. sine waves, then resynthesizing them. But instead of resynthesizing using sine waves, Ablinger does it mechanically with the player piano! For another (hilarious) example of this, there's A Letter From Schoenberg, in which the famously abrasive composer Arnold Schoenberg recites a letter eviscerating his publisher for violating a contract. As crazy as this effect is, Ablinger himself seems to think of it as little more than a parlor trick: "My main concern is not the literal reproduction itself but precisely this border-zone between abstract musical structure and the sudden shift into recognition." His other compositions, like the aforementioned Voices and Piano, dwell in this border-zone. In Voices and Piano, a live pianist actually does perform along with the recordings. Obviously a live performer can't play nearly so much nearly so fast, but what's amazing is what aspects of the voice Ablinger chooses to bring out -- they're like little character studies. The music isn't just analytical anymore; it's also incredibly expressive. And people say electronics have no soul! And, the pieces are fun to play. In conclusion, Ablinger rules.

By: empath

empath — Wed, 07 Oct 2009 17:10:15 -0800

This is similar -- Sine Wave Speech. Also: ...a head, an intricately worked bust, cloisonné over platinum, studded with seed pearls and lapis. ..The thing was a computer terminal...it could talk. And not in a synth-voice, but with a beautiful arrangement of gears and miniature organ pipes. It was a baroque thing for anyone to have constructed, a perverse thing, because synth-voice chips cost next to nothing...Smith jacked the head into his computer and listened as the melodious, inhuman voice piped the figures of last year's tax return.

By: matkline

matkline — Wed, 07 Oct 2009 17:13:49 -0800

Is it just me, or does that kid have an excellent accent?

By: Aquaman

Aquaman — Wed, 07 Oct 2009 18:07:58 -0800

I concur with Speicus; the piano, by itself, is replicating the voice of the kid, by means of re-creating each tiny slice (aka "sample") of the kid's voiced audio with a precisely played note or combination of notes on the piano. There is no remixing or overdubbing; the instrument is being manipulated in such a way to (more or less) replicate the sound of human speech. Very elegant and sweet piano hack.

By: koeselitz

koeselitz — Wed, 07 Oct 2009 18:31:24 -0800

speicus: Most of you are misinformed in this thread -- the original sound is NOT mixed in with the piano. It's true that some sounds can't be duplicated by the piano -- which is why the words are not audible unless you are reading the text. Your brain fills in the missing information. It's also important to note that this would not be possible with a live performer. The player piano has a "resolution" of about 16 attacks per second, far faster than any human being could perform continuously and accurately. This is necessary because the sound has to at least approach the rate of digital synthesis. Hey, now. If anyone was 'misinformed,' it was the author of the article linked, who claims that this couldn't possibly be piano alone. If you have an issue, take it up with him.

By: DU

DU — Wed, 07 Oct 2009 18:35:19 -0800

SHUT UP AND LET US HEAR IT GODDAMN I HATE SELF-IMPORTANT "NEWS" ORGANIZATIONS

By: empath

empath — Wed, 07 Oct 2009 19:04:24 -0800

A Letter From Schoenberg, The reverb on that video makes it VERY difficult to hear, but even so, I could pick out the words in the last 3rd of that, reading along with the text.

By: speicus

speicus — Wed, 07 Oct 2009 19:06:36 -0800

koeselitz: Hey, now. If anyone was 'misinformed,' it was the author of the article linked, who claims that this couldn't possibly be piano alone. If you have an issue, take it up with him. I should probably comment over there too, you're right. But in my defense, all I really mean by "misinformed" was that you'd been given bad information (by the author who was himself misinformed, as you pointed out). I just wanted to make sure the misinformation didn't spread any further, and cast any more doubts on Ablinger's achievement. I realize, though, that in the context of the internet, "misinformed" often means "YOU ARE WRONG AND CAN THEREBY SUCK IT," and I probably should have phrased things a little differently. Hey look! Another video, without commentary this time.

By: koeselitz

koeselitz — Wed, 07 Oct 2009 19:21:02 -0800

That's fair, speicus - sorry I took that a little harsher than I'm sure you meant it. By the way: I'm certain you're correct. I'm too lazy to go to the trouble of uploading it, but I downloaded that youtube video of the child's-voice piece and slowed it down a whole bunch in VLC Player - there is clearly no smooth backing track, only the piano making the tones. What's interesting is that this works much better with the child than it does with the Schoenberg piece - I believe this is partially because the piano is place in such an echo-filled room for the Schoenberg piece, but I think it's also because a child's voice seems to be easier to replicate, being on a higher and perhaps narrower frequency.

By: xorry

xorry — Wed, 07 Oct 2009 20:03:05 -0800

This is a much better version of the Schoenberg than the vimeo clip that mostly sounds like room.

By: theDTs

theDTs — Wed, 07 Oct 2009 23:29:54 -0800

We're so used to listening to music and audio so distorted by the compression used to fit it on our MP3 players, or for streaming it over the internet, that we're quite able to accept that we must be hearing some of the original track. I specifically remember cringing at how WMA compression would transform the sibilants on my music tracks into a tinnitus-like ringing very much like the rat-a-tat-tatting of that rightmost key. In fact, solve the portability issue and add a Click Wheel, and they just might be on to something!

By: empath

empath — Wed, 07 Oct 2009 23:37:46 -0800

Btw, this same method is the way that some compressed audio works, like g729 (if you use voip). If you've ever noticed that audio books from itunes or audible.com sound 'funny', that's why-- it's also vocoded. It offers huge compression gains for a cost of sounding slightly dehumanized. and it's how they can fit 2 hours of audio into a 35 meg file.

By: Blazecock Pileon

Blazecock Pileon — Thu, 08 Oct 2009 00:31:27 -0800

Excellent post. Thank you.

By: DecemberBoy

DecemberBoy — Thu, 08 Oct 2009 00:57:56 -0800

I wrote a shitload about this earlier today, but it was all on Facebook. I thought this was really amazing, and wrote a lot of words about how I believe it works, why it's so neat, etc. I'm gonna repost some of it here, if that's not cool, then mods feel free to baleet it. Explaining what I'm pretty sure it's doing: Or, here's a better explanation: you know how a piano, tuned as it is by ratios/formulae/whatever (equal temperament tuning, i.e. basically all western instruments), one note will be X Hz (hertz), and the next note will be X + Y Hz, and so on, and it's a significant leap between each one? This takes a more complex sound like speech, and sort of "rounds" out all those "between-the-notes" frequencies down to only the frequencies that comprise equal tempered tuning. 10 hours ago · Delete Oh, also, in MIDI each note event number (0-255, or 8 bits) maps to a frequency/note in the equal tempered tuning frequency series (most higher end synths let you change the frequency table for, say, Indian tuning, microtones, etc.), so then it's easy to generate MIDI note events. I'm pretty sure that's what it's doing. And more, featuring some explanation of synthesis and MIDI in general and how it relates to all this: OK, so this is following on from the post about the speech-to-MIDI thing. First of all, if you look at, say, a single piano note's waveform in an audio editor, it's pretty simple: a little noise from the percussive hammer strike, then a more or less constant frequency. In ye olde tymes, when there were only analog synths, they were based on a simple oscillator. This oscillator can produce any frequency within its range in the form of a simple waveform (square, tri, saw, sine, pulse, etc.). Then you filter out overtones, etc. etc. producing a more complex, yet still pretty simple sound. The frequency-to-scale mapping was done by what was called control voltage, and every manufacturer had a different system, and it doesn't matter here how it works anyway. MIDI, on the other hand, works like this: you hit a key. It sends a "note on" event, with a channel (irrelevant here what a "channel" is) number and a note number from 0-255. The synth, which by the time MIDI arrived would have a microprocessor, does a table lookup on that note number, and it corresponds to a frequency. The table, unless it's been altered (Indian tuning, etc.), will consist of column A, 0-255, and column B, the frequencies of the Western (equal tempered) scale. How an audible note actually gets played depends on the synth, but that's how the process starts. Most sounds musical instruments make, in terms of the frequencies they're made up of, are quite simple. Bells make probably the most complex waveforms of the common instruments, and couldn't be synthesized accurately until FM came along (mid 80s). Speech, on the other hand, is extremely complex by comparison. This is why it's only now that speech synthesizers sound only sort of almost natural. So you've got this extremely complex sound made up of many frequencies. The speech-to-MIDI thing "simplifies" it in the way I already explained, leaving only the equal tempered note frequencies. Here, the PureData patch is what is mapping the remaining frequencies to MIDI note numbers, and the way a note is played is that the PD patch, on a computer, outputs MIDI note on/off events, and the piano is equipped with an electronic/mechanical system which strikes the corresponding piano string to that MIDI note. You could also use it to send note events to a common synthesizer or anything that accepts MIDI (except a drum machine type instrument, which works a bit differently). I may have oversimplified/poorly explained some of this, particularly the "table lookup" bit, but I was really trying to explain why this is so awesome, and the audience was a group of my friends who don't necessarily have much interest in synthesis/electronic music/etc., at least not like I do. But yes, this is really, really awesome, I think. I also could be totally wrong about how the thing works, so please tell me if I am. I want to understand this as best I can, but the math at least is certainly beyond me.

By: DecemberBoy

DecemberBoy — Thu, 08 Oct 2009 01:04:36 -0800

Oh, and you could technically use it with a drum machine/sampler, I guess. I just didn't want to go into the way MIDI works with samplers/drum machines (very briefly, note events trigger a certain drum sample on a drum machine, and a sampler can do the same or pitch a single sample up/down, instead of the whole "note frequency in Hz via table" thing).

By: ob

ob — Thu, 08 Oct 2009 03:12:37 -0800

I just saw this. This is pretty remarkable. I've never heard of Ablinger and will be checking out more of his work. This particular thing reminds me a little of certain works of Clarence Barlow (or however he spells his name these days.)

By: flapjax at midnite

flapjax at midnite — Thu, 08 Oct 2009 04:26:46 -0800

I couldn't make out a word of it without the subtitles. Thus showing once again that the drum (as in West African talking drum) is more sophisticated than the piano. note to phronitist: that's tongue IN cheek.

By: idiopath

idiopath — Thu, 08 Oct 2009 12:12:42 -0800

DecemberBoy: great explanation except for a minor quibble: MIDI is not nearly so reasonable in implementation as you make it sound on the bit level, most (all?) values are 7 bit, or multiples of 7 bits in size. A midi note value of 255 gives you a frequency of approximately 20,390,000 hz, which is over a thousand times too high for a female teenager with undamaged hearing to perceive, not to mention your typical adult.

By: DecemberBoy

DecemberBoy — Thu, 08 Oct 2009 14:41:31 -0800

Yeah, I freely admit to not fully grasping the MIDI spec. You gotta admit it's pretty damn arcane, though. It was basically one of those consensuses among manufacturers (I think Sequential Circuits was behind a lot of it, right?) that was a "consensus" in that no one hated it TOO much.

By: speicus

speicus — Fri, 09 Oct 2009 10:33:29 -0800

OK, so: inspired by this lively discussion, here is my attempt to recreate this effect, for what it's worth. It doesn't sound nearly as good as Ablinger's version, but I had fun trying.