BRAVO1 learns to speak again
July 14, 2021 9:00 PM Subscribe
“Neuroprosthesis” Restores Words to Man with Paralysis [UCSF article] tells of using computer-brain interface to think words for communication by people with speech loss. 21st Century medical breakthrough doesn't feel like an overstatement.
As so often with these projects, the achievement is less impressive on examination. They don’t actually have the general ability to tell directly from neural signals to the vocal tract what words were intended. Instead, working with the individual, they painstakingly learn a set of neural signals which that individual can use to indicate items in a vocabulary of fifty words, with median 75% accuracy.
The fact that they’re using vocal tract signals is not essential - they could use other neural activity equally well. Perhaps they think vocal tract signals will feel more natural to the subject, but IMO there’s a slight feeling of a con trick about that choice, even if it’s one they’re playing on themselves as well as us.
posted by Phanx at 11:54 PM on July 14, 2021 [8 favorites]
The fact that they’re using vocal tract signals is not essential - they could use other neural activity equally well. Perhaps they think vocal tract signals will feel more natural to the subject, but IMO there’s a slight feeling of a con trick about that choice, even if it’s one they’re playing on themselves as well as us.
posted by Phanx at 11:54 PM on July 14, 2021 [8 favorites]
As so often with these projects, the achievement is less impressive on examination.
To me this is the kind of thing that needs to be looked at from the help people having a hard time communicating angle rather than the where's my mind machine interface already angle (and don't get me wrong, I will be waiting in line when they start selling those). If it's working and it's better than what they had before, it's a win.
posted by each day we work at 3:59 AM on July 15, 2021 [6 favorites]
To me this is the kind of thing that needs to be looked at from the help people having a hard time communicating angle rather than the where's my mind machine interface already angle (and don't get me wrong, I will be waiting in line when they start selling those). If it's working and it's better than what they had before, it's a win.
posted by each day we work at 3:59 AM on July 15, 2021 [6 favorites]
I wonder if, with 75% accuracy, and 50 words, there's something more effective than 50 commonly used words - you could maybe double up some words so that they also represent letters, almost like having a meta key but just from context. You'd definitely want a yes and no with better accuracy, but I'm guessing some signals are clearer than others, and a yes no might be clearest.
posted by condour75 at 4:20 AM on July 15, 2021
posted by condour75 at 4:20 AM on July 15, 2021
See also Toki Pona, a conlang with 123 words.
posted by condour75 at 4:25 AM on July 15, 2021 [1 favorite]
posted by condour75 at 4:25 AM on July 15, 2021 [1 favorite]
So, this is adjacent to my area of expertise. I take care of people with Parkinson's disease, which over time blocks the signals from the language cortex to the muscles of the vocal cords. Different mechanism than this patient (stroke), same cruel effect of knowing what you want to say but being unable to say it. I also work with deep brain stimulation, which uses electrical signals to override some of that blockage and restore function. Ironically, speech often worsens after DBS, due to the motor pathways for the vocal tracts and the limbs being super close together, and it's easy to get overflow stim where you don't want it.
Some preliminary thoughts:
0. Speech (physical) is distinct from language (cognitive).
1. Based on the description of spastic quadriparesis and anarthria with preserved cortical language function, this patient appears to have had locked-in syndrome, which is also what Jean Dominique Bauby (of the Diving Bell and the Butterfly) and M. Noirtier (of the Count of Monte Cristo) had. Among neurologists, it's quite probably the most feared disease. (Non-neurologists worry they'll get dementia. Neurologists worry about dementia, too, but we worry more about being locked-in.)
2. If you take as a first principle that self-expression is a fundamental human right, restoration of the ability to communicate, for someone who has been cognitively intact and completely unable to express themselves freely, for 15 years, is a good thing.
3. Existing ways of communication for patients with locked-in syndrome are painstaking and often reliant on another person to translate. J-D Bauby, for instance, had his assistant read the alphabet over and over, and he blinked when they got to the right letter. Eye-trackers remove the other person, but still rely on the "one letter at a time" method. But this method of communication appears more self-reliant, and is "one word at a time" i.e. closer to natural speech.
4. This is a single patient; unclear whether results will translate (sorry) to others.
5. This is not deep brain stim; this is use of a cortical array (I'm guessing similar to what is used in planning for epilepsy surgery), except instead of merely detecting, it adds a decoding feature to translate neuronal impulses into words.
6. Study was funded by Facebook, sets off alarm bells for me.
7. I have concerns about what this might do to the Deaf community (see "0" above).
More thoughts forthcoming after I have a chance to read and digest the original article.
posted by basalganglia at 4:26 AM on July 15, 2021 [22 favorites]
Some preliminary thoughts:
0. Speech (physical) is distinct from language (cognitive).
1. Based on the description of spastic quadriparesis and anarthria with preserved cortical language function, this patient appears to have had locked-in syndrome, which is also what Jean Dominique Bauby (of the Diving Bell and the Butterfly) and M. Noirtier (of the Count of Monte Cristo) had. Among neurologists, it's quite probably the most feared disease. (Non-neurologists worry they'll get dementia. Neurologists worry about dementia, too, but we worry more about being locked-in.)
2. If you take as a first principle that self-expression is a fundamental human right, restoration of the ability to communicate, for someone who has been cognitively intact and completely unable to express themselves freely, for 15 years, is a good thing.
3. Existing ways of communication for patients with locked-in syndrome are painstaking and often reliant on another person to translate. J-D Bauby, for instance, had his assistant read the alphabet over and over, and he blinked when they got to the right letter. Eye-trackers remove the other person, but still rely on the "one letter at a time" method. But this method of communication appears more self-reliant, and is "one word at a time" i.e. closer to natural speech.
4. This is a single patient; unclear whether results will translate (sorry) to others.
5. This is not deep brain stim; this is use of a cortical array (I'm guessing similar to what is used in planning for epilepsy surgery), except instead of merely detecting, it adds a decoding feature to translate neuronal impulses into words.
6. Study was funded by Facebook, sets off alarm bells for me.
7. I have concerns about what this might do to the Deaf community (see "0" above).
More thoughts forthcoming after I have a chance to read and digest the original article.
posted by basalganglia at 4:26 AM on July 15, 2021 [22 favorites]
What’s the concern with this development and the Deaf community?
(I’m deaf, not Deaf)
posted by Jubal Kessler at 5:20 AM on July 15, 2021
(I’m deaf, not Deaf)
posted by Jubal Kessler at 5:20 AM on July 15, 2021
The fact that they’re using vocal tract signals is not essential - they could use other neural activity equally well. Perhaps they think vocal tract signals will feel more natural to the subject, but IMO there’s a slight feeling of a con trick about that choice, even if it’s one they’re playing on themselves as well as us.
HAMILTON: But Pandarinath says communication devices wired directly to the brain could make it hard for people to separate private thoughts from those they choose to make public.posted by hippybear at 5:32 AM on July 15, 2021 [3 favorites]
PANDARINATH: We want to make sure the devices that we create allow that separation, allow people to be able to continue to think their private thoughts without anything just being broadcast to the world. Experimental Brain Implant Lets Man With Paralysis Turn His Thoughts Into Words [NPR]
6. Study was funded by Facebook, sets off alarm bells for me.
Sure enough....
Facebook pulls plug on mind-reading neural interface that restored a user's speech
Facebook is abandoning a project to develop a brain-computer interface (BCI), even as the researchers it funded have showcased the device helping someone with severe speech loss communicate with nothing more than thought....
posted by Umami Dearest at 7:20 AM on July 15, 2021 [1 favorite]
Sure enough....
Facebook pulls plug on mind-reading neural interface that restored a user's speech
Facebook is abandoning a project to develop a brain-computer interface (BCI), even as the researchers it funded have showcased the device helping someone with severe speech loss communicate with nothing more than thought....
posted by Umami Dearest at 7:20 AM on July 15, 2021 [1 favorite]
What’s the concern with this development and the Deaf community?
Some Deaf advocates I know are concerned about the potential uses of BCI to eliminate Deafness by emphasizing verbal speech as equivalent to language (which they are not). Similar to concerns around cochlear implants, I think. But I have not heard directly from them on this specific study, and I am not Deaf, so I don't want to over-interpret what I've heard in other contexts.
posted by basalganglia at 7:28 AM on July 15, 2021 [4 favorites]
Some Deaf advocates I know are concerned about the potential uses of BCI to eliminate Deafness by emphasizing verbal speech as equivalent to language (which they are not). Similar to concerns around cochlear implants, I think. But I have not heard directly from them on this specific study, and I am not Deaf, so I don't want to over-interpret what I've heard in other contexts.
posted by basalganglia at 7:28 AM on July 15, 2021 [4 favorites]
I was surprised, given that the mechanism is attempt-to-speak based, that they went with 50 words rather than "40 sounds". Teaching a kid to read you learn that english has (supposedly) 42 basic sounds (but qu is really k-w and x is k-s, so actually just 40). I know that's not strictly true and there are slight variations in pronunciation etc. But still, if this is coding attempts to control speech muscles, it seems like you should be able to train it to pick up those 40 sounds and from there you can get ANY word, right? You might have to train it for something that amounts to accent or whatever (just like speech to text software), I guess, but still that seems like it could use the same mechanism and be way more useful. I wonder why they didn't go that way?
Also, great for BRAVO1 who can now talk, but I'd love to see this with someone whose loss of speech is more recent since their muscle memory of how to control speech would probably be better. If this works you could set people up as soon as they have a speech-impacting injury. And I wonder if -- since they'd be exercising all their speech-making neural pathways -- it might help people to recover speech?
And yes, I am aware that both comments are idle speculation from a person who is in no way qualified to idly speculate.
Anyway, I think this is great and I can't wait for it to get even greater.
posted by If only I had a penguin... at 7:33 AM on July 15, 2021
Also, great for BRAVO1 who can now talk, but I'd love to see this with someone whose loss of speech is more recent since their muscle memory of how to control speech would probably be better. If this works you could set people up as soon as they have a speech-impacting injury. And I wonder if -- since they'd be exercising all their speech-making neural pathways -- it might help people to recover speech?
And yes, I am aware that both comments are idle speculation from a person who is in no way qualified to idly speculate.
Anyway, I think this is great and I can't wait for it to get even greater.
posted by If only I had a penguin... at 7:33 AM on July 15, 2021
it seems like you should be able to train it to pick up those 40 sounds and from there you can get ANY word, right?
Most English speakers don't think in terms of phonetic transcription. Can you tell me what the voiced labio-dental fricative is in English off the top of your head? (I'm sure Languagehat can.) In order to say anything, you'd basically have to do an IPA translation of the word first.
posted by praemunire at 8:16 AM on July 15, 2021
Most English speakers don't think in terms of phonetic transcription. Can you tell me what the voiced labio-dental fricative is in English off the top of your head? (I'm sure Languagehat can.) In order to say anything, you'd basically have to do an IPA translation of the word first.
posted by praemunire at 8:16 AM on July 15, 2021
No, we don't think in terms of phonetic transcription, but we know (without thinking about it) how to shape our lips and tongues and vibrate our vocal cords to make a P sound. Since this thing is already picking up on the brain's attempts to move lips/tongues/vibrate vocal cords, it seems like it should be able to note "ok, it's sending the signal to move tongue that way, lips that way, etc." and translate it into a sound and then sound combinations into words.
posted by If only I had a penguin... at 8:23 AM on July 15, 2021
posted by If only I had a penguin... at 8:23 AM on July 15, 2021
No, we don't think in terms of phonetic transcription, but we know (without thinking about it) how to shape our lips and tongues and vibrate our vocal cords to make a P sound. Since this thing is already picking up on the brain's attempts to move lips/tongues/vibrate vocal cords, it seems like it should be able to note "ok, it's sending the signal to move tongue that way, lips that way, etc." and translate it into a sound and then sound combinations into words.
If only... The lack of invariance problem exists not only for speech perception, but in some way extends to speech production. In that we may use multiple combinations of acoustic and articulatory components and arrive at them at different ways to produce sounds that constitute speech.
Eg.. Control and representations in speech production
posted by ssri at 4:10 PM on July 15, 2021
If only... The lack of invariance problem exists not only for speech perception, but in some way extends to speech production. In that we may use multiple combinations of acoustic and articulatory components and arrive at them at different ways to produce sounds that constitute speech.
Eg.. Control and representations in speech production
posted by ssri at 4:10 PM on July 15, 2021
As a person whose dad lost all ability to communicate/was probably "locked in" due to a Parkinson's plus disease, well....I guess the idea was nice while it lasted.
posted by jenfullmoon at 5:22 PM on July 15, 2021 [1 favorite]
posted by jenfullmoon at 5:22 PM on July 15, 2021 [1 favorite]
OK, further details:
-81 weeks of training
- single patient out of a planned 8. they have not started the other 7.
-two types of training: (1) participant reads a single word on screen, from a set of 50 pre-selected words and attempts to say it aloud; (2) participant reads a sentence on screen (consisting only of the 50 pre-selected words) and attempts to say it aloud
-the words are an interesting mix of standard things (hungry, thirsty, comfortable) and things that I assume were unique to this person (faith, music)
-certain sentences were eliminated prior to training due to "undesirable content" (e.g. family is bad)
-prior to this, the participant's "speech" rate via head/eye tracking was 5 words per minute.
-after training, "speech" rate via cortical detection method was 15.2 words per minute
-word error rate was 47% (compared to 2% chance rate). Sentence error rate with predictive modeling was 25.6%.
- funded by Facebook, but also by NIH and several named foundations.
So:
- less "self-expression" and more "decode a defined wordset," at least at this point
- 81 weeks is freaking forever. i wonder if this is why FB pulled out.
- this is really intriguing but is at least 5-10 years away from primetime.
posted by basalganglia at 9:59 AM on July 16, 2021
-81 weeks of training
- single patient out of a planned 8. they have not started the other 7.
-two types of training: (1) participant reads a single word on screen, from a set of 50 pre-selected words and attempts to say it aloud; (2) participant reads a sentence on screen (consisting only of the 50 pre-selected words) and attempts to say it aloud
-the words are an interesting mix of standard things (hungry, thirsty, comfortable) and things that I assume were unique to this person (faith, music)
-certain sentences were eliminated prior to training due to "undesirable content" (e.g. family is bad)
-prior to this, the participant's "speech" rate via head/eye tracking was 5 words per minute.
-after training, "speech" rate via cortical detection method was 15.2 words per minute
-word error rate was 47% (compared to 2% chance rate). Sentence error rate with predictive modeling was 25.6%.
- funded by Facebook, but also by NIH and several named foundations.
So:
- less "self-expression" and more "decode a defined wordset," at least at this point
- 81 weeks is freaking forever. i wonder if this is why FB pulled out.
- this is really intriguing but is at least 5-10 years away from primetime.
posted by basalganglia at 9:59 AM on July 16, 2021
If Facebook pulled the plug on this, does that mean BRAVO1 doesn't get to use it anymore?
posted by pelvicsorcery at 12:22 AM on July 18, 2021
posted by pelvicsorcery at 12:22 AM on July 18, 2021
« Older The Poor Man's Exploitation of the Multiverse for... | scromiting Newer »
This thread has been archived and is closed to new comments
I hate voice to text. I think the way I type and I type the way I think. I type and it hurts and I tell myself to stop, but it's hard, especially when typing feels so natural and the pain generally doesn't come until later or even the next day, especially when I've been behaving and limiting how much strain I put on my joints.
This. uh.
I was having a really bad day today.
I'm not anymore.
Just knowing this is out there puts it one step closer to something that would help those of us that physically struggle to type. I know its not as profound because I can still speak, and I can type with varying levels of difficulty (I'm typing now, even though I've been on a tear the last few days and my hands suck, (prepare for dropping things!))
And honestly, I'm someone that I'm sure has typed many more words than I've spoken, the idea of limiting (for now) and losing my "speech" via typing is terrifying. I cannot help but be hopeful this could change that.
posted by [insert clever name here] at 11:37 PM on July 14, 2021 [21 favorites]