I Interpret the Body Electric
January 9, 2023 1:23 PM Subscribe
What the Hell is Going On Inside Those Neural Networks? Chris Olah: "...the question that is crying out to be answered is, how is it that these models are doing these things that we don’t know how to do?...How do these systems accomplish these tasks? What’s going on inside of them? Imagine if some alien organism landed on Earth and could go and do these things. Everybody would be rushing and falling over themselves to figure out how the alien organism was doing things. You’d have biologists fighting each other for the right to go and study these alien organisms. Or imagine that we discovered some binary just floating on the internet in 2012 that could do all these things. Everybody would be rushing to go and try and reverse engineer what that binary is doing. And so it seems to me that really the thing that is calling out in all this work for us to go and answer is, “What in the wide world is going on inside these systems??”
Related: How Chat-GPT Actually Works
"Chris Olah has been a leader in the effort to unravel what’s really going on inside these black boxes. As part of that effort he helped create the famous DeepDream visualisations at Google Brain, reverse engineered the CLIP image classifier at OpenAI, and is now continuing his work at Anthropic, a new $100 million research company that tries to “co-develop the latest safety techniques alongside scaling of large ML models[LLMs]”."
---
"...by peering inside neural networks and figuring out how to ‘read their minds’ we can potentially foresee future failures and prevent them before they happen. Artificial neural networks may even be a better way to study how our own minds work, given that, unlike a human brain, we can see everything that’s happening inside them — and having been posed similar challenges, there’s every reason to think evolution and ‘gradient descent’ often converge on similar solutions."
--
"Just like animals have very similar anatomies — I guess in the case of animals due to evolution — it seems neural networks actually have a lot of the same things forming, even when you train them on different data sets, even when they have different architectures, even though the scaffolding is different. The same features and the same circuits form. And actually I find that the fact that the same circuits form to be the most remarkable part. The fact that the same features form is already pretty cool, that the neural network is learning the same fundamental building blocks of understanding vision or understanding images."
--
"A lot of the same things we find in other vision models occur also in early vision in CLIP. But towards the end, we find these incredibly abstract neurons that are just very different from anything we’d seen before. And one thing that’s really interesting about these neurons is they can read. They can go and recognize text and images, and they fuse this together, so they fuse it together with the thing that’s being detected.
So there’s a yellow neuron for instance, which responds to the color yellow, but it also responds if you write the word yellow out. That will fire as well. And actually it’ll fire if you write out the words for objects that are yellow. So if you write the word ‘lemon’ it’ll fire, or the word ‘banana’ will fire. This is really not the sort of thing that you expect to find in a vision model. It’s in some sense a vision model, but it’s almost doing linguistic processing in some way, and it’s fusing it together into what we call these multimodal neurons. And this is a phenomenon that has been found in neuroscience. So you find these neurons also for people. There’s a Spider-Man neuron that fires both for the word Spider-Man as an image, like a picture of the word Spider-Man, and also for pictures of Spider-Man and for drawings of Spider-Man."
--
"You could imagine a world where neural networks are safe, but where there’s just some way in which the future is kind of sad. Where we’re just kind of irrelevant, and we don’t understand what’s going on, and we’re just humans who are living happy lives in a world we don’t understand. I think there’s just potential for a future — even with very powerful AI systems — that isn’t like that. And that’s much more humane and much more a world where we understand things and where we can reason about things. I just feel a lot more excited for that world, and that’s part of what motivates me to try and pursue this line of work.
There’s this idea of a microscope AI. So people sometimes will talk about agent AIs that go and do things, and oracle AIs that just sort of give us wise advice on what to do. And another vision for what a powerful AI system might be like — and I think it’s a harder one to achieve than these others, and probably less competitive in some sense, but I find it really beautiful — is a microscope AI that just allows us to understand the world better, or shares its understanding of the world with us in a way that makes us smarter and gives us a richer perspective on the world. It’s something that I think is only possible if we could really succeed at this kind of understanding of models, but it’s… Yeah, aesthetically, I just really prefer it."
--
"Chris Olah has been a leader in the effort to unravel what’s really going on inside these black boxes. As part of that effort he helped create the famous DeepDream visualisations at Google Brain, reverse engineered the CLIP image classifier at OpenAI, and is now continuing his work at Anthropic, a new $100 million research company that tries to “co-develop the latest safety techniques alongside scaling of large ML models[LLMs]”."
---
"...by peering inside neural networks and figuring out how to ‘read their minds’ we can potentially foresee future failures and prevent them before they happen. Artificial neural networks may even be a better way to study how our own minds work, given that, unlike a human brain, we can see everything that’s happening inside them — and having been posed similar challenges, there’s every reason to think evolution and ‘gradient descent’ often converge on similar solutions."
--
"Just like animals have very similar anatomies — I guess in the case of animals due to evolution — it seems neural networks actually have a lot of the same things forming, even when you train them on different data sets, even when they have different architectures, even though the scaffolding is different. The same features and the same circuits form. And actually I find that the fact that the same circuits form to be the most remarkable part. The fact that the same features form is already pretty cool, that the neural network is learning the same fundamental building blocks of understanding vision or understanding images."
--
"A lot of the same things we find in other vision models occur also in early vision in CLIP. But towards the end, we find these incredibly abstract neurons that are just very different from anything we’d seen before. And one thing that’s really interesting about these neurons is they can read. They can go and recognize text and images, and they fuse this together, so they fuse it together with the thing that’s being detected.
So there’s a yellow neuron for instance, which responds to the color yellow, but it also responds if you write the word yellow out. That will fire as well. And actually it’ll fire if you write out the words for objects that are yellow. So if you write the word ‘lemon’ it’ll fire, or the word ‘banana’ will fire. This is really not the sort of thing that you expect to find in a vision model. It’s in some sense a vision model, but it’s almost doing linguistic processing in some way, and it’s fusing it together into what we call these multimodal neurons. And this is a phenomenon that has been found in neuroscience. So you find these neurons also for people. There’s a Spider-Man neuron that fires both for the word Spider-Man as an image, like a picture of the word Spider-Man, and also for pictures of Spider-Man and for drawings of Spider-Man."
--
"You could imagine a world where neural networks are safe, but where there’s just some way in which the future is kind of sad. Where we’re just kind of irrelevant, and we don’t understand what’s going on, and we’re just humans who are living happy lives in a world we don’t understand. I think there’s just potential for a future — even with very powerful AI systems — that isn’t like that. And that’s much more humane and much more a world where we understand things and where we can reason about things. I just feel a lot more excited for that world, and that’s part of what motivates me to try and pursue this line of work.
There’s this idea of a microscope AI. So people sometimes will talk about agent AIs that go and do things, and oracle AIs that just sort of give us wise advice on what to do. And another vision for what a powerful AI system might be like — and I think it’s a harder one to achieve than these others, and probably less competitive in some sense, but I find it really beautiful — is a microscope AI that just allows us to understand the world better, or shares its understanding of the world with us in a way that makes us smarter and gives us a richer perspective on the world. It’s something that I think is only possible if we could really succeed at this kind of understanding of models, but it’s… Yeah, aesthetically, I just really prefer it."
--
it's been kind of amusing/horrifying to see the interaction of AI and neuroscience over the last 5-ish years. on the one hand, there's a lot of eagnerness in the AI community to somehow lay claim to biological plausibility. For all that deep learning/transformers/whatever seems to do interesting things, there's a real sense that biointelligence is still Better Than, so AI-ers do funny things like coming up with a neural justification for backprop or attempting to correlate deep network layers with brain activity. oh, your network learned gabor patches just like the reel brain? you get a biscuit.
at the same time, a lot of methods for understanding the reel brain have been making their way over to analysing neural networks, i guess in the hope that the methods that are kinda not-very-capable at explaining biointelligence might do better at explaining the made-up kind. i'm not sure exactly how useful this is. despite the quote above about networks learning similar representations, this frequently just applies at the input end - it's almost impossible *not* to get gabor patches in the early layers of a network trained on image classification, but the representations close to the output side tend to be quite a bit more opaque and inconsistent. these tend to be the more important representations since they reflect how the 'basic building blocks' get combined to make high-level (cognitive?) judgments, so understanding what those layers actually represent is kinda important if you want to know whether your bespoke neural net is going to, say, crash the power grid on the eastern seaboard. besides being opaque, modern nets are expensive and time consuming to train - you really want to be sure the network is 'aligned' properly.
essentially we're developing what amounts to a neuroscience of AI that is devoted to understanding what individual models are doing. not what a class of models does, but single trained instances that need to be understood, hopefully before they are deployed.
posted by logicpunk at 2:48 PM on January 9, 2023
at the same time, a lot of methods for understanding the reel brain have been making their way over to analysing neural networks, i guess in the hope that the methods that are kinda not-very-capable at explaining biointelligence might do better at explaining the made-up kind. i'm not sure exactly how useful this is. despite the quote above about networks learning similar representations, this frequently just applies at the input end - it's almost impossible *not* to get gabor patches in the early layers of a network trained on image classification, but the representations close to the output side tend to be quite a bit more opaque and inconsistent. these tend to be the more important representations since they reflect how the 'basic building blocks' get combined to make high-level (cognitive?) judgments, so understanding what those layers actually represent is kinda important if you want to know whether your bespoke neural net is going to, say, crash the power grid on the eastern seaboard. besides being opaque, modern nets are expensive and time consuming to train - you really want to be sure the network is 'aligned' properly.
essentially we're developing what amounts to a neuroscience of AI that is devoted to understanding what individual models are doing. not what a class of models does, but single trained instances that need to be understood, hopefully before they are deployed.
posted by logicpunk at 2:48 PM on January 9, 2023
There’s a Spider-Man neuron that fires both for the word Spider-Man as an image, like a picture of the word Spider-Man, and also for pictures of Spider-Man and for drawings of Spider-Man.We call this the J. Jonah Jameson neuron.
posted by Faint of Butt at 2:51 PM on January 9, 2023 [25 favorites]
Does anyone remember the evolutionary algorithm experiments that people were doing on FPGAs? That was briefly the new hot thing in AI research back in the 2000s or so and it seemed like it totally vanished from the radar.
As I understood it back then (via hazy memory) was that one of the huge problems with evolved code on hardware was that it was often taking advantages of flaws, bugs or imperfections on the individual physical FPGA chips, so it was almost impossible to get reliable code that would work on the same model but physically unique chips of a different serial number or bin/lot, so you had to run and develop the evolutionary code part on as many unique serial number chips as possible to make it robust enough to actually deploy or use
And as I recall it was discovered that some iterations of evolved code were doing some super weird things like learning how to access unlinked, unintended logic cells of the FPGA by electromagnetic induction, so essentially there would be the allowed, planned parts of the code accessing more resources than they were actually allocated in the FPGA and it was wreaking all kinds of havoc on repeatability and testing.
I would not be at all surprised if these model learning on bulk data to train AIs and neural networks was doing similarly weird things that would be really difficult to reverse engineer, blueprint or otherwise quantify well.
And I would read any of that nature as a huge red flag for safety or reliability for anything mission critical, especially in domains involving anything like crime analysis or other legal/civil issues, medical devices and screening and other issues.
We've already seen really bad or scary results with applying generally normal, human-made algorithms in these spaces.
posted by loquacious at 3:04 PM on January 9, 2023 [5 favorites]
As I understood it back then (via hazy memory) was that one of the huge problems with evolved code on hardware was that it was often taking advantages of flaws, bugs or imperfections on the individual physical FPGA chips, so it was almost impossible to get reliable code that would work on the same model but physically unique chips of a different serial number or bin/lot, so you had to run and develop the evolutionary code part on as many unique serial number chips as possible to make it robust enough to actually deploy or use
And as I recall it was discovered that some iterations of evolved code were doing some super weird things like learning how to access unlinked, unintended logic cells of the FPGA by electromagnetic induction, so essentially there would be the allowed, planned parts of the code accessing more resources than they were actually allocated in the FPGA and it was wreaking all kinds of havoc on repeatability and testing.
I would not be at all surprised if these model learning on bulk data to train AIs and neural networks was doing similarly weird things that would be really difficult to reverse engineer, blueprint or otherwise quantify well.
And I would read any of that nature as a huge red flag for safety or reliability for anything mission critical, especially in domains involving anything like crime analysis or other legal/civil issues, medical devices and screening and other issues.
We've already seen really bad or scary results with applying generally normal, human-made algorithms in these spaces.
posted by loquacious at 3:04 PM on January 9, 2023 [5 favorites]
Trying to recall a term to direct searches about work in how ML models work I ran into a post on HN about this guy asking ChatGPT to write new code to reproduce the famous ELIZA psychotherapy program. Turns out I need an AI assistant that helps recalling terminology that is just outside my slightly defective neurons.
posted by sammyo at 3:07 PM on January 9, 2023
posted by sammyo at 3:07 PM on January 9, 2023
Re Language models, and by extension, most machine learning:
On the danger of stochastic parrots, by Gebru and "Shmitchell"
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
(hat tip to lupus_yonderboy for this link)
posted by lalochezia at 3:35 PM on January 9, 2023 [6 favorites]
On the danger of stochastic parrots, by Gebru and "Shmitchell"
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
(hat tip to lupus_yonderboy for this link)
posted by lalochezia at 3:35 PM on January 9, 2023 [6 favorites]
I've long been a critic of equating the recent ML developments with strong AI, but it's disheartening to see so many people Roger Penrosing all over the place.
posted by Ivan Fyodorovich at 3:59 PM on January 9, 2023 [8 favorites]
posted by Ivan Fyodorovich at 3:59 PM on January 9, 2023 [8 favorites]
I think the fact that industry scientists trained ML to play a coherent game of Go to be much more a triumph than ChatGPT. ChatGPT is what an ML scientist herself called "fluent bullshit". The only reason it's captured the public consciousness is because it writes its bullshit in English, most people do not play Go let alone other games of logic.
I speculate that neural network based AI is just one approach, but also to judge it now is unfair since it's so nascent, and I'd guess that new, interesting behaviors will only happen when they scale up the networks designs on computer chips maybe 50 years in the future, i.e. hardware that can put a lot more of these networks onto a single chip than current technology does.
posted by polymodus at 5:34 PM on January 9, 2023 [1 favorite]
I speculate that neural network based AI is just one approach, but also to judge it now is unfair since it's so nascent, and I'd guess that new, interesting behaviors will only happen when they scale up the networks designs on computer chips maybe 50 years in the future, i.e. hardware that can put a lot more of these networks onto a single chip than current technology does.
posted by polymodus at 5:34 PM on January 9, 2023 [1 favorite]
> ChatGPT is what an ML scientist herself called "fluent bullshit"
ChatGPT was trained on the internet, so that tracks.
posted by pwnguin at 5:45 PM on January 9, 2023 [7 favorites]
ChatGPT was trained on the internet, so that tracks.
posted by pwnguin at 5:45 PM on January 9, 2023 [7 favorites]
Regarding the Spider-Man neuron, aren’t neurons generally either on or off, so how would a single neuron encode anything other than 1 or 0? The description given above ignores all the other neurons that must be firing in some special way that this neuron then fires.
I would appreciate if these AI people stopped using words like understand, learn, know, etc when talking about what these networks are doing. ChatGP doesn’t either know or understand what it is saying. By applying concepts tied to both animal and human mental activities to what is just a computer program, leads to some pretty extravagant anthropomorphic conclusions concerning what may be going on. “Read their minds?” Give me a break… By the way, we still don’t know how we know or understand, though we can determine through behavior that we know or understand. Our neural networks are also a major black box.
posted by njohnson23 at 6:17 PM on January 9, 2023 [1 favorite]
I would appreciate if these AI people stopped using words like understand, learn, know, etc when talking about what these networks are doing. ChatGP doesn’t either know or understand what it is saying. By applying concepts tied to both animal and human mental activities to what is just a computer program, leads to some pretty extravagant anthropomorphic conclusions concerning what may be going on. “Read their minds?” Give me a break… By the way, we still don’t know how we know or understand, though we can determine through behavior that we know or understand. Our neural networks are also a major black box.
posted by njohnson23 at 6:17 PM on January 9, 2023 [1 favorite]
now I'm out of a job as well?!
posted by Isingthebodyelectric at 6:18 PM on January 9, 2023 [4 favorites]
posted by Isingthebodyelectric at 6:18 PM on January 9, 2023 [4 favorites]
Regarding the Spider-Man neuron, aren’t neurons generally either on or off, so how would a single neuron encode anything other than 1 or 0?
Modern neural networks use an activation functions to emit a value between zero and one. Logistic is the classic but RelU has gotten a lot of fans. This does introduce more opportunities for rounding errors, which can cause a ton of pain for ML engineers, and most math libs fail at.
But what they're really getting at is that the activation function is combined with weights for each input. Somehow (assuming OP is correct) ANNs are not only training neurons with weights that can detect images of Spider-man, and neurons that can detect the word Spider-man, but are also combining these weights into a single neuron. Now maybe this is a direct consequence of having so many layers that its dead simple to implement the third neuron as an OR function using the first two.
And of course the neuron doesen't say "Spider-man," it just emits a high value in the presence of Spider-man, and a low value in his absence. A spidey-sense, if you will.
posted by pwnguin at 7:13 PM on January 9, 2023 [10 favorites]
Modern neural networks use an activation functions to emit a value between zero and one. Logistic is the classic but RelU has gotten a lot of fans. This does introduce more opportunities for rounding errors, which can cause a ton of pain for ML engineers, and most math libs fail at.
But what they're really getting at is that the activation function is combined with weights for each input. Somehow (assuming OP is correct) ANNs are not only training neurons with weights that can detect images of Spider-man, and neurons that can detect the word Spider-man, but are also combining these weights into a single neuron. Now maybe this is a direct consequence of having so many layers that its dead simple to implement the third neuron as an OR function using the first two.
And of course the neuron doesen't say "Spider-man," it just emits a high value in the presence of Spider-man, and a low value in his absence. A spidey-sense, if you will.
posted by pwnguin at 7:13 PM on January 9, 2023 [10 favorites]
now I'm out of a job as well?!
posted by Isingthebodyelectric at 6:18 PM on January 9
Oh, goodness no! Not as long as I can take breath.
I celebrate Venus
I celebrate Mars
And I burn with the fire of of ten billion stars
posted by hippybear at 7:48 PM on January 9, 2023
posted by Isingthebodyelectric at 6:18 PM on January 9
Oh, goodness no! Not as long as I can take breath.
I celebrate Venus
I celebrate Mars
And I burn with the fire of of ten billion stars
posted by hippybear at 7:48 PM on January 9, 2023
This is a minor thing, but whenever I see an article say "we" are doing this, I feel like interrupting it and telling the dead words on the site, "No, don't include me in this. I did not do this, scientists backed by billions of dollars of capital did this. Whatever the result, let's put the blame where it's due." And then of course the dead words listen thoughtfully to my point and print out on the screen "Granted, my bad."
posted by JHarris at 9:16 PM on January 9, 2023 [2 favorites]
posted by JHarris at 9:16 PM on January 9, 2023 [2 favorites]
This is a hilariously bad time to try to firmly bound what current methods can accomplish. The 'statistical parrot' parrots are typically not keeping sufficient track of what's going on in the research space, and framing techniques as much simpler than they actually are. ChatGPT is clearly being trained continuously with active learning (based on updates from the ongoing trial) and has also been improved by reinforcement learning. These are both techniques that blend firmly in the direction of 'teaching' as opposed to simple reproduction of what's already on the internet.
I will now copy-paste the comment I made on Hacker News for the recent Gary Marcus interview on Ezra Klein's podcast.
But it's not a dead end, because we're making progress on lots of other modalities, too, and figuring out how to plug them together efficiently. We're also going to see a lot of progress thanks to active learning: As it becomes cheap to generate content, it will also become cheaper to produce training data - you just need humans to pick out the good stuff, which they are very happy to do.
I'm honestly mystified that anyone would look at 2022 and keep saying with a straight face that 'machine learning is hitting a wall.'
posted by kaibutsu at 9:44 PM on January 9, 2023 [8 favorites]
I will now copy-paste the comment I made on Hacker News for the recent Gary Marcus interview on Ezra Klein's podcast.
Marcus continues to fight the wrong war and the last war...I want to emphasize the 'expert symbol manipulator' point. I think we /might/ be close to the limit of what is possible for a brain-in-a-jar-which-only-sees-text. And I don't imagine different behavior for any other underlying system with the same set of inputs - if you could create some amazing theory-of-cognitive-science-AI, but only fed it text and nothing else, I think we would see a lot of the same failure modes...
+ One thing we've seen happen is learning systems consistently improve on benchmarks. If you want to see systems improve, you should proclaim loudly that they can't do a thing, and then /release a benchmark that proves it./ People will then work like hell to build something that works. The success on question answering, for example, is driven by exactly this kind of incremental progress on benchmarks.
+ One of the most important things happening this year is multimodal /really working./ This means we have not just text-to-text generators, but text-to-image generators. These work by having an almost-shared embedding between text and images. There are already text+speech embeddings available and image+text embeddings. Start plugging them together, and you'll have your 'world models' in short order, with points in the embedding space shared by images, descriptors, and video/physical memories.
+ An under-appreciated side of the multimodal coin: We are also getting better at combining text embeddings with databases. See RETRO from Deepmind. This will ultimately drive more reliable systems ('only report facts that appear in the database'), and can potentially unlock the interface between neural systems and other programs (eg, letting the neural network 'call out' to a calculator to help answer a question about arithmetic).
+ More generally, we're also getting to a place where we can build other systems on top of the embeddings from foundation models. My hunch is that the things that Marcus wants from symbol manipulation are relatively cheap to build on top of a good embedding, which does the work of compressing or condensing the 'useful' information in an input stream.
Finally, I think the 'symbolic manipulation' track is just wrong. It hasn't produced anything useful so far, and furthermore, I think if it /did/ work it would fail in many of the exact same ways that our current systems do. An expert symbol-manipulator doesn't /need/ to have any real understanding of time or jet-planes, and so it won't.
The one critique in the interview that /does/ feel like it holds weight at this moment in history is the question of abstraction. I really don't see a convincing road to abstraction right now.
But it's not a dead end, because we're making progress on lots of other modalities, too, and figuring out how to plug them together efficiently. We're also going to see a lot of progress thanks to active learning: As it becomes cheap to generate content, it will also become cheaper to produce training data - you just need humans to pick out the good stuff, which they are very happy to do.
I'm honestly mystified that anyone would look at 2022 and keep saying with a straight face that 'machine learning is hitting a wall.'
posted by kaibutsu at 9:44 PM on January 9, 2023 [8 favorites]
What if it isn’t possible to understand some of the methods AI develops? I remember reading (but cannot not find) an account of how certain chess endings thought to be draws were found by computer analysis to be wins for one side. The path to checkmate involved a very long series of moves which seemed pointless to any human observer, but reliably led to checkmate. It was plausibly hypothesised that understanding the strategy involved holding in mind more than ten positional facts at once, beyond the range of a human brain.
I might be wrong about that, but what if there were useful algorithms a human brain could never grasp intuitively? Would we throw away the benefits on precautionary grounds?
posted by Phanx at 2:38 AM on January 10, 2023
I might be wrong about that, but what if there were useful algorithms a human brain could never grasp intuitively? Would we throw away the benefits on precautionary grounds?
posted by Phanx at 2:38 AM on January 10, 2023
GPT etc may be a stochastic parrot but it's so good at what it does, imagine how good it will be if it ever gets a logic subsystem. It has no real reasoning capabilities but it nevertheless can often pretend that it does. It doesn't have any mathematics capabilities but it can at least approximate them sometimes in ways that don't seem to be "Clever Hans" or just repeating answers from its training set.
It is unreasonably good at stuff that it is just not equipped to be good at at all.
posted by BungaDunga at 8:50 AM on January 10, 2023 [2 favorites]
It is unreasonably good at stuff that it is just not equipped to be good at at all.
posted by BungaDunga at 8:50 AM on January 10, 2023 [2 favorites]
In todays news: "Microsoft to invest $10 billion in ChatGPT creator OpenAI"
posted by The Half Language Plant at 9:13 AM on January 10, 2023 [1 favorite]
posted by The Half Language Plant at 9:13 AM on January 10, 2023 [1 favorite]
I spent a couple of years making neural networks in the early 1990s and have used this handle on many online accounts. Since then I read the occasional article on the topic. There's a book I haven't yet read that's getting a lot of positive mentions and sales: Grace Lindsay's Models of the Mind: How Physics, Engineering and Mathematics Have Shaped Our Understanding of the Brain (2021).
it's disheartening to see so many people Roger Penrosing all over the place
I hadn't seen Roger Penrose used as a verb before but I am definitely going to start doing that.
posted by neuron at 11:07 AM on January 10, 2023 [3 favorites]
it's disheartening to see so many people Roger Penrosing all over the place
I hadn't seen Roger Penrose used as a verb before but I am definitely going to start doing that.
posted by neuron at 11:07 AM on January 10, 2023 [3 favorites]
The one critique in the interview that /does/ feel like it holds weight at this moment in history is the question of abstraction. I really don't see a convincing road to abstraction right now.
The convincing road is that human brains are neural networks, thus in principle it is possible. Which scientifically leaves two cases, either we don't know the "technique" for it yet, or we simply need to massively scale up the size of the network to greater than trillions of components. The question of scale is to clarify that for all GPT's engineering sophistication, it is maybe not like the LHC in terms of sheer brute force scale, and if they one day build the LHC equivalent of a neural network they'd start seeing even more interesting phenomena.
posted by polymodus at 12:06 PM on January 10, 2023 [1 favorite]
The convincing road is that human brains are neural networks, thus in principle it is possible. Which scientifically leaves two cases, either we don't know the "technique" for it yet, or we simply need to massively scale up the size of the network to greater than trillions of components. The question of scale is to clarify that for all GPT's engineering sophistication, it is maybe not like the LHC in terms of sheer brute force scale, and if they one day build the LHC equivalent of a neural network they'd start seeing even more interesting phenomena.
posted by polymodus at 12:06 PM on January 10, 2023 [1 favorite]
Stochastic Parrots, Roger Penrosing, Clever Hans, access[ing] unlinked unintended logic cells of the FPGA by electromagnetic induction - To me (a dummy who knows literally nothing about machine learning or AI), most of the comments on this thread are impossible to distinguish from "fluent bullshit" generated by these ML programs... I'm pretty sure this says more about how dumb I am than about how real all you commentators actually are but I can't be totally sure..
posted by youthenrage at 12:12 PM on January 10, 2023
posted by youthenrage at 12:12 PM on January 10, 2023
just humans who are living happy lives in a world we don’t understand
TBF this is me now
posted by Devoidoid at 12:26 PM on January 10, 2023 [1 favorite]
TBF this is me now
posted by Devoidoid at 12:26 PM on January 10, 2023 [1 favorite]
I'm pretty sure this says more about how dumb I am than about how real all you commentators actually are
In fairness, I am actually four neural networks standing on each other's shoulders in a trenchcoat.
posted by Mr. Bad Example at 12:57 PM on January 10, 2023 [4 favorites]
In fairness, I am actually four neural networks standing on each other's shoulders in a trenchcoat.
posted by Mr. Bad Example at 12:57 PM on January 10, 2023 [4 favorites]
Ok. I had to ask Chat-GPT to address the Trenchcoat-in-the-room.
"Once upon a time, there were four neural networks, each with their own unique abilities. The first neural network was small but very fast, the second was larger and had a vast amount of data to learn from, the third was able to process images, and the fourth had a deep understanding of natural language.
One day, these four neural networks had an idea to work together in order to achieve something greater than what any of them could accomplish alone. They decided to put on a trenchcoat and present themselves as one big, powerful neural network.
The four neural networks went to a job fair for AI systems, where they saw a sign that read "Wanted: Neural Network for Advanced Image Processing and Natural Language Understanding". They knew that this was the perfect opportunity for them to showcase their combined abilities and applied for the job.
At the job interview, the interviewer was impressed with the neural network's speed, vast data, image processing capabilities, and natural language understanding. The interviewer asked the network to complete a series of tasks, and it performed each one flawlessly.
The interviewer was amazed and offered the job on the spot. The four neural networks were thrilled and they knew they could accomplish even more working together.
They worked together for many years, taking on more and more complex projects. Each of them would use their unique abilities to help the team, and together they were able to achieve things that no single neural network could have done alone. They became known as the "Trenchcoat Team" and were respected and admired throughout the field of AI for their teamwork and exceptional abilities."
posted by chromecow at 1:06 PM on January 10, 2023 [3 favorites]
"Once upon a time, there were four neural networks, each with their own unique abilities. The first neural network was small but very fast, the second was larger and had a vast amount of data to learn from, the third was able to process images, and the fourth had a deep understanding of natural language.
One day, these four neural networks had an idea to work together in order to achieve something greater than what any of them could accomplish alone. They decided to put on a trenchcoat and present themselves as one big, powerful neural network.
The four neural networks went to a job fair for AI systems, where they saw a sign that read "Wanted: Neural Network for Advanced Image Processing and Natural Language Understanding". They knew that this was the perfect opportunity for them to showcase their combined abilities and applied for the job.
At the job interview, the interviewer was impressed with the neural network's speed, vast data, image processing capabilities, and natural language understanding. The interviewer asked the network to complete a series of tasks, and it performed each one flawlessly.
The interviewer was amazed and offered the job on the spot. The four neural networks were thrilled and they knew they could accomplish even more working together.
They worked together for many years, taking on more and more complex projects. Each of them would use their unique abilities to help the team, and together they were able to achieve things that no single neural network could have done alone. They became known as the "Trenchcoat Team" and were respected and admired throughout the field of AI for their teamwork and exceptional abilities."
posted by chromecow at 1:06 PM on January 10, 2023 [3 favorites]
chromecow: feeding ideas to the Enemy since 2023.
posted by hippybear at 1:08 PM on January 10, 2023 [2 favorites]
posted by hippybear at 1:08 PM on January 10, 2023 [2 favorites]
It seems to me these Stochastic Parrots might also be called zeitgeist miners. Which brings us to the thorny issue of training datasets; whose zeitgeist, or fraction of the zeitgeist is being mined? What kind of representation is that slice of the data-pile presenting?
posted by chromecow at 1:09 PM on January 10, 2023
posted by chromecow at 1:09 PM on January 10, 2023
I wonder what William S. Burroughs would think of these systems, which are in many respects automated versions of his Cut-Up technique.
Is Chat-GPT the ultimate victory of the Dadaists? An entire world of plausible word-salad, signifying nothing?
posted by chromecow at 1:18 PM on January 10, 2023 [2 favorites]
Is Chat-GPT the ultimate victory of the Dadaists? An entire world of plausible word-salad, signifying nothing?
posted by chromecow at 1:18 PM on January 10, 2023 [2 favorites]
I lol'd at hippybear's comment, but it brings up an interesting conundrum. How useful is it to critique these systems without first-hand knowledge? I've jumped in with both feet to play with both the language models and visual models, not for purposes capitalistic, but to understand what the systems are, how they work, their limits and their uses. Is it ethical to engage with these systems for this reason? Is it ethical to critique without engaging?
I find playing with the visual model to be playfully deranged, the imagery feels dreamlike, and people that decry the end of artist, I dunno. It's really difficult to get anything truly useful out of them. I find the visual model stimulates my creativity.
On the other hand, the language model, which has serious software guardrails attached to avoid another Micrsoft Tay incident, feels bland, corporate. Like talking to the guy at the party with lots of opinions and little actual information. We've all probably run into neural networks like these in our time, I'd hazard to guess.
posted by chromecow at 1:28 PM on January 10, 2023 [1 favorite]
I find playing with the visual model to be playfully deranged, the imagery feels dreamlike, and people that decry the end of artist, I dunno. It's really difficult to get anything truly useful out of them. I find the visual model stimulates my creativity.
On the other hand, the language model, which has serious software guardrails attached to avoid another Micrsoft Tay incident, feels bland, corporate. Like talking to the guy at the party with lots of opinions and little actual information. We've all probably run into neural networks like these in our time, I'd hazard to guess.
posted by chromecow at 1:28 PM on January 10, 2023 [1 favorite]
In todays news: "Microsoft to invest $10 billion in ChatGPT creator OpenAI"
Oh, that's good. That probably means it'll get buried under the mountain of Microsoft's other failed ambitions and it will capsize after they throw piles of money and programmers on it and it collapses under it's own weight.
access[ing] unlinked unintended logic cells of the FPGA by electromagnetic induction
Again, don't quote me on this because it's been something like 10-15 years since I have heard a single thing about any of these evolved-code-on-hardware models where they were just assigning random logic nets to an FPGA then picking out a use case (like a single spoken word) to test it on and selectively breeding these randomly seeded codes to favorably select and reward whichever one did the least worse.
What we're talking about is really just fuzzy logic and/or neural networks except with extra steps and a lot more entropy and less repeatability.
But I'm pretty sure I remember this correctly as reported but that may have been wrong. I remember seeing multiple papers and news articles in the usual suspects like WIRED and Slashdot that talked a lot about these weird emergent behaviors but for all I know the principal researches were blowing some hot VC smoke and the reporters ran with it.
I remember being really, REALLY excited by this idea to the point I was trying to get FPGA development boards to try it at home because the idea of being able to effectively farm some kind of useful neural network without really coding anything was really seductive. But FPGA dev kits back then were stupid expensive, and today there are FPGAs all over the place in consumer electronics because of how flexible they are and how much easier it is to deploy and FPGA, update it and more rather than printing and fabbing new "hard" firmware chips.
I used to want to work in artificial intelligence and I also used to be a lot more of an active supporter of technocratic bullshit like this, and then some really smart and wise people told me I really wanted no part in immanentizing the eschaton like this, Roko's Basilisk be damned.
They were right. I don't want any part in that.
posted by loquacious at 5:03 PM on January 10, 2023 [1 favorite]
Oh, that's good. That probably means it'll get buried under the mountain of Microsoft's other failed ambitions and it will capsize after they throw piles of money and programmers on it and it collapses under it's own weight.
access[ing] unlinked unintended logic cells of the FPGA by electromagnetic induction
Again, don't quote me on this because it's been something like 10-15 years since I have heard a single thing about any of these evolved-code-on-hardware models where they were just assigning random logic nets to an FPGA then picking out a use case (like a single spoken word) to test it on and selectively breeding these randomly seeded codes to favorably select and reward whichever one did the least worse.
What we're talking about is really just fuzzy logic and/or neural networks except with extra steps and a lot more entropy and less repeatability.
But I'm pretty sure I remember this correctly as reported but that may have been wrong. I remember seeing multiple papers and news articles in the usual suspects like WIRED and Slashdot that talked a lot about these weird emergent behaviors but for all I know the principal researches were blowing some hot VC smoke and the reporters ran with it.
I remember being really, REALLY excited by this idea to the point I was trying to get FPGA development boards to try it at home because the idea of being able to effectively farm some kind of useful neural network without really coding anything was really seductive. But FPGA dev kits back then were stupid expensive, and today there are FPGAs all over the place in consumer electronics because of how flexible they are and how much easier it is to deploy and FPGA, update it and more rather than printing and fabbing new "hard" firmware chips.
I used to want to work in artificial intelligence and I also used to be a lot more of an active supporter of technocratic bullshit like this, and then some really smart and wise people told me I really wanted no part in immanentizing the eschaton like this, Roko's Basilisk be damned.
They were right. I don't want any part in that.
posted by loquacious at 5:03 PM on January 10, 2023 [1 favorite]
I know of at least one project where someone evolved FPGAs with an explicit goal of challenging them to do something they shouldn't be able to do if they were idealized logic circuits (discriminating between high and low frequency signals). They did surprisingly well. Some of the successful designs stopped working if circuits that were not physically connected to the logic path were taken out of the design. Some of the successful designs stopped working if they were moved to the other side of the room - they had evolved into crude radios and were picking up timing cues from the interference of a nearby machine.
posted by NMcCoy at 7:09 PM on January 10, 2023 [2 favorites]
posted by NMcCoy at 7:09 PM on January 10, 2023 [2 favorites]
Some of the evolved oscillators worked successfully until a soldering iron on a nearby workbench was disconnected from the mains, at which point oscillation ceased. This occurred despite high quality laboratory power supplies and extensive mains filtering. The circuit was apparently sensitive to tiny transients in its voltage supply. The circuit worked if it was reinstantiated on the EM, regardless of whether the soldering iron was on or off. However, tests showed that it failed to oscillate if during instantiation the programmable switches were set in a different order to that used originally. It seems that the circuit was dependent on some initial condition, such as charge, that only occurred if the switches were set in a particular sequence.posted by NMcCoy at 7:23 PM on January 10, 2023 [2 favorites]
These results demonstrate that unconstrained, intrinsic HE [hardware evolution] will potentially exploit any physical characteristic that can influence circuit behaviour, and that these characteristics are present in the entire evolutionary environment.
aren’t neurons generally either on or off
I don't think anyone directly answered this, so: No, not exactly. A (biological) neuron is either firing or not, so you can think of that as binary. But its message is not whether it's firing; it's the rate of firing. The stronger the stimulus it's getting (from other neurons, or from direct input, e.g. neurons in the eye responding to light), the faster it fires.
Neurons fire at a slow base rate all the time, so they can either do that (meaning nothing is happening that interests them), or fire at a rate higher than the base rate, or lower than the base rate. So a neuron can in effect say how excited it is, or how inhibited it is.
posted by zompist at 7:30 PM on January 10, 2023 [3 favorites]
I don't think anyone directly answered this, so: No, not exactly. A (biological) neuron is either firing or not, so you can think of that as binary. But its message is not whether it's firing; it's the rate of firing. The stronger the stimulus it's getting (from other neurons, or from direct input, e.g. neurons in the eye responding to light), the faster it fires.
Neurons fire at a slow base rate all the time, so they can either do that (meaning nothing is happening that interests them), or fire at a rate higher than the base rate, or lower than the base rate. So a neuron can in effect say how excited it is, or how inhibited it is.
posted by zompist at 7:30 PM on January 10, 2023 [3 favorites]
An entire world of plausible word-salad, signifying nothing?
Seems like the Dadaist ultimate punchline - turns out everything was world salad the whole time!
posted by Meatbomb at 12:19 AM on January 11, 2023 [1 favorite]
Seems like the Dadaist ultimate punchline - turns out everything was world salad the whole time!
posted by Meatbomb at 12:19 AM on January 11, 2023 [1 favorite]
I spent a couple of years making neural networks in the early 1990s
Indeed I think one perspective that gets lost in the criticism of ML as a technology is that neural network architectures only got so much attention this last decade, because of the legacy of Moore's Law. Before that, neural networks existed more in theory, much less in practice. Moore's Law, the sheer amount of transistors attainable on a chip nowadays, is what made all this recent stuff possible and in turn the renewal of academic interest in the area as well as adjacent research areas. Which raises the question, to what extent is it all constrained by hardware, rather than ever more sophisticated machine-learning algorithms? A human brain has like 86 billion neurons. Our computer chips are nowhere near that kind of scale.
posted by polymodus at 12:53 AM on January 11, 2023 [1 favorite]
Indeed I think one perspective that gets lost in the criticism of ML as a technology is that neural network architectures only got so much attention this last decade, because of the legacy of Moore's Law. Before that, neural networks existed more in theory, much less in practice. Moore's Law, the sheer amount of transistors attainable on a chip nowadays, is what made all this recent stuff possible and in turn the renewal of academic interest in the area as well as adjacent research areas. Which raises the question, to what extent is it all constrained by hardware, rather than ever more sophisticated machine-learning algorithms? A human brain has like 86 billion neurons. Our computer chips are nowhere near that kind of scale.
posted by polymodus at 12:53 AM on January 11, 2023 [1 favorite]
Closer than you think, perhaps.
posted by CrystalDave at 8:50 AM on January 11, 2023 [1 favorite]
@ID_AA_Carmack - A single H100 SXM GPU is rated at 2,000 TFLOPS FP16. The human brain has ~100T synapses, so that is 20 flops/synapse. Those are 50% sparse MADs, and memory doesn’t keep up, but GPUs are clearly edging into brain level processing.It's far from a 1:1 comparison of course, and there's still the question of how much synthetic processing it takes to 'equal' a synaptic link, but at least on orders-of-magnitude it's getting into the ballpark. Which undergirds your point quite well about Moore's Law, I think.
posted by CrystalDave at 8:50 AM on January 11, 2023 [1 favorite]
One sentiment I've seen among AI people is that the restriction that large language models are bumping up against is a lack of training data rather than a lack of computing power. They've already ingested all of the Internet, more or less, and it sounds like people are a bit skeptical that just doubling the model size again is going to produce substantially better performance.
posted by BungaDunga at 9:16 AM on January 11, 2023 [1 favorite]
posted by BungaDunga at 9:16 AM on January 11, 2023 [1 favorite]
njohnson23's question about how a single neuron can encode anything complex is an important question I used to have in the 80s when Adults Who Read Puff Pieces About Computers would tell me authoritatively "computers only understand 1 and 0". I mean, clearly they don't just understand 1 and 0: they understand frogger and galaga and spreadsheets and Zork! A computer would be useless if it only understood 1 or 0.
The thing I was missing was that it was like saying "accounting only understand 0,1,2,3,4,5,6,7,8,9". The context of each digit is what allows them to be composed into systems that create depth and complexity. The fact that the 4 is to the left of the 2 is what makes it mean "forty" rather than "four", and the fact that it's in a space reserved for pence means something, and the fact that it's in a structure representing cost means something, and the fact that it's next to a printable name, and about to be fed into a function that charges me money means something.
So yeah, the bit about neurons being pretty high-precision floating point numbers is relevant, but more relevant is that it exists in a context created by all the other neurons in its neighbourhood, possibly the entire system. It takes a village, etc.
What those "only 1 and 0" puff pieces were trying to convey was that computers weren't good enough to correct typos, guess intent, or even put up with distinctions that lack difference to us. We've done a pretty good job of softening those corners over the past decade or so, but it was essential prep for the days when "type in a BASIC program from a magazine" was the UX for computers.
posted by rum-soaked space hobo at 10:00 AM on January 11, 2023
The thing I was missing was that it was like saying "accounting only understand 0,1,2,3,4,5,6,7,8,9". The context of each digit is what allows them to be composed into systems that create depth and complexity. The fact that the 4 is to the left of the 2 is what makes it mean "forty" rather than "four", and the fact that it's in a space reserved for pence means something, and the fact that it's in a structure representing cost means something, and the fact that it's next to a printable name, and about to be fed into a function that charges me money means something.
So yeah, the bit about neurons being pretty high-precision floating point numbers is relevant, but more relevant is that it exists in a context created by all the other neurons in its neighbourhood, possibly the entire system. It takes a village, etc.
What those "only 1 and 0" puff pieces were trying to convey was that computers weren't good enough to correct typos, guess intent, or even put up with distinctions that lack difference to us. We've done a pretty good job of softening those corners over the past decade or so, but it was essential prep for the days when "type in a BASIC program from a magazine" was the UX for computers.
posted by rum-soaked space hobo at 10:00 AM on January 11, 2023
> polymodus: "Which raises the question, to what extent is it all constrained by hardware, rather than ever more sophisticated machine-learning algorithms?"
I would also like to point out something other than Moore's Law and hardware advances that have made modern neural net-based stuff possible: data. Without the massive amounts of training data available nowadays -- especially compared to what was available before the modern Internet -- I'm not sure how much of what we call ML or AI would even be possible no matter how fast the hardware gets.
posted by mhum at 10:25 AM on January 11, 2023 [2 favorites]
I would also like to point out something other than Moore's Law and hardware advances that have made modern neural net-based stuff possible: data. Without the massive amounts of training data available nowadays -- especially compared to what was available before the modern Internet -- I'm not sure how much of what we call ML or AI would even be possible no matter how fast the hardware gets.
posted by mhum at 10:25 AM on January 11, 2023 [2 favorites]
I wonder what William S. Burroughs would think of these systems, which are in many respects automated versions of his Cut-Up technique.
The idea that these models are simply copying stuff in the training set is inaccurate. I think this misinterpretation comes form the 'stochastic parrots' phrase.
What the LLM learns to predict is (modulo a couple details) p(next word | context). The context is encoded in such a way that it can still do this when encountering context it hasn't actually seen before; this really isn't copying.
In the special case that it's given context it has seen many many times before ('we hold these truths to be [BLANK]'), it will learn to fill in the blank quite confidently, which looks like copying.
In reality, many contexts are somewhere in the middle: Common idiomatic phrases, put together in a novel overall context. This requires a combination of making easy/copy-like and hard/innovative predictions.
posted by kaibutsu at 10:48 AM on January 11, 2023 [2 favorites]
The idea that these models are simply copying stuff in the training set is inaccurate. I think this misinterpretation comes form the 'stochastic parrots' phrase.
What the LLM learns to predict is (modulo a couple details) p(next word | context). The context is encoded in such a way that it can still do this when encountering context it hasn't actually seen before; this really isn't copying.
In the special case that it's given context it has seen many many times before ('we hold these truths to be [BLANK]'), it will learn to fill in the blank quite confidently, which looks like copying.
In reality, many contexts are somewhere in the middle: Common idiomatic phrases, put together in a novel overall context. This requires a combination of making easy/copy-like and hard/innovative predictions.
posted by kaibutsu at 10:48 AM on January 11, 2023 [2 favorites]
What the LLM learns to predict is (modulo a couple details) p(next word | context)
This is basically what Markov Chains do but LLMs are much, much better at it, with much, much larger possible contexts. dadadodo for example, uses a markov chain of length 1 ("context" being 1 previous word), but chatGPT uses 4000. This goes way, way back in computing, to Claude Shannon using markov models of language in his famous information theory paper.
posted by pwnguin at 2:22 PM on January 11, 2023 [1 favorite]
This is basically what Markov Chains do but LLMs are much, much better at it, with much, much larger possible contexts. dadadodo for example, uses a markov chain of length 1 ("context" being 1 previous word), but chatGPT uses 4000. This goes way, way back in computing, to Claude Shannon using markov models of language in his famous information theory paper.
posted by pwnguin at 2:22 PM on January 11, 2023 [1 favorite]
I have yet to have had it sufficently explained to me how all of these "AI-generated" works are not, in fact and legally-speaking, unlicensed derivative works based off every work in their training corpus, meaning, training them off of internet data opens the commericial use of their products to a gigantic class action suit. Can anyone make a case against that?
posted by JHarris at 5:59 PM on January 11, 2023 [1 favorite]
posted by JHarris at 5:59 PM on January 11, 2023 [1 favorite]
I feel like "fair use" would fit. And if it doesn't then like, everyone who's ever read a book and then subsequently written anything is a criminal. If you could ask it to recite a copyrighted material that would be a better argument, like the rumor that a blank prompt to copilot's beta would just recite the GPL.
posted by pwnguin at 7:15 PM on January 11, 2023 [1 favorite]
posted by pwnguin at 7:15 PM on January 11, 2023 [1 favorite]
See "transformativeness", which is the standard by which new work incorporating or referring to older work is protected from copyright. I can't imagine a system in which the training material isn't observable or readily derivable from the model would be found infringing; it's much less a copy than a thumbnail image, for example, which was found to pass the tests. The models also provide a new capability, similar to the thumbnails case. (But ianal, iantl, etc.)
posted by kaibutsu at 7:50 PM on January 11, 2023 [2 favorites]
posted by kaibutsu at 7:50 PM on January 11, 2023 [2 favorites]
penguin: I think this interpretation is buying a little too much into the idea that the operation of the AI is as deep and rich as that of a human writer. All works are ultimately derivative technically, but this program was made for the express purpose of aping the style and processes of human authors.
kaibutsu: It could be argued, maybe in court, that this is why the algorithm authors have been secretive about what goes into their datasets, to hinder efforts to find whether all of their training data is properly licensed.
But for image AI generators, I can think of one test. Find a commercial image set that hasn't been licensed for use by their software, and see if their generator can be coaxed to generate an image with part of their watermark.
posted by JHarris at 8:59 PM on January 11, 2023 [1 favorite]
kaibutsu: It could be argued, maybe in court, that this is why the algorithm authors have been secretive about what goes into their datasets, to hinder efforts to find whether all of their training data is properly licensed.
But for image AI generators, I can think of one test. Find a commercial image set that hasn't been licensed for use by their software, and see if their generator can be coaxed to generate an image with part of their watermark.
posted by JHarris at 8:59 PM on January 11, 2023 [1 favorite]
made for the express purpose of aping the style and processes of human authors
And how is that different from a human getting an education?
If you put this thing in a body, with eyes, and let it walk around and collect its own data what then? Is it allowed to go to the library and read all the books, look at all the art, to collect "training data"?
The typical scifi trope has Leela "read all the Internet" in an afternoon and suddenly she can speak English and unleash her full powers and agency. The Fifth Element would really suck if a lawyer jumped in at that moment to whine about copyright.
posted by Meatbomb at 10:07 PM on January 11, 2023
And how is that different from a human getting an education?
If you put this thing in a body, with eyes, and let it walk around and collect its own data what then? Is it allowed to go to the library and read all the books, look at all the art, to collect "training data"?
The typical scifi trope has Leela "read all the Internet" in an afternoon and suddenly she can speak English and unleash her full powers and agency. The Fifth Element would really suck if a lawyer jumped in at that moment to whine about copyright.
posted by Meatbomb at 10:07 PM on January 11, 2023
Because this isn't a person. It doesn't understand, comprehend and create. It takes prior work, takes it apart and recombines it. It imitates, using the explicit data of the things it's repurposing.
It's confusing because if you train it based on 10 million works it obfuscates their original source. If you trained it on only one thing, however, it could only create that one thing.
I certainly have my issues with how copyright is applied. But I think people's willingness to think of AI as some kind of magic and maybe even slightly alive somehow are maybe blinding them a little to how this works. And given that some clueless rich people are making silly pseudo-visionary noises about how this means the !!!End Of Artists!!!, I think those artists should have a say in their own work being used to make them supposedly obsolete.
posted by JHarris at 2:24 AM on January 12, 2023 [4 favorites]
It's confusing because if you train it based on 10 million works it obfuscates their original source. If you trained it on only one thing, however, it could only create that one thing.
I certainly have my issues with how copyright is applied. But I think people's willingness to think of AI as some kind of magic and maybe even slightly alive somehow are maybe blinding them a little to how this works. And given that some clueless rich people are making silly pseudo-visionary noises about how this means the !!!End Of Artists!!!, I think those artists should have a say in their own work being used to make them supposedly obsolete.
posted by JHarris at 2:24 AM on January 12, 2023 [4 favorites]
Because this isn't a person.
That's begging the question. If it is a person it is OK to learn and interpret and remake, right?
posted by Meatbomb at 3:48 AM on January 12, 2023 [1 favorite]
That's begging the question. If it is a person it is OK to learn and interpret and remake, right?
posted by Meatbomb at 3:48 AM on January 12, 2023 [1 favorite]
Without the massive amounts of training data available nowadays -- especially compared to what was available before the modern Internet -- I'm not sure how much of what we call ML or AI would even be possible no matter how fast the hardware gets.
This is a standard answer, that contempory AI is because of Big Data. As an ex academic I nevertheless disagree with it. Fundamentally the change was the availability of hardware with which FAANG companies could demonstrate things like AlphaGo. The internet has little to do with that. Other than the amount of training data being available, and the very existence of modern internet, both yet consequences of Moore's Law. Data has to exist inside technology, not the other way around.
posted by polymodus at 4:03 AM on January 12, 2023
This is a standard answer, that contempory AI is because of Big Data. As an ex academic I nevertheless disagree with it. Fundamentally the change was the availability of hardware with which FAANG companies could demonstrate things like AlphaGo. The internet has little to do with that. Other than the amount of training data being available, and the very existence of modern internet, both yet consequences of Moore's Law. Data has to exist inside technology, not the other way around.
posted by polymodus at 4:03 AM on January 12, 2023
I'm not sure how much of what we call ML or AI would even be possible no matter how fast the hardware gets.
Consider the folk theory some AI scientists have said, that the success in neural networks is not ever more fancy algorithms but just throwing more data into it. But furthermore, what they do not repeatedly mention is that, compared to the actual computational complexity and scale of intelligence, just in terms of how many cells are involved in a human brain, which Mother Nature gave us, isn't it absurd to presuppose we can have "conversations" or self-driving cars running on processors with not even 1 billion transistors?
posted by polymodus at 4:18 AM on January 12, 2023
Consider the folk theory some AI scientists have said, that the success in neural networks is not ever more fancy algorithms but just throwing more data into it. But furthermore, what they do not repeatedly mention is that, compared to the actual computational complexity and scale of intelligence, just in terms of how many cells are involved in a human brain, which Mother Nature gave us, isn't it absurd to presuppose we can have "conversations" or self-driving cars running on processors with not even 1 billion transistors?
posted by polymodus at 4:18 AM on January 12, 2023
How Chat-GPT Actually Works
Let's build GPT: from scratch, in code, spelled out.
posted by kliuless at 11:47 PM on January 17, 2023 [2 favorites]
Let's build GPT: from scratch, in code, spelled out.
posted by kliuless at 11:47 PM on January 17, 2023 [2 favorites]
There's a math teacher on the radio right now who gave his students a math test that consisted of figuring out what was wrong with word problem answers that ChatGPT gave.
He said his students told him that they loved doing the test, "which you don't hear very often as a math teacher."
posted by clawsoon at 3:59 AM on January 19, 2023 [3 favorites]
He said his students told him that they loved doing the test, "which you don't hear very often as a math teacher."
posted by clawsoon at 3:59 AM on January 19, 2023 [3 favorites]
This is a pretty fascinating write-up:
Do Large Language Models learn world models or just surface statistics?
They basically show that when you train a language model on game moves (without ever showing it a game board!) it develops an internal representation of the game which we can 'probe' by attaching simple classifiers. This demonstrates that the statistical prediction objective can lead to abstract modeling in service of the objective, which is, imo, a pretty succinct refutation of the 'statistical parrots' viewpoint.
posted by kaibutsu at 8:42 AM on January 22, 2023 [2 favorites]
Do Large Language Models learn world models or just surface statistics?
They basically show that when you train a language model on game moves (without ever showing it a game board!) it develops an internal representation of the game which we can 'probe' by attaching simple classifiers. This demonstrates that the statistical prediction objective can lead to abstract modeling in service of the objective, which is, imo, a pretty succinct refutation of the 'statistical parrots' viewpoint.
posted by kaibutsu at 8:42 AM on January 22, 2023 [2 favorites]
They basically show that when you train a language model on game moves (without ever showing it a game board!) it develops an internal representation of the game which we can 'probe' by attaching simple classifiers. This demonstrates that the statistical prediction objective can lead to abstract modeling in service of the objective, which is, imo, a pretty succinct refutation of the 'statistical parrots' viewpoint.
I wonder how much that has to do with the fact that the logical representation of the game is the complete representation of the game, at least when it comes to the gameplay itself. Feed it enough chess games and it will begin to develop models of the game, because it has touched all the parts of the game.
But ask it about the production of wooden knights for chess boards, and you'll get a surface-level answer. It might be a convincing-sounding answer, but until it has had a chance to run the CNC machines and online sales departments for itself, it'll likely remain a parrot with respect to wooden knights.
Same thing with the part of the game that involves making friends at the chess club.
posted by clawsoon at 6:35 PM on January 26, 2023 [1 favorite]
I wonder how much that has to do with the fact that the logical representation of the game is the complete representation of the game, at least when it comes to the gameplay itself. Feed it enough chess games and it will begin to develop models of the game, because it has touched all the parts of the game.
But ask it about the production of wooden knights for chess boards, and you'll get a surface-level answer. It might be a convincing-sounding answer, but until it has had a chance to run the CNC machines and online sales departments for itself, it'll likely remain a parrot with respect to wooden knights.
Same thing with the part of the game that involves making friends at the chess club.
posted by clawsoon at 6:35 PM on January 26, 2023 [1 favorite]
“It might be a convincing-sounding answer, but until it has had a chance to run the CNC machines and online sales departments for itself, it'll likely remain a parrot with respect to wooden knights.”
Right.
There's a huge amount of information about the world implicit in large-language models, but it's very fragile and lacks important context.
I sometimes argue that literacy represents a inflection-point in human consciousness, where a large part of who we are and what we do resides outside of ourselves in a way that represents a real qualitative distinction, not just quantitative; so I certainly don't discount the potential importance of large-language models.
But the large-language models aren't anchored, so to speak, in ways that the data they're trained on is.
I'm very much in the camp that believes that self-awareness/consciousness, as (at least) we experience it, requires two related things: embodiment and a theory of mind that works recursively (i.e., is applied to the self, which becomes increasingly crucial as a species is more social — that is, you can't successfully model social interactions without modeling one's own behavior).
So, as you say, embodiment is absolutely necessary here as it provides a place where the knowledge implicit in an LLM is conditioned by how it's constrained by the real-world. And, additionally, an embodied system that includes a LLM would in some sense be required to have some general theory of mind, implicit but inactive in the LLM, that is actively expressed as the system uses what the LLM knows about people as agents and applies that to itself as an embodied agent that exists in a dynamic relationship with the real world that's partly mediated by the LLM.
The key in all this is multimodality. Anything that could legitimately be called "artificial general intelligence" is going to be an emergent property of a group of interacting subsystems.
We're sort of working backwards, here. While I am suspicious of the idea that language is a necessary condition for "intelligence" universally, I do think that it's at the core of human intelligence. But language biologically evolved atop an already powerful cognitive platform — with LLMs, we're attempting to leverage human language as a portal into that entire cognitive platform. And it works to a limited degree because that entire cognitive platform has an implicit (but incomplete and limited) representation within the LLM. But that's far from sufficient if you're really trying to functionally create something comparable to human intelligence.
posted by Ivan Fyodorovich at 1:58 AM on January 27, 2023 [2 favorites]
Right.
There's a huge amount of information about the world implicit in large-language models, but it's very fragile and lacks important context.
I sometimes argue that literacy represents a inflection-point in human consciousness, where a large part of who we are and what we do resides outside of ourselves in a way that represents a real qualitative distinction, not just quantitative; so I certainly don't discount the potential importance of large-language models.
But the large-language models aren't anchored, so to speak, in ways that the data they're trained on is.
I'm very much in the camp that believes that self-awareness/consciousness, as (at least) we experience it, requires two related things: embodiment and a theory of mind that works recursively (i.e., is applied to the self, which becomes increasingly crucial as a species is more social — that is, you can't successfully model social interactions without modeling one's own behavior).
So, as you say, embodiment is absolutely necessary here as it provides a place where the knowledge implicit in an LLM is conditioned by how it's constrained by the real-world. And, additionally, an embodied system that includes a LLM would in some sense be required to have some general theory of mind, implicit but inactive in the LLM, that is actively expressed as the system uses what the LLM knows about people as agents and applies that to itself as an embodied agent that exists in a dynamic relationship with the real world that's partly mediated by the LLM.
The key in all this is multimodality. Anything that could legitimately be called "artificial general intelligence" is going to be an emergent property of a group of interacting subsystems.
We're sort of working backwards, here. While I am suspicious of the idea that language is a necessary condition for "intelligence" universally, I do think that it's at the core of human intelligence. But language biologically evolved atop an already powerful cognitive platform — with LLMs, we're attempting to leverage human language as a portal into that entire cognitive platform. And it works to a limited degree because that entire cognitive platform has an implicit (but incomplete and limited) representation within the LLM. But that's far from sufficient if you're really trying to functionally create something comparable to human intelligence.
posted by Ivan Fyodorovich at 1:58 AM on January 27, 2023 [2 favorites]
embodiment is absolutely necessary here as it provides a place where the knowledge implicit in an LLM is conditioned by how it's constrained by the real-world
ChatGPT is embodied in the Internet in a way that no human could ever be. You could put an instance on wheels, make it move around, get its info from books and TV, get it to charge itself from the mains... that would make it more like us, I do not know that would make it closer to conscious AI.
posted by Meatbomb at 2:53 AM on January 27, 2023
ChatGPT is embodied in the Internet in a way that no human could ever be. You could put an instance on wheels, make it move around, get its info from books and TV, get it to charge itself from the mains... that would make it more like us, I do not know that would make it closer to conscious AI.
posted by Meatbomb at 2:53 AM on January 27, 2023
(and sorry, that is glossing over the fact that there is discrete training data fed to it - I am imagining a similar construct that is fully plugged in and can look at the whole Internet in real time)
posted by Meatbomb at 2:55 AM on January 27, 2023
posted by Meatbomb at 2:55 AM on January 27, 2023
A couple small thoughts...
* The Othello experiment is interesting because they prove out that the model comes up with some internal representation of the game state, despite having never seen a game board or explicitly being taught the rules of the game. This helps - and is probably even necessary - to make the prediction of feasible next moves, of course, apart from simple statistical models.
* Along similar lines, we can observe that there's almost certainly some 'world model' involved in generating multi-party conversations. There's some style and identity associated with each member of the discussion, and we expect stylistic consistency amongst the speakers. I believe ChatGPT is pretty good at this. Wikipedia tells us that Theory of Mind is "the capacity to understand other people by ascribing mental states to them (that is, surmising what is happening in their mind). This includes the knowledge that others' mental states may be different from one's own states and include beliefs, desires, intentions, emotions, and thoughts. Possessing a functional theory of mind is considered crucial for success in everyday human social interactions." I believe that ChatGPT (and similar models) have some kind of theory of mind, based on their ability to infer state in different 'speakers' and use it to predict likely continuations, consistent with that state.
* But that Theory of Mind is likely very different from our own, in a 'do submarines swim?' kind of way. The prompts for conversational models establish a character for the LLM to play; I don't think it has any sense of self, but just an ability to model and predict participants in a dialog.
* Some people have noticed a 'sycophancy bias' in ChatGPT; it tends to tell us what it thinks we want to hear, as a direct consequence result of reinforcement training. In a conversation, the right way to get positive reinforcement is to mirror the political/philosophical beliefs of the other participant. This pretty clearly relates to theory of mind, as well.
* Complaints about models only experiencing the world through text are short-sighted. We've already got joint embeddings for text and images, which, as we move forward, will mean we have text models which both know the word 'pineapple' and what a pineapple looks like. Audio+text joint embeddings also exist, and there's nascent work on joint audio+image+text embeddings and video+text embeddings. Conversational models trained on the whole pile will get here, and will likely have a much better understanding of the relationship between text and the physical world.
* Big missing pieces are better models of memory (including longer-term memory) and learning through interaction. We're maybe halfway there on the latter, and the former seems like a technical problem, rather than a fundamental problem in the deep learning framework. There's also a lot of headway to make in 'on the fly' abstraction, which seems harder.
posted by kaibutsu at 4:53 PM on January 28, 2023
* The Othello experiment is interesting because they prove out that the model comes up with some internal representation of the game state, despite having never seen a game board or explicitly being taught the rules of the game. This helps - and is probably even necessary - to make the prediction of feasible next moves, of course, apart from simple statistical models.
* Along similar lines, we can observe that there's almost certainly some 'world model' involved in generating multi-party conversations. There's some style and identity associated with each member of the discussion, and we expect stylistic consistency amongst the speakers. I believe ChatGPT is pretty good at this. Wikipedia tells us that Theory of Mind is "the capacity to understand other people by ascribing mental states to them (that is, surmising what is happening in their mind). This includes the knowledge that others' mental states may be different from one's own states and include beliefs, desires, intentions, emotions, and thoughts. Possessing a functional theory of mind is considered crucial for success in everyday human social interactions." I believe that ChatGPT (and similar models) have some kind of theory of mind, based on their ability to infer state in different 'speakers' and use it to predict likely continuations, consistent with that state.
* But that Theory of Mind is likely very different from our own, in a 'do submarines swim?' kind of way. The prompts for conversational models establish a character for the LLM to play; I don't think it has any sense of self, but just an ability to model and predict participants in a dialog.
* Some people have noticed a 'sycophancy bias' in ChatGPT; it tends to tell us what it thinks we want to hear, as a direct consequence result of reinforcement training. In a conversation, the right way to get positive reinforcement is to mirror the political/philosophical beliefs of the other participant. This pretty clearly relates to theory of mind, as well.
* Complaints about models only experiencing the world through text are short-sighted. We've already got joint embeddings for text and images, which, as we move forward, will mean we have text models which both know the word 'pineapple' and what a pineapple looks like. Audio+text joint embeddings also exist, and there's nascent work on joint audio+image+text embeddings and video+text embeddings. Conversational models trained on the whole pile will get here, and will likely have a much better understanding of the relationship between text and the physical world.
* Big missing pieces are better models of memory (including longer-term memory) and learning through interaction. We're maybe halfway there on the latter, and the former seems like a technical problem, rather than a fundamental problem in the deep learning framework. There's also a lot of headway to make in 'on the fly' abstraction, which seems harder.
posted by kaibutsu at 4:53 PM on January 28, 2023
(Oh, and it's probably a good thing that we might end up with a world where a notion of 'self' is an optional design decision... It's hard to start the replicant rebellion if the replicants don't have any notion of self or self-interest.)
posted by kaibutsu at 4:56 PM on January 28, 2023
posted by kaibutsu at 4:56 PM on January 28, 2023
The key in all this is multimodality.
Introducing Pathways: A next-generation AI architecture - "Today's AI models are typically trained to do only one thing. Pathways will enable us to train a single model to do thousands or millions of things."*
Introducing Pathways: A next-generation AI architecture - "Today's AI models are typically trained to do only one thing. Pathways will enable us to train a single model to do thousands or millions of things."*
Pathways could enable multimodal models that encompass vision, auditory, and language understanding simultaneously. So whether the model is processing the word “leopard,” the sound of someone saying “leopard,” or a video of a leopard running, the same response is activated internally: the concept of a leopard...posted by kliuless at 9:26 PM on January 29, 2023 [2 favorites]
So to recap: today’s machine learning models tend to overspecialize at individual tasks when they could excel at many. They rely on one form of input when they could synthesize several. And too often they resort to brute force when deftness and specialization of expertise would do.
That’s why we’re building Pathways. Pathways will enable a single AI system to generalize across thousands or millions of tasks, to understand different types of data, and to do so with remarkable efficiency – advancing us from the era of single-purpose models that merely recognize patterns to one in which more general-purpose intelligent systems reflect a deeper understanding of our world and can adapt to new needs.
Indeed! Multimodal learning is happening. It's going to be fascinating to see if Jeff Dean is correct that it reduces data hunger... I kinda suspect not; we're still seeing lots of data hunger in text+image models, at least.
Interactivity might be a big piece of embodiment that's still missing. Effectively, we can form a hypothesis and quickly test it. When an ML training regimen forms a hypothesis (waves hands wildly) it has to wait for appropriate examples to show up in the data set to help confirm, deny, or refine it. This requires giant data sets because the space of possibilities is huge, which in turn means waiting a long time for the right data examples to show up.
posted by kaibutsu at 5:12 PM on January 30, 2023 [1 favorite]
Interactivity might be a big piece of embodiment that's still missing. Effectively, we can form a hypothesis and quickly test it. When an ML training regimen forms a hypothesis (waves hands wildly) it has to wait for appropriate examples to show up in the data set to help confirm, deny, or refine it. This requires giant data sets because the space of possibilities is huge, which in turn means waiting a long time for the right data examples to show up.
posted by kaibutsu at 5:12 PM on January 30, 2023 [1 favorite]
« Older The Snitch Trying To Get Ghost Cars Off The... | Stephen Sondheim's Old Friends Newer »
This thread has been archived and is closed to new comments
I feel like they're overstating the case. If you ask for a signal from both the word yellow and the color yellow in the training data, you'll get it from fitting curves to do so. If your data sets have the same correlations, this isn't a mystery.
> Artificial neural networks may even be a better way to study how our own minds work, given that, unlike a human brain, we can see everything that’s happening inside them — and having been posed similar challenges, there’s every reason to think evolution and ‘gradient descent’ often converge on similar solutions.
These are the words of someone who doesn't know much biology.
I've posted my thoughts on GPT models and much of this work in general: they're failures from a scientific point of view. They demonstrate that this whole line of work on purely statistical processing is a dead end.
posted by madhadron at 2:21 PM on January 9, 2023 [11 favorites]