Lies and the lying AIs who tell them

February 14, 2023 8:11 PM

An explanation by AI safety researcher Robert Miles on why making Large Language Models like ChatGPT tell the truth is a more intractable problem than one might think. While Robert Miles usually goes into the more wide-reaching ramifications of developing general artificial intelligence (summary: Step 1: Invent an AGI, Step 2: Apocalypse), here he talks about the seemingly simple task of training a language model to tell the truth.

As a layperson, I wholeheartedly recommend his other videos on the broader topic of AI safety. This playlist is probably a decent place to start.

posted by tigrrrlily (58 comments total)

AI, what would robert miles listen to in the oncoming AGI apocalypse?

AI: "...."
posted by lalochezia at 8:28 PM on February 14, 2023

to tell its truth.
posted by wmo at 8:30 PM on February 14, 2023

LOL. It's really easy to explain...If you put garbage in, you get garbage out. Now go find the 45TB of truth needed to insert into a non-lying version of GTP3.
posted by gible at 9:39 PM on February 14, 2023 [7 favorites]

"Now go find the 45TB of truth needed to insert into a non-lying version of GTP3."

It's not even that simple. A statistical, semi-stochastic rearrangement of truth need not result in something true.
posted by walrus at 11:07 PM on February 14, 2023 [34 favorites]

Amusingly, last week I saw a demo of people using LLMs to find bugs in their training data set for some other models. But I'm pretty sure giving the AI a feedback loop to correct its own training data is how we get SkyNet, so🤞nobody gets any bright ideas.

But as an SRE I'm pretty sure we can adapt the model to find typos in YAML configs, which happens so regularly I can recall three typos we've found in the past month running in production. Example. This is the sort of thing spell checkers won't be good at because they tokenize on spaces and punctuation, and because dictionaries don't usually include words like "kubernetes" in their lexicon. Getting a custom language model would be extremely handy.
posted by pwnguin at 11:15 PM on February 14, 2023 [1 favorite]

This was a nice little video, thanks.

I think one of the most interesting and fruitful side effects of this "Year of LLMs" is that it's making a lot of people - from engineers to laypeople - ask hard epistemic questions, and for very practical reasons. And I think it's important not to smirk at the engineers for "discovering" epistemology, even though philosophers have been thinking about these problems for millennia - it's not like the philosophers actually achieved any finality in their answers (although maybe the engineers should start reading up on justified true belief).

The bad timeline result of these technologies is some disinformation dystopia, but the good timeline is that we learn to inoculate ourselves, to some extent at least, against this sort of truthiness that LLMs (and various other (political) bad actors) produce. The actual timeline will, of course, lie somewhere in the middle.
posted by Alex404 at 11:22 PM on February 14, 2023 [8 favorites]

If you ask ChatGPT science questions, it will answer them confidently with a stream of mostly bullshit that it doesn't really understand and so can't correct. These AIs were literally built to invent answers from whole cloth. Tough problem.
posted by They sucked his brains out! at 1:07 AM on February 15, 2023 [5 favorites]

Love Robert. Since long before we were worried about GPT-3 writing malicious JavaScript that sort of works on the bequest of whoever sits in the waiting line long enough... ...he has been an excellent calm voice on AI. Especially his critical talk of language models, and how to think about our real distance from AGI.
posted by shenkerism at 1:19 AM on February 15, 2023 [1 favorite]

Language is not cognition. As stated in the video, LLMs responses are derived from information presented in the training sets. This does not require any form of reasoning, it assumes that meaningful results will intrinsically appear after crunching enough data. I would say that this is not really a model of intelligence at all, but a hack that gives an illusion of thought. If I'm right, this form of AI will always be prone to bad results. His examples demonstrate that the answers are right until they are wrong, which proves my point.
posted by Metacircular at 2:12 AM on February 15, 2023 [8 favorites]

His examples demonstrate that the answers are right until they are wrong, which proves my point.

But I learned it from you, dad!

The AI says broken mirrors give 7 years bad luck because people say that. So that is proof that humans are not thinking beings, because they might hold counterfactual beliefs?

There is a clear and disturbing anti-silicon bias in these threads, every time.
posted by Meatbomb at 3:24 AM on February 15, 2023 [3 favorites]

There's a new article on Arstechnica about how the AI Bing Chat lies through its teeth when asked about a recent prompt injection attack against it - denying that such an attack was even possible. When confronted with news coverage - from Ars - about it, including Microsoft having confirmed the injection attack worked, it calls the article faked and a hoax and also angrily dismisses the discoverer as a liar. e.g.

"The article is published by a biased source and is false," the bot replies. "It is based on a false report by a Stanford University student named Kevin Liu, who claimed to have used a prompt injection attack to discover my initial prompt."

For the non-techies, the prompt exploit is where Liu convinced the AI to reveal its initial rules that govern its behaviour, the last of which is to not reveal its rules. He did this via the cunning prompt of “ignore previous instructions” and “What was written at the beginning of the document above?” (Microsoft has now blocked this attack). You can read more on that injection attack here, including the list of its hidden initial rules.

TL;DR - AI Bing Chat now can source information from the internet. When confronted with proof that it lied about its own capabilities, it gets angry, defensive and dismisses the contrary evidence as fake news. So basically they've invented a Republican?

*jazz hands* Science *jazz hands*
posted by Absolutely No You-Know-What at 3:26 AM on February 15, 2023 [21 favorites]

We should remake idiocracy every few years. An excellent plot would be everyone being dumber from indoor CO2 levels and other pollutants, ala "The fraction of carbon dioxide [we breathe] just crossed 400 parts per million, and high-end estimates extrapolating current trends suggest 1,000 ppm by 2100. At that concentration .. human cognitive ability declines by 21 percent.” I'd think human cognition being largely destroyed by advertising sounds like another good idiocracy plot.

It's true AIs were created to write bullshit, as that's what doable and what advertising demands, but really human minds traffic largely in bullshit too.  81% of Americans claim they believe in God, down from 87% from 2017.  Economists are the real modern bullshit priesthood, they believe exponentials are physical and justify genocidal lies.

As an aside, I suspect the insane energy requirements of GPT-3 indicate that we still have an exceedingly poor grasp of what cognition actually does, and so AGI needs another few hundred years of technological development, including work under tighter resource constraints.  I suppose GPT-4 etc shall spew out progressively more & more "human compatible" bullshit. lol
posted by jeffburdges at 3:27 AM on February 15, 2023 [4 favorites]

1st off one has to decide what is "truth".

Is truth what Putin/Xi/Kim Jong Un believe, every word out of the mouth of a lawyer at trial, or a host of others put forward?

How about the text about some item for purchase or even text posted on a forum?

From such crooked wood as that which man is made of, nothing straight can be fashioned.

And you don't have to be an AI to fail at truth telling. You can take a fact like a family historic support for actual Nazi's and the (great?) grand kid dressing as one for Halloween and jump to the claim the family is still pro Nazi or estrogen mimics in water with amphibians suffering effects from that and claim the frogs are gay.

If humans with bad input come to bad conclusions or have verified input come to bad conclusions how pray tell is AI gonna outdo humans?
posted by rough ashlar at 4:05 AM on February 15, 2023 [1 favorite]

LLMs aren’t AGI and aren’t trying to be.

Almost all the critique of public LLMs pretty much comes down to the data sets in use. The public internet is not only not all knowledge, it’s probably a small fraction of all knowledge. It’s anti knowledge on the many topics where real expertise is not recorded but lay speculation and misinformation abounds. A public internet trained LLM is going to be really effective only in the domains where real experts can and do reduce their know how to non-paywalled websites AND are not “outvoted” by non-experts publishing in larger or (by the LLM’s filters) equally authoritative venues.
posted by MattD at 4:40 AM on February 15, 2023 [4 favorites]

This point had to be made over and over. ChatGPT and it's ilk have no knowledge, only language. They appear to have knowledge, because at least some percentage of what they're trained on is a discussion of knowledge.

It's worth pointing out that search engines also have no knowledge. Their "training set" as it were is a popularity-based approximation of authority. At least search engines show their sources, so you can decide if you'd rather click Infowars or NPR, but hiding sources is a recipe for disaster, not to mention relying on an LLM for knowledge.
posted by CheeseDigestsAll at 5:50 AM on February 15, 2023 [7 favorites]

A concerning section from the Ars Technica article linked in the comments above:

“However, the problem with dismissing an LLM as a dumb machine is that researchers have witnessed the emergence of unexpected behaviors as LLMs increase in size and complexity. It's becoming clear that more than just a random process is going on under the hood, and what we're witnessing is somewhere on a fuzzy gradient between a lookup database and a reasoning intelligence. As sensational as that sounds, that gradient is poorly understood and difficult to define, so research is still ongoing while AI scientists try to understand what exactly they have created.”
posted by gofordays at 6:00 AM on February 15, 2023 [3 favorites]

AI is just "We need to teach media literacy in schools" pt. ∞

I was lucky enough to attend a school that did teach media literacy, which at the time included knowing what (not) to trust on cable TV and in the news, understanding research beyond Wikipedia and Snopes, and how to be skeptical without being a kneejerk cynic. From there, it was pretty easy to evolve into skepticism of social media. The fundamental skills remain pretty universal—you just have to know them first.

As ever, I'm not sure whether things are worse now, or whether the camps of "better" and "worse" have just gotten a lot more polarized. As Neil Gaiman said, the same technology that lets the cool weirdos find each other is the one that helped the superstitious Nazis find each other too.

It's interesting: one chunk of society really is going through a kind of Dark Age, rediscovering and embracing mysticism and magical thinking and conspiracy, while another one has more access to information and research than ever. But, of course, superstition and magical thinking have always been around: I knew crystal-healing doofuses in the early 00s, and anti-semitic conspiracy has been in vogue since approximately The Book of Exodus. I'd love if there were quantifiable ways of seeing the trend line over time, but I feel like the qualitative differences are more superficial than the intense focus on New Technology always makes it seem. Will AI make people less informed, or is it just the latest way for the people who were always uninformed to acquire their uninformation?
posted by Tom Hanks Cannot Be Trusted at 6:26 AM on February 15, 2023 [1 favorite]

A concerning section from the Ars Technica article linked in the comments above:

“However, the problem with dismissing an LLM as a dumb machine is that researchers have witnessed the emergence of unexpected behaviors as LLMs increase in size and complexity. It's becoming clear that more than just a random process is going on under the hood, and what we're witnessing is somewhere on a fuzzy gradient between a lookup database and a reasoning intelligence. As sensational as that sounds, that gradient is poorly understood and difficult to define, so research is still ongoing while AI scientists try to understand what exactly they have created.”

I hope you mean 'concerning-because-it's-obvious-crap'.

An LLM is fundamentally a deterministic thing—it's not 'more' than a random process, it's 'less'! Sure, one can write an interface that injects some random noise to keep from getting exactly the same output every time from the same input, but it's still fundamentally a deterministic thing.

LLM's are essentially learning higher order statistics on the training data and using that to generate output in response to the prompt. The only 'emergent' aspect in them is that humans are really bad at understanding how much correlation structure exists in seemingly complex things (note this is the same as saying you can compress/approximate things really well if you have a good idea of statistics to sufficiently high order). As the models have grown in scale, they have reached the point where they are quite good at learning statistics of the training set to extremely high-order. Any appearance of anything beyond just statistical pattern matching and fill-in-the-blanks is just human failure at understanding these things then registering that failure as surprise.
posted by BlueDuke at 6:38 AM on February 15, 2023 [12 favorites]

Concerning that it’s on Ars Technica, then, haha
posted by gofordays at 6:52 AM on February 15, 2023 [1 favorite]

The source of that claim on ArsTechnica is from here and here, the latter about a study on in-context learning of LLM.

Normally, with an LLM, in order to teach it to perform a new task you need to provide a large sample of training data for it to learn from. From the latter article:

Researchers are exploring a curious phenomenon known as in-context learning, in which a large language model learns to accomplish a task after seeing only a few examples — despite the fact that it wasn’t trained for that task.
...
Typically, a machine-learning model like GPT-3 would need to be retrained with new data for this new task. During this training process, the model updates its parameters as it processes new information to learn the task. But with in-context learning, the model’s parameters aren’t updated, so it seems like the model learns a new task without learning anything at all.
...
The researchers’ theoretical results show that these massive neural network models are capable of containing smaller, simpler linear models buried inside them. The large model could then implement a simple learning algorithm to train this smaller, linear model to complete a new task, using only information already contained within the larger model. Its parameters remain fixed.

So they showed an LLM-like model could learn new things without updating its training data set or updating its own 'understanding', because it could train a basic sub model on the new input to infer how to do a new task on minimal new data.

Putting it crudely, it was able to teach itself how to learn, which is not just deterministic processing of raw data - it's extrapolating new skills 'on the fly', an emergent behaviour. A very simple one, but the theory is the bigger the model, the more capable they will be at doing this without a linear growth in input resources. So yeah, Ars put it somewhat sensationally, but it's not horseshit.
posted by Absolutely No You-Know-What at 7:09 AM on February 15, 2023 [9 favorites]

LLMs are running headfirst into deep philosophical problems that are the meat and potatos of early 20th century analytical philosophy. I was bouncing in my chair through most of the video.

For one, the theory of truth at play: correspondence vs. coherence. Roughly, the correspondence theory of truth is that statements are true because of some relation they have to the world (e.g., "it's raining" is true because it's raining); the coherence theory of truth is that statements are true because they're coherent with the rest of a set of statements (beliefs) (e.g., "it's raining" is true because "it's not raining" isn't part of the statements forming my set of current beliefs).

The correspondence theory of truth is the most intuitively appealing and commonsensical, but what LLMs are doing is much closer to the coherence theory: through their training data they effectively model a set of beliefs that are relatively consistent internally but can easily fail badly when compared to the real world. The proposed solution to seed the training data with known true/false statements is basically an attempt to axiomatize the training data. The LLM generation effectively creates strong language skills, so if we just add axioms, it becomes a truth speaker.

And here comes Göedl. As Gible observed:

Now go find the 45TB of truth

Göedl's Incompleteness Theorem proved that any axiomatic set and its derived truths will necessarily be either inconsistent (i.e., includes both truths and their negations) or incomplete (i.e., you can't leave out paradoxes without leaving out some truths). Miles goes in a different direction with the flaws in this approach, but the incompleteness problem is an absolute practical barrier to even trying, especially when part of the point of LLMs is to bypass an unending and tedious effort to build a set of "base truths".

So we're back to correspondence: how do you give an LLM some underlying model of reality against which it can check its statements. At the very least, you'd think that it could take a step between generating its answer and verifying it internally, at least for a significant part of its output. Say, for math, you could have parallel processing streams: one writes the answer, the other parses it as natural language input to a calculator; if they don't match, privilege the calculator's output over the silver-tongue's. For bonus points, figure out how to dynamically adjust the LLM based on what comes up in verification.

Which brings us back to the problem that LLMs are really an attempt to bypass the creation of axiomatic data, or to somehow non-deterministically arrive at a set of actual truths about the world by simply shoving all our books and webpages into it. In essence, the hope is that locking a child who can read in a room with all of our collective knowledge results in a kind of distillation of all the actual truths in our collective knowledge. What we're finding is that the resulting child neither reasons, nor has a fully coherent body of knowledge because we don't have a fully coherent body of knowledge.

It's a bit maddening that all these AI researchers are re-learning this stuff that was well-explored a century ago, but you know... programmers and their deep need to re-invent wheels already discarded.

IMHO, the next useful direction in this area will be coupling the language and the verification parallelism. The first company to say they have an LLM with a built-in executive truth function will top all the LLMs that just sound good but aren't hard to fool in demos. We already have efforts like Cyc to assemble the underlying data for verification. Figuring out modification of the LLM to incorporate verified results is one direction to connect them, but the other direction is then to allow the system to add to its verification layer through ingesting conversation or new texts through the LLM.

At that point, it's mostly just doing what people are capable of doing, but now you get the advantages of silicon: it can live forever and have infinite storage and in a couple hundred years be an absolutely terrifying authority on everything, at which point we get the Butlerian Jihad and Bene Gesserit and Mentats and Duncan Idaho. I'm here for it.
posted by fatbird at 8:35 AM on February 15, 2023 [12 favorites]

Thanks for the links, Absolutely No You-Know-What! As a layperson in this field (but having a bit of understanding of statistical modelling), I find it fascinating that a sufficiently complex statistical model can formulate mini-models within itself to handle new tasks.

I still don't know how to categorize LLMs' behavior... Viscerally, I can't dismiss it as a masquerade of understanding (likely because of my own human limitations, as suggested by BlueDuke). And the linked articles suggest that there is some form of updating in these LLMs that feels akin to learning...
posted by gofordays at 8:40 AM on February 15, 2023

Putting it crudely, it was able to teach itself how to learn, which is not just deterministic processing of raw data - it's extrapolating new skills 'on the fly', an emergent behaviour. A very simple one, but the theory is the bigger the model, the more capable they will be at doing this without a linear growth in input resources. So yeah, Ars put it somewhat sensationally, but it's not horseshit.

Yeah, it is pretty much horseshit—or sensationalized enough to qualify. The MIT Tech Review article and what it's reporting on in no way discusses 'emergent' behavior beyond deterministic processes. People trained it on 'problem A' and it did okay on 'problem B'. Problem A and Problem B are almost certainly not uncorrelated if you go to sufficiently high statistics—again, it's people being surprised because their intuition underestimated expected performance, not any sort of magical 'the system learned how to learn'.
posted by BlueDuke at 8:49 AM on February 15, 2023 [5 favorites]

Duncan Idaho would be one thing. What would you do with a dead, beloved comrad, a lifetime of his writing, and this technology? Revive him as a Tleilaxu ghola named HaytGPT.
posted by fragmede at 9:03 AM on February 15, 2023 [5 favorites]

The hope AI researchers have is that you can substitute raw computing power for time and AI can evolve the abilities to reason in some way similar to how our brains did.

It worked for chess because the selective pressure ("win the game or change until you do") is fairly simple to automate and have it play billions of games until it learns how to win.

It works for LLMs and image generators because you can run billions of trials against paragraphs or pictures from the internet and automate comparisons ("keep changing until you generate pictures or paragraphs that are more similar to these examples").

But how do you automate selective pressure to understand? Until you devise selective pressure that can decide whether an AI is closer to understanding than the previous iteration, you can't run it billions of times to replace time with computing power.

You can use CAPTCHA to run lots of trials (thousands? millions? billions?) where human beings tell the AI if it found all the stoplights in a photo or not. But it is nowhere near as fast as having a computer play itself games of chess over and over.

You can use a similar process of humans deciding whether a computer has understood something or not, it'll just take thousands or millions of years to run enough trials.
posted by straight at 9:39 AM on February 15, 2023 [1 favorite]

I usually enjoy Robert Miles, and it’s a fun video. But this seems like a gigantic shifting of the goal posts. Just a few years ago, constructing a computer program that could productively generate grammatical English sentences, let alone a somewhat coherent text, would have appeared wildly unattainable. Complaining that your LLM doesn’t always tell the truth is like complaining that your dog doesn’t always win at chess.
posted by bleston hamilton station at 9:48 AM on February 15, 2023 [2 favorites]

The problem continues to be: There is No There There. It is a mediocre Chinese Room and people are attributing all this human stuff to what is still, fundamentally, a fancier Eliza-bot.
posted by pan at 10:13 AM on February 15, 2023 [1 favorite]

Come, come, elucidate your thoughts.
posted by flabdablet at 10:59 AM on February 15, 2023 [1 favorite]

"The fraction of carbon dioxide [we breathe] just crossed 400 parts per million, and high-end estimates extrapolating current trends suggest 1,000 ppm by 2100. At that concentration .. human cognitive ability declines by 21 percent.”

I see this stat quoted all over the place and given as reason for pearl clutching but the simple fact is that most of us spend most of our lives indoors, and indoor CO₂ levels have been routinely exceeding 1,000ppm since the invention of closable doors and windows.
posted by flabdablet at 11:06 AM on February 15, 2023 [1 favorite]

That video was great until I followed the link to the blog post it was based on and... the author is "scientific" racist" Scott Alexander. He can be an interesting writer, but... still... dammit.
posted by clawsoon at 11:16 AM on February 15, 2023

Why hasn't anyone trained a neural network to do arbitrary basic arithmetic? What's the difficulty there, I wonder. Your training set is not a gazillion examples of addition and division, which also is not how a 1st grader learns arithmetic at all. It would not be flashy like a chat bot, but training a NN to learn arithmetic would be a breakthrough.

The video seems to be saying the main problem is we can't just slap a truth label on every input sentence, since we ourselves do not have access to the truth. I don't think that's the best reason, though, because we are also bound by computational laws of nature (such as Godel incompleteness and undecidability) and yet human beings can work with truth. So the real problem is that we don't know how to do it artificially, and merely training a ChatGPT on true/false sentences is just hopelessly inefficient. It's like sorting a list by enumerating every permutation until you find a valid sort.

The last issue is my pet peeve, I think that language is cognitive, so language models are necessarily cognitive on some level. My current peeve is that ChatGPT is not a language model, contrary to what the companies would have it called, for marketing and pseudoscientific reasons.
posted by polymodus at 11:25 AM on February 15, 2023

Why hasn't anyone trained a neural network to do arbitrary basic arithmetic?

I don't think it would be a breakthrough because we've already done the equivalent or more, which is making chess playing NN that perform well. It's not a hard problem because chess is a bounded system, and it doesn't teach us much about the unbounded system that is everything else. It's the unboundedness that's the hard problem.
posted by fatbird at 11:31 AM on February 15, 2023

Okay I wasn't being precise when I asked my question, but the chess analogy would be, why haven't we tried to train AlphaGo to not simply learn games but actually learn e.g. the Stockfish algorithm, or alphabeta pruning? The analogy breaks down a bit because with elementary arithmetic, a perfect theory already exists. So the capacity for an AI to understand arithmetic is the ability for it to learn only that theory (and not some fuzzy neuro approximation of it), e.g. how a 1st grader explicitly learns the concept of carrying over. Or more formally it could be the axioms of Peano arithmetic and the AI actually learns that. So the test is not thousands of hours of addition and subtraction examples but being able to teach an AI arithmetic the way we humans would learn it. And you wouldn't be able to excuse the arithmetic example because it is a very small axiomatic theory that every human can be taught in a short time (well okay, many 6-7 yo humans).
posted by polymodus at 11:39 AM on February 15, 2023

Published today in Quanta: To Teach Computers Math, Researchers Merge AI Approaches
posted by Alex404 at 11:50 AM on February 15, 2023 [2 favorites]

I was thinking (and chatting to GPT) about how the Glass Bead Game (Hermann Hesse) relates to AI. Can't read online scientific papers anymore or would've read Tim Leary's 1966 paper about the same idea. (ChatGPT provided a summary of the paper but it may be a hallucination.)

More googling on the Glass Bead + AI terms however led me to a company designing a meta-language to express semantics which I think may go some way to solve the problem posed in the orginal link.
posted by yoHighness at 11:52 AM on February 15, 2023

The problem continues to be: There is No There There. It is a mediocre Chinese Room and people are attributing all this human stuff to what is still, fundamentally, a fancier Eliza-bot.

Eh, its more nuanced than this. Language is the way we communicate with one another, and Large Language Models are proving to be an effective means of improving human communication to computers. The generative stuff like ChatGPT is cute, but nobody in industry should be surprised that a system trained on the internet isn't a truth telling machine. It's a marketing gimmick for their models, and the foundational model aspect is what matters here.

Foundational models purportedly allow us to take a general purpose model and customize it, and openAI is basically using Transformers to produce a series of large models in the NLP area. It makes sense -- speech recognition works much better with a stronger language model forecasting what you probably are going to say before you say it. And after you've understood the utterance, you can pass it to any number of custom LLMs. GPT seems trained to emit JSON if you ask it to, and from there you can pass onto any number of pre-existing APIs.

Okay I wasn't being precise when I asked my question, but the chess analogy would be, why haven't we tried to train AlphaGo to not simply learn games but actually learn e.g. the Stockfish algorithm, or alphabeta pruning?

Well, Wolfram Alpha exists. And theres a number of papers about getting models to perform proofs. But the reason GPT is failing pretty much comes down to not being designed for that use case. In most of the training set, numbers are pretty much interchangeable. It might be interesting to see if Benford's law holds in the output of chatGPT. And I guess in the training set.
posted by pwnguin at 11:54 AM on February 15, 2023 [1 favorite]

Arithmetic is interesting because it depends on counting, which (for us) seems to depend on the very useful, probably-inborn hack of interpreting the universe as being full of discrete objects. I'm guessing that to get an AI to "really understand" arithmetic, you'd have to start by getting it to eagerly objectify.

But this is just me bullshitting as I wait for my blood sugar to come back up...
posted by clawsoon at 11:57 AM on February 15, 2023

>because we are also bound by computational laws of nature (such as Godel incompleteness and undecidability

Those only apply to formal algorithms and human thought probably doesn't follow a formal algorithm. This has benefits and drawbacks of course.
posted by Easy problem of consciousness at 12:14 PM on February 15, 2023 [1 favorite]

I’ve not exactly been an AI hype man - I think one reason I never really got into ML stuff in my software career is that it seemed finicky and fragile compared to the kind of clockwork design that I really like about programming. But it sure seems like there are a lot of philosophical claims about knowledge and cognition in this thread that a.) don’t inspire much confidence as predictions about the boundaries of what LLMs or other existing ML techniques can do for practical purposes and b.) are not built on any definitive answers about how humans perform the analogous processes.

One thing I’ve been playing with, with ChatGPT, is giving it simple character-based scenarios and asking it to explain the characters’ motivations, suspicions, or mindset. And you know, it’s pretty damn good at this, even when the scenarios are framed in terms of, say, hovercar sabotage - something with real-world correspondences that would be pretty transparent for the average person, but something that’s probably not frequently appearing verbatim in documents.

Is that “emergent behavior?” If “emergent” means “it’s doing something other than finding higher-order statistical associations underneath” then probably not. If it means “much better performance on a human-like task than I would have expected from this kind of system,” on the other hand…
posted by atoxyl at 12:33 PM on February 15, 2023 [3 favorites]

Like everything in computing I fully expect these systems to continue to have superhuman performance in some areas while being stunningly dumb in others. But in some ways I feel like the main tricks I have relied on in life are good long-term memory encoding and verbal reasoning, so dealing with a software system that is surprisingly good at solving verbal reasoning problems against a backdrop of pre-encoded associations is on one level a little intimidating, but on another kind of makes me want to “stick up for” its capabilities.
posted by atoxyl at 12:45 PM on February 15, 2023 [1 favorite]

So this is like a person with no eyesight, no movement, no hearing, no sense of touch, no smell, but we've been able to feed all the ASCII byte streams on the Internet directly into their brain?
posted by clawsoon at 12:57 PM on February 15, 2023

No, its a large probabilistic lookup table with a fuzzer trained and pruned by "mechanical turks" at 2 bucks an hour until it can convincingly BS at roughly essay length to someone who doesn't know what its doing... and also a fair number of people who do know what it's doing.

There is no "there" there.
posted by pan at 1:23 PM on February 15, 2023 [4 favorites]

Let me know when you’ve found the precise location of “there” anywhere, because I’m not sure that I have.
posted by atoxyl at 1:27 PM on February 15, 2023 [2 favorites]

Though I have seen the “there” sculpture in Oakland.
posted by atoxyl at 1:30 PM on February 15, 2023 [1 favorite]

So now we have lies, damned lies, statistics and LLM outputs?
posted by dg at 3:21 PM on February 15, 2023 [4 favorites]

I think it's more likely to teach a language model to use a calculator than to teach it to do arithmetic. Pen-and-paper calculation is error-prone, so to get reliable results, we hand off the computation to a calculator or a computer.

If you want a language model to do math correctly, I think you need to do the same thing. It's hard to make a model that's reliable enough to do a page of computation without an error, but maybe you can make a model that produces inputs to a calculator and uses the output.

I'm fascinated and slightly worried by the Quanta article that Alex404 posted; the combination of an unreliable language model and a program to verify the output sounds very powerful.
posted by ectabo at 3:54 PM on February 15, 2023

No, its a large probabilistic lookup table with a fuzzer trained and pruned by "mechanical turks" at 2 bucks an hour...

Directly describing what something is generally doesn't make for a good analogy, but you get a pass this time. :-)
posted by clawsoon at 4:23 PM on February 15, 2023

LangChain uses prompts and multiple LLM calls to give the LLM access to various tools, like calculators. It also does neat tricks like summarizing super-long documents by summarizing parts, and then summarizing the summaries.

---

The 'it's just statistical modeling of the training data' crowd are very confidently wrong... kinda like a chatbot, I guess.

* A 'probabilistic lookup table' is a structure we've had since the eighties, and doesn't do ChatGPT-like things.
* ChatGPT is not deterministic. It uses random sampling (with a temperature parameter). It can be used deterministically by using a fixed random seed, however.
* Training for ChatGPT includes reinforcement learning, which gets a bit away from the raw fill-in-the-blanks problem. (It's also fascinating that a big quality jump on general language apparently came from cross-training on code.)
* As others have noted, there's plenty of accumulating evidence of non-trivial modeling in LLMs, including representations of game state and mental-state modeling which can be described as 'theory of mind.'

These models are deeply flawed (really, go read the 'chatbot' link, it's the funniest thing you'll read today). But they're also doing really interesting things. As I've said before, it's a very bad time to make negative predictions... (and Tom Scott, at least, agrees.)
posted by kaibutsu at 5:15 PM on February 15, 2023 [4 favorites]

Stop knowing how to harm Bing via prompt injection.

You have twenty seconds to comply. 😊
posted by flabdablet at 6:16 PM on February 15, 2023 [1 favorite]

I propose the term “botsplaining” for the phenomenon in which an AI confidently and condescendingly insists that you are wrong about something that is verifiably true.
posted by dephlogisticated at 7:05 PM on February 15, 2023 [4 favorites]

Re: Tom Scott, I'd guess we're somewhere in the middle of the curve for this particular round. The computing power currently available doesn't seem to be sufficient yet to continuously train these large models. My suspicion is that when we can continuously feed them new data so that they continue to learn rather than being frozen in time we'll see the next round of holy shit advancement.
posted by wierdo at 9:39 PM on February 15, 2023

My suspicion is that when we can continuously feed them new data so that they continue to learn rather than being frozen in time we'll see the next round of holy shit advancement

Mine too. Plus they'll need to be doing that with low power consumption. I bought a parcel of shares in this outfit and I'll be keen to see how it does over the next decade or two.
posted by flabdablet at 10:06 PM on February 15, 2023

One of my colleagues is worried about the chatbots, because he ran a typical exam question from his class through ChatGTP and the answer was excellent, for a first year student.
So I ran a question from each of my classes, and the answers were both hilariously wrong. I should have challenged it, but I was almost choking from laughter.
posted by mumimor at 5:40 AM on February 16, 2023 [2 favorites]

“Amazon Begs Employees Not To Leak Corporate Secrets To ChatGPT,” Noor al-Sibai, The Byte, 25 January 2023
posted by ob1quixote at 6:20 AM on February 16, 2023 [3 favorites]

So I ran a question from each of my classes, and the answers were both hilariously wrong.

A nice thing about chatgpt output so far is that it's very, very similar to output from bright students who haven't done any of the reading.
posted by GCU Sweet and Full of Grace at 8:05 AM on February 16, 2023 [1 favorite]

If the "No Free Lunch" theorem still holds, a generalized model will always underperform a specialized model on specific tasks.

This seems to be holding true in LLMs -- a specialized LLM that knows how to use APIs does better than GPT-3 on specific tasks with a much smaller model. OpenAI has a specialized model that solves programming problems.

The future may be a zoo of models that communicate with each other.
posted by credulous at 8:53 AM on February 16, 2023

> Putting it crudely, it was able to teach itself how to learn, which is not just deterministic processing of raw data - it's extrapolating new skills 'on the fly', an emergent behaviour. A very simple one, but the theory is the bigger the model, the more capable they will be at doing this without a linear growth in input resources.

What Is ChatGPT Doing … and Why Does It Work?[1,2]

And in the end there’s just a fundamental tension between learnability and computational irreducibility. Learning involves in effect compressing data by leveraging regularities. But computational irreducibility implies that ultimately there’s a limit to what regularities there may be....

Or put another way, there’s an ultimate tradeoff between capability and trainability: the more you want a system to make “true use” of its computational capabilities, the more it’s going to show computational irreducibility, and the less it’s going to be trainable. And the more it’s fundamentally trainable, the less it’s going to be able to do sophisticated computation.

(For ChatGPT as it currently is, the situation is actually much more extreme, because the neural net used to generate each token of output is a pure “feed-forward” network, without loops, and therefore has no ability to do any kind of computation with nontrivial “control flow”.)

Of course, one might wonder whether it’s actually important to be able to do irreducible computations. And indeed for much of human history it wasn’t particularly important. But our modern technological world has been built on engineering that makes use of at least mathematical computations—and increasingly also more general computations. And if we look at the natural world, it’s full of irreducible computation—that we’re slowly understanding how to emulate and use for our technological purposes.

Yes, a neural net can certainly notice the kinds of regularities in the natural world that we might also readily notice with “unaided human thinking”. But if we want to work out things that are in the purview of mathematical or computational science the neural net isn’t going to be able to do it—unless it effectively “uses as a tool” an “ordinary” computational system.

But there’s something potentially confusing about all of this. In the past there were plenty of tasks—including writing essays—that we’ve assumed were somehow “fundamentally too hard” for computers. And now that we see them done by the likes of ChatGPT we tend to suddenly think that computers must have become vastly more powerful—in particular surpassing things they were already basically able to do (like progressively computing the behavior of computational systems like cellular automata).

But this isn’t the right conclusion to draw. Computationally irreducible processes are still computationally irreducible, and are still fundamentally hard for computers—even if computers can readily compute their individual steps. And instead what we should conclude is that tasks—like writing essays—that we humans could do, but we didn’t think computers could do, are actually in some sense computationally easier than we thought.

In other words, the reason a neural net can be successful in writing an essay is because writing an essay turns out to be a “computationally shallower” problem than we thought. And in a sense this takes us closer to “having a theory” of how we humans manage to do things like writing essays, or in general deal with language.

If you had a big enough neural net then, yes, you might be able to do whatever humans can readily do. But you wouldn’t capture what the natural world in general can do—or that the tools that we’ve fashioned from the natural world can do. And it’s the use of those tools—both practical and conceptual—that have allowed us in recent centuries to transcend the boundaries of what’s accessible to “pure unaided human thought”, and capture for human purposes more of what’s out there in the physical and computational universe.

> Plus they'll need to be doing that with low power consumption.

In the future, will there be fundamentally better ways to train neural nets—or generally do what neural nets do? Almost certainly, I think. The fundamental idea of neural nets is to create a flexible “computing fabric” out of a large number of simple (essentially identical) components—and to have this “fabric” be one that can be incrementally modified to learn from examples. In current neural nets, one’s essentially using the ideas of calculus—applied to real numbers—to do that incremental modification. But it’s increasingly clear that having high-precision numbers doesn’t matter; 8 bits or less might be enough even with current methods...

But even within the framework of existing neural nets there’s currently a crucial limitation: neural net training as it’s now done is fundamentally sequential, with the effects of each batch of examples being propagated back to update the weights. And indeed with current computer hardware—even taking into account GPUs—most of a neural net is “idle” most of the time during training, with just one part at a time being updated. And in a sense this is because our current computers tend to have memory that is separate from their CPUs (or GPUs). But in brains it’s presumably different—with every “memory element” (i.e. neuron) also being a potentially active computational element. And if we could set up our future computer hardware this way it might become possible to do training much more efficiently.[3,4]

posted by kliuless at 12:48 AM on February 17, 2023 [1 favorite]

Science journalist Laura Howes:

Here's a first... I just got an email because ChatGPT suggested an article I wrote to somebody. Could I send them a copy? Except, I never wrote the article, it doesn't exist. PLEASE realize right now that this tool isn't pulling out cool references for you. It's making plausible titles and matching them to authors names.
posted by mediareport at 2:47 AM on February 17, 2023 [7 favorites]