Finally a killer AI

August 30, 2023 7:43 PM

“There are hundreds of poisonous fungi in North America and several that are deadly, They can look similar to popular edible species. A poor description in a book can mislead someone to eat a poisonous mushroom.” - AI generated mushroom foraging books are spreading on Amazon, placing the public at risk.
posted by Artw (51 comments total)

Luckily there's All That the Rain Promises, and More..., a book whose vibe is so delightfully weird it could only have been made by honest to god humans. Accept no substitutes, my friends
posted by potrzebie at 7:59 PM on August 30, 2023 [29 favorites]

This isn't even funny. Someone's going to die because someone else decided to crank out AI trash and flood the world with bullshit to make a few bucks.
posted by star gentle uterus at 7:59 PM on August 30, 2023 [40 favorites]

Well. Thanks, AI bros. Thanks!
posted by egypturnash at 8:00 PM on August 30, 2023 [3 favorites]

Couldn't we instead have some mushroom-generated books spreading using Amazon's AI? That might be a more interesting thing.
posted by hippybear at 8:02 PM on August 30, 2023 [14 favorites]

It's cute how the "AI-generated text detection tool" describes 'Tomson' and 'Stella's' experiences in the past tense! Like blockchain I've yet to see a real use for this. 404 is great journalism, thanks for posting Artw
posted by unearthed at 8:05 PM on August 30, 2023 [3 favorites]

Hmm, I've wondered if machine learning based ID apps like Seek would even try to identify mushrooms given the potential consequences. This is much worse that what I've been concerned about. Which I guess shouldn't surprise me.
posted by mollweide at 8:06 PM on August 30, 2023 [2 favorites]

It's hard not to be fearful of the future when pillars of formerly reliable information are getting flooded with misinformation. I wish I could believe that AI will only benefit humanity, but I feel like I'm developing a sense of doom that I didn't think I'd have until I became a grumpy old man.
posted by donuy at 8:06 PM on August 30, 2023 [9 favorites]

Remember when Neti pots first got big and experts initially couldn't figure out why people were dying from brain infections? Well, here in Australia there was a recent mushroom-poisoning case that killed multiple people. Gotta wonder.
posted by BiggerJ at 8:10 PM on August 30, 2023 [4 favorites]

Tip for the room. You can grow your own mushrooms from spores or liquid cultures. The Uncle Ben's subreddit has all the info you need.
posted by leotrotsky at 8:28 PM on August 30, 2023 [6 favorites]

It's hard not to be fearful of the future present when pillars of formerly reliable information are getting flooded with misinformation.
posted by fairmettle at 8:33 PM on August 30, 2023 [7 favorites]

North Spore literally has all you need, if you're willing to order substrate and other things off the internet.
posted by hippybear at 8:36 PM on August 30, 2023 [2 favorites]

I've gotten into mushroom hunting as a hobby since the pandemic made being alone outdoors a choice activity. It's aggravating that death by lying mushroom robots can be a thing now.

But there is a special joy in mushroom foraging you don't get from growing from logs. Of course you have to do it responsibly. There are some mushroom varieties that will grow only in the wild. There's so much to learn about and enjoy: some mushrooms are associated with some kinds of trees more than others, I'm learning which mushrooms are in season when, and I have favorite spots I go back to. I started with medicinal polypores like turkey tail and reishi, which are hard to misidentify. Chicken of the woods, cauliflower mushroom, maitake, oyster (which you can grow on logs), lion's mane, bear's tooth, coral tooth, chaga, voluminous milk caps, boletes. Lobster mushroom, which is one fungus parasitizing another and taking over its DNA. Last week my first "hedgehog mushrooms". Today I found my first "ringless honey mushrooms", which I may have passed over before because I've been adding in gilled mushrooms slowly as I learn more.
posted by Schmucko at 8:47 PM on August 30, 2023 [26 favorites]

I wish I could believe that AI will only benefit humanity

As long as it's controlled by people who have most of the money in the world, there's no way it can be a net good. It will do their bidding, which is only good for them.
posted by Ickster at 9:11 PM on August 30, 2023 [17 favorites]

Though somewhat breathless (and in need of a couple of copy edits), and fundamentally speculative (no actual examples of misidentified species cited from any of these junk books), I guess it’s useful to call out these more colourful sides to the AI sludge (and Amazon barrage) that we are going to find ourselves inundated with. From the perspective of a couple of years of in-depth mycological learning, it’s true that mistaken identification can cost lives (a common emerging scenario currently is destitute immigrants mistaking poisonous species for similar edible ones they knew and consumed safely back home), but it’s a bit of a recurring reflex for media to seasonally sound the mushroom foraging alarm, so AI is mostly just the new flavour, here…
posted by progosk at 9:20 PM on August 30, 2023 [4 favorites]

Remember when Neti pots first got big and experts initially couldn't figure out why people were dying from brain infections?

Huh? Deaths from brain infections caused by neti pots are extremely rare.
posted by ultraviolet catastrophe at 9:23 PM on August 30, 2023 [6 favorites]

This isn't even funny. Someone's going to die because someone else decided to crank out AI trash and flood the world with bullshit to make a few bucks.

We kept hearing about cyber 9/11 from people who are now doing their damndest to profit off of it after it's occurred.
posted by StarkRoads at 9:25 PM on August 30, 2023 [1 favorite]

what does that even mean?
posted by ryanrs at 9:49 PM on August 30, 2023 [1 favorite]

Isn't this how The Last of Us got started?
posted by dances_with_sneetches at 9:57 PM on August 30, 2023 [1 favorite]

How The Last of Us got started.
posted by Artw at 10:16 PM on August 30, 2023 [2 favorites]

I’m a forager and certified mushroom reseller so needless to say this has been generating a lot of talk in my social media circles. Books don’t worry me as much as apps do. People really rely on apps to identify stuff these days and while they can be good for plants sometimes, they are generally crap for mushrooms. We’ll see how this all plays out. Hopefully it’ll just end of being a lot of people eating stinkhorns.
posted by misterpatrick at 11:00 PM on August 30, 2023 [20 favorites]

This is fairly consistent with how businesses like Jeff Jorgensen's sell products that have been banned or recalled or are fraudulent or hazardous. That technology has empowered entreprenuers to adulterate and poison knowledge in addition to just adulterating and poisoning milk, honey, rice, pet food, bread, alcohol, drinking water etc... well, they don't call it late-stage for nothing. Killing the world for a quick buck is not a bug, it is the purpose of these technologies. What is the business case for a non-harmful to humans use for AI? What can't we do now with humans that requires an anti-human machine to be built? The whole purpose of AI is to replace humans. Wait until it designs viruses. Creative destruction and 'being disruptive' was always about them destroying us in new and exciting ways, to their profit.
posted by AnchoriteOfPalgrave at 12:15 AM on August 31, 2023 [5 favorites]

Hopefully it’ll just end of being a lot of people eating stinkhorns.

Depending on where in the world you forage, people have been enjoying various stinkhorns for generations, turns out, whether at the egg stage in several Slavic and Indigenous cultures “down under”, or skirted (but capped) in various Asian regions. The actually hazardous species of fungi really are few and far between (but do require study), there are many mushroom-worlds we know all too little about…
posted by progosk at 12:40 AM on August 31, 2023 [1 favorite]

There was a recent incident in Australia where poisonous mushrooms appear to have killed three people; and landed a fourth person in Intensive-Care.

It is not clear at this stage if the poisoning was accidental (as the woman who prepared the meal claims it was) or non-accidental.
posted by chariot pulled by cassowaries at 1:19 AM on August 31, 2023 [5 favorites]

The books may be generated by AI, but there are human beings behind them and they should be held responsible. If they can’t be found, Amazon should be held responsible (or perhaps anyway) but no doubt governments are too frightened of very rich people to contemplate anything ‘unrealistic’ like that.
posted by Phanx at 1:41 AM on August 31, 2023 [10 favorites]

Every type of mushroom is edible.

Some of them only once, though.
posted by DreamerFi at 1:51 AM on August 31, 2023 [21 favorites]

Amazon cannot be trusted. Isn't this the obvious conclusion?
posted by jeffburdges at 1:58 AM on August 31, 2023 [8 favorites]

FungAI
posted by chavenet at 2:12 AM on August 31, 2023 [17 favorites]

Several years ago I went mushroom picking in NE Poland. It can get pretty busy in the woods especially near to access roads during fine weekends in the picking season. I was getting pretty smug about being able to pick the right sort until I brought one to the communal basket and was told to a) throw it far away and b) go down to the lake niezwłocznie to wash my hands. In the car back to town, my hosts were lamenting the passing of an 80-something professor who was the acknowledged local expert on all things fungal. Turned out that he'd made a tragic error in the field and died from it.
posted by BobTheScientist at 2:21 AM on August 31, 2023 [17 favorites]

It says something that people would choose these books over other, obviously 'legit' guides.
Caveat Emptor you dolts! look it up! or are people downloading these books for free? At which point wouldn't the producers be responsible for the content and liable for any misinformation they contained?

Maybe this is the plan ... the "Swoop!" of AI gaining general purpose, super-intelligence, has already happened and this is how it's thinning the herd.

Yeah, that sounds legit... maybe I should get it to write that as a novel.
posted by From Bklyn at 2:35 AM on August 31, 2023 [3 favorites]

It says something that people would choose these books over other, obviously 'legit' guides.

Actually while they're being spammed all over Amazon, there's no evidence that anyone is choosing them or buying them.

Caveat Emptor you dolts! look it up! or are people downloading these books for free?

Not usually on Amazon, but AI-generated dreck web pages that can be read for free are probably a far larger problem than the one this article is talking about.
posted by mmoncur at 2:48 AM on August 31, 2023 [6 favorites]

So if there are tools that can identify bot text, why isn't Amazon culling bot-written books?

Why isn't Google deranking bot-written pages?
posted by Baethan at 3:56 AM on August 31, 2023 [11 favorites]

There was a recent incident in Australia where poisonous mushrooms appear to have killed three people; and landed a fourth person in Intensive-Care.

The Leongatha incident, and whether any actual mushrooms played a role in it, is far from clear, so far...

I brought one to the communal basket and was told to a) throw it far away and b) go down to the lake niezwłocznie to wash my hands.

Do you remember what sort of mushroom it was? Poland is often cited as a country of extreme mycophiles, but it was also where the particularly pernicious Cortinarius orellanus was first identified in the '50s, after a couple of generations of foragers kept losing members, having apparently forgotten the distinguishing details to keep from confusing it with C. caperatus...

In the car back to town, my hosts were lamenting the passing of an 80-something professor who was the acknowledged local expert on all things fungal. Turned out that he'd made a tragic error in the field and died from it.

This might be an apocryphal version of the actual, still insufficiently studied case of Bavarian mycologist J. Schaeffer, who after years of having eaten Paxillus involutus, as was habitual (and still conditionally is in some Central European regions), but having gone off mushrooms altogether for a few years, on being tempted to reenjoy them one October in 1944, fell violently ill and, due to difficulties in wartime health assistance, succumbed to them (details provided by his wife here, in German).

In general, it's wise to study well, from verified/corroborated human-expert sources, and not to gorge oneself on any single species....
posted by progosk at 4:14 AM on August 31, 2023 [2 favorites]

Top Russian rocket scientist dies from ‘mushroom poisoning’ just weeks after Putin’s failed moon landing [CW: The Sun]
posted by chavenet at 5:30 AM on August 31, 2023

I've wondered if machine learning based ID apps like Seek would even try to identify mushrooms given the potential consequences.

Seek does have a click-through warning that comes up each time you open it, but it didn't hesitate to give me an identification of a Western Sulphur Shelf when I pointed it at a mushroom.

I personally find mushrooms to be one of those things whose risk is easily mitigated by simply choosing to not eat them ever.
posted by RonButNotStupid at 5:37 AM on August 31, 2023 [10 favorites]

Top Russian rocket scientist dies from ‘mushroom poisoning’ just weeks after Putin’s failed moon landing [CW: The Sun]

Here's more (googletranslated) detail from the (Kremlin-aligned) MK...
posted by progosk at 5:41 AM on August 31, 2023 [1 favorite]

I personally find mushrooms to be one of those things whose risk is easily mitigated by simply choosing to not eat them ever

There are dozens, if not hundreds, of known poisonous vegetables, fruits and plants. Would you also never eat any of those?
posted by progosk at 5:43 AM on August 31, 2023 [1 favorite]

There are dozens, if not hundreds, of known poisonous vegetables, fruits and plants. Would you also never eat any of those?

Yeah. If given the choice between not eating any plants or accepting the slight risk that any given plant I eat could be poisonous....I'll accept the slight risk. It's just easier for me to completely zero out the risk of mushrooms by not eating them at all. Also, I don't particularly like mushrooms.

Other things I choose to zero-out my risk factor by voluntary abstinence:

Fugu
Skydiving
Bungee jumping
Diving to extreme depths in a submersible with an unrated carbon-fiber hull
posted by RonButNotStupid at 5:53 AM on August 31, 2023 [6 favorites]

So if there are tools that can identify bot text, why isn't Amazon culling bot-written books?

It's very hard to automatically detect AI-written text. Language models are not usually trained using an adversarial framework (that is, trained specifically to avoid detection tools)- yet- but if detection became a threat to LLM viability I'm sure adversarial training would become more common.

ChatGPT creator pulls AI detection tool due to ‘low rate of accuracy’.
posted by a faded photo of their beloved at 6:13 AM on August 31, 2023

> …flood the world with bullshit to make a few bucks.

The foundation of our modern economy…

And our economy is unfortunately structured to where too many people can’t live comfortably doing society-positive sorts of work (and/or aren’t even given the resources to learn society-positive skills in the first place), so they’re largely obligated to come up with some sort of scam or hustle if they want to achieve any sort of financial security. So blame the scammer. But also blame the system.
posted by chasing at 6:48 AM on August 31, 2023 [3 favorites]

Also to add a quick side note - touching a mushroom is not dangerous. In fact a “spit test” is a common mention for getting an exact ID for many species. This is taking a slice and letting it sit on your tongue for a little bit. The only way for a toxic mushroom to harm you is ingesting it. Just saw a little clip of a forager vigorously chewing on a Destroying Angel and then spitting it out to prove this point.

There is a lot of incorrect information and just plain woo about mushrooms out there and always good to try and correct them.
posted by misterpatrick at 8:21 AM on August 31, 2023 [3 favorites]

I'll say it again: I'm not afraid of AI per se, but I do not believe that humans, as a whole, can be trusted with it.
posted by Faint of Butt at 8:28 AM on August 31, 2023 [5 favorites]

I wish I could believe that AI will only benefit humanity

As long as it's controlled by people who have most of the money in the world, there's no way it can be a net good. It will do their bidding, which is only good for them.


I swear we are heading right down the path to the Butlerian Jihad. And it ain't the Golden Path either!
posted by KaizenSoze at 9:41 AM on August 31, 2023 [1 favorite]

I wonder if this is the sort of thing that the people working on "AI risks" are looking at? My general sense is that for most of the big-time, splashy names that show up in the general media *cough* Altman *cough* Bostrom *cough*, it ranges from "not that much" to "not even a little bit". They all seem to be preoccupied with obsessing over sci-fi fantasy scenarios more than examining actually existing AI and actually existing risks.
posted by mhum at 9:47 AM on August 31, 2023 [3 favorites]

Exactly that. There’s a media focus on total bullshit scenarios where the chat bots become a real intelligence whilst the actual harms they perpetrate are ignored.
posted by Artw at 9:54 AM on August 31, 2023 [5 favorites]

I'll say it again: I'm not afraid of AI per se, but I do not believe that humans, as a whole, can be trusted with it.

Ted Chiang’s observation that most of what people say about AI is actually just capitalism becomes very applicable here.
posted by Artw at 9:56 AM on August 31, 2023 [8 favorites]

First they came for the Primary Sector and I said nothing - because more production was good for me.

Then they came for the Secondary Sector and I said nothing - because more manufacturing output was good for me.

Then they came for the Tertiary Sector and I said nothing - because cheaper services were good for me.

Then they came for me . . .

(With apologies to Niemöller)
posted by Heywood Mogroot III at 10:27 AM on August 31, 2023 [2 favorites]

I saw a post on Facebook about AI generated foraging books and did some of my own digging on Amazon. “The Forager’s Harvest Bible” is one title that algorithm gamers seem to have landed on. They switch up the author every couple of months with a new AI generated bio and headshot. One of the many 2023 editions of the Forager’s Harvest Bible still had the phrase “(Insert Author Name Here)” in the book description. Oops.

They also will use ghost accounts to buy a few copies of their own book and leave 5-star reviews that also read as AI generated.

Honest reviewers have posted photos of some pages and there indeed is no page formatting or pictures (pictures are kind of important in a foraging book, no?) It’s wall of text stuff that’s trying to sound authoritative in that way that hallucinating ChatGPT sounds authoritative

I reposted the original post to a Facebook group and a dude showed up who was defending the books left and right and swearing up and down that he’s a happy reader of several of the “Forager’s Harvest Bible” books. When I asked him why he would own several books from 2023 with the exact same name but different unknown-to-Google authors, he didn’t have a response 🤷‍♂️
posted by Skwirl at 2:26 PM on August 31, 2023 [3 favorites]

Amazon is full of weird bullshit/AI-generated/Wikipedia-derived/spambot books on all sorts of nonfiction topics (not just foraging but health, recipes, biographies). It fucking sucks.
posted by hungrytiger at 4:33 PM on August 31, 2023 [5 favorites]

TheBlackForager just posted about this (it's also, as Skwirl said, happening with foraging books) and it's really, really concerning. I vividly remember a story from the responsible folks over at Chestnut School of Herbal Medicine about someone who misidentified young foxglove leaves and put them in her smoothie (foxglove foliage being much less distinctive than the flowers, and easy to confuse with several edible plants). Reader, she died.

The best-case scenario is that people will get these books and realize they're bullshit, and not actually forage with them, but that still sucks! People are still getting scammed. And the worst case scenario is that someone eats something that makes them sick or kills them, and there's nobody to sue.

This is 100% happening in all kinds of genres. Imagine the implications of shitty AI electrical instructions. Or car repair. Or first aid.
posted by Nibbly Fang at 7:02 PM on August 31, 2023 [2 favorites]

This is 100% happening in all kinds of genres. Imagine the implications of shitty AI electrical instructions. Or car repair. Or first aid.

Or AI prompts.
posted by From Bklyn at 2:02 AM on September 1, 2023

In 2014, Ursula Le Guin warned that corporate power wanted to stop us from even imagining freedom.

Nearly a decade later, the tech industry seems intent on using generative AI to do just that. It’s our responsibility to stop them.
posted by jeffburdges at 4:43 AM on September 2, 2023 [2 favorites]