Baloney Detection Kit

June 25, 2009 11:49 AM

The Baloney Detection Kit. "With a sea of information coming at us from all directions, how do we sift out the misinformation and bogus claims, and get to the truth? Michael Shermer, Publisher of Skeptic magazine, lays out a 'Baloney Detection Kit' — ten questions we should ask when encountering a claim."
posted by homunculus (52 comments total)

Previously.
posted by homunculus at 11:50 AM on June 25, 2009 [2 favorites]

1. Does it have a first name?
posted by Sys Rq at 11:55 AM on June 25, 2009 [18 favorites]

As homunculus' comment points out above, the phrase was coined by Carl Sagan. Good stuff. I can still remember the mind-expanding feeling I had when I first read that list at the age of sixteen or so.

Wikipedia has two excellent enlightening (and, humbling) lists that are relevant:
List of cognitive biases
List of fallacies
posted by Idle Curiosity at 11:59 AM on June 25, 2009 [6 favorites]

This really just seems like common scientific reasoning, though putting it into top 10 list form likely makes it more palatable to the late-nite Letterman crowd.

...not that scientists and Letterman watchers are orthogonal sets.
posted by scrutiny at 11:59 AM on June 25, 2009

Where's the Baloney Detection Kit that alerts me when I need to go to the deli?
posted by espire at 12:00 PM on June 25, 2009 [1 favorite]

11. Is it on Snopes.com?
posted by msalt at 12:04 PM on June 25, 2009

Another useful guide is:

John Baez's crackpot scoring system
posted by Erberus at 12:11 PM on June 25, 2009 [4 favorites]

Bologna? Baloney? Potato? Potahto?
posted by Oriole Adams at 12:16 PM on June 25, 2009

It's bologna, just like the song.
posted by Mister_A at 12:17 PM on June 25, 2009

Shermer is awesome. I recommend his excellent book, Why People Believe Weird Things. As its title would suggest, it helps to explain why otherwise intelligent people get caught up in ridiculous (and often offensive) fringe movements. In particular, reading it helped me deal with an ex-roommate who was a little too into conspiracy theories.
posted by Afroblanco at 12:19 PM on June 25, 2009

Great list of skeptic terms.
posted by HumanComplex at 12:22 PM on June 25, 2009 [3 favorites]

Testing one, two, three... check... check...

Is this thing on?
posted by Balonious Assault at 12:27 PM on June 25, 2009

homunculus:

I wrote a wide-eyed review of Carl Sagan's book, The Demon Haunted World, (leaving off the 'Science as a Candle in the Darkness' part of the title) and emailed it to Art Bell. I was vague but enthusiastic about the book's contents, and its title must have led Art to believe that the book was about how unseen demons haunt the world.

The next day he pitched the bookt to his listeners.

:D
posted by jfrancis at 12:33 PM on June 25, 2009 [12 favorites]

12. Is the claimant playing by the rules of science?

I read this as "Is the clarinet playing by the rules of science?" and thought, "Well, if it isn't, we should probably figure out why, because that is fucking crazy."

I do like that #2 is basically asking, 'has the source demonstrated that they are full of shit in the past?'
posted by quin at 12:35 PM on June 25, 2009 [1 favorite]

okay ... so let's make sure it gets added to the high school curriculum. Nobody gets to graduate unless they can recite all ten points backwards.

Full recitation of the entire Baez system gets you straight into grad school.
posted by philip-random at 12:42 PM on June 25, 2009

A good list, but there's a basic assumption that the person asking the questions is asking from a rational standpoint. Baloney will easily pass through the filter if baloney is the basis for that filter.

For example, I believe in String Theory and the Unified Field Theory. Neither of these are, as yet, provable, but from the basis of that belief I'm fairly credulous and could take any number of outlandish claims as truth using this list.

That said, it's always good to have people trying to improve the over-all level of rationality.
posted by lekvar at 12:42 PM on June 25, 2009

I love The Demon-Haunted World. And Sagan followed his own rules and admitted that there were two? three? little niches in parapsychology he thought could bear more investigation rather than dismiss them outright. Whether or not you buy into that, it's a good display of integrity for a scientist and someone who is promoting a heuristic for rating the junkiness of a claim.
posted by adipocere at 12:50 PM on June 25, 2009

from teh John Baez link:
40 points for comparing those who argue against your ideas to Nazis, stormtroopers, or brownshirts.

is global warming 'denial' close enough?
posted by Confess, Fletch at 12:55 PM on June 25, 2009

We have a baloney detection kit--his name is Rocco. My messenger bag is carefully scanned each afternoon at the door.
posted by everichon at 12:58 PM on June 25, 2009 [6 favorites]

I call bullshit on baloney.
posted by Superfrankenstein at 12:59 PM on June 25, 2009

I call bullshit on baloney.

Special order for table 4.
posted by fleetmouse at 1:04 PM on June 25, 2009 [6 favorites]

Please tell me that I'm not the only one who read the link as "Joan Baez's crackpot scoring system" and wondered what in the world any of this had to do with folk music.
posted by leftcoastbob at 1:10 PM on June 25, 2009 [6 favorites]

You're not the only one.
posted by Sys Rq at 1:23 PM on June 25, 2009

The Ten Commandments Dialogues

IATIA: You shall have no other gods before me.
IRFH: How reliable is the source of the claim?
IATIA: I am that I am.
IRFH: Well, I guess Popeye wouldn't lie to me. Does the source Sailor Man make similar claims?
IATIA: You shall not make for yourself any carved image, or any likeness of anything that is in heaven above, or that is in the earth beneath, or that is in the water under the earth; you shall not bow down to them nor serve them. For I, the Lord your God, am a jealous God, visiting the iniquity of the fathers on the children to the third and fourth generations of those who hate me, but showing mercy to thousands, to those who love Me and keep My commandments. A-gah-gah-gah-gah-gah-gah!
IRFH: I'd say that's a big, "Yes." Have the claims been verified by somebody else?
IATIA: Moses was there. He saw me.
IRFH: He saw a wildfire. He heard voices on the mountain. Not what I would call a reliable witness. Does this fit with the way the world works?
IATIA: You shall not take the name of the Lord your God in vain, for the Lord will not hold him guiltless who takes His name in vain.
IRFH: Four out of ten commandments are about honoring you. I'd say you have "vain" pretty well covered, sir. So - Has anyone tried to disprove the claim?
IATIA: Remember the Sabbath day, to keep it holy. Six days you shall labor and do all your work, but the seventh day is the Sabbath of the Lord your God. In it you shall do no work: you, nor your son, nor your daughter, nor your manservant, nor your maidservant, nor your cattle, nor your stranger who is within your gates. For in six days the Lord made the heavens and the earth, the sea, and all that is in them, and rested the seventh day. Therefore the Lord blessed the Sabbath day and hallowed it.
IRFH: Six days? Really? Is that your final answer? You want to call a lifeline?
IATIA: Honor your father and your mother, that your days may be long upon the land which the Lord your God is giving you.
IRFH: My father and mother? Lord bless them, my father and mother couldn't explain their way out of a paper sack with a Funk and Wagnall's and an Elements of Style. It's all on you, now. Where does the preponderance of evidence point?
IATIA: You shall not murder.
IRFH: You're killing me, here. You call this playing by the rules of science?
IATIA: You shall not commit adultery.
IRFH: See, now you're not even trying. How about providing some positive evidence?
IATIA: You shall not steal.
IRFH: Sagan? Is that you?
IATIA: You shall not bear false witness against your neighbor.
IRFH: I never claimed these were original questions. Does Sagan as God account for as many phenomena as the old theory?
IATIA: You shall not covet your neighbor’s house; you shall not covet your neighbor’s wife, nor his manservant, nor his maidservant, nor his ox, nor his donkey, nor anything that is your neighbor’s.
IRFH: How about your Pulitzer? Can I covet your Pulitzer?
IATIA: You shall have no other gods before me.
IRFH: I think we've been over this one already. Are personal beliefs driving the claim?
IATIA: I am that I am. Billions and billions of I am.
posted by It's Raining Florence Henderson at 1:25 PM on June 25, 2009 [3 favorites]

I'll do you one better, leftcoastbob. I read it as "Joan Baez's crockpot scoring system."
posted by MrMoonPie at 1:26 PM on June 25, 2009 [4 favorites]

A good list, but there's a basic assumption that the person asking the questions is asking from a rational standpoint. Baloney will easily pass through the filter if baloney is the basis for that filter.

Good point. One Shermer himself might do well to remember. For example, there was a piece Shermer wrote for SciAm a while back, in which he opined:

Folk economics caused us to disdain excessive wealth, label usury a sin and mistrust the invisible hand of the market.

Ironically, it's the concept of "the invisible hand of the market" that owes its origins most to folk science (in early free market theory, the "invisible hand" of the market was explicitly identified with "the divine hand of providence"--in other words, it was an article of religious faith that became an axiom of free market theory, rather than one that had validated as a rigorously tested scientific hypothesis).

And since the early days of free market theory, rational choice theory has persisted as the dominant model in economic theory, despite mounting empirical evidence that human economic behavior by and large isn't nearly as rational as theory supposes.

Here, nevertheless, Shermer goes on working from the assumption that free market economic theory has a rational, scientific basis, seemingly oblivious to the fact that science actually isn't on his side.

So either he's full of baloney, too, or he left his baloney detection kit home that day.
posted by saulgoodman at 1:32 PM on June 25, 2009 [4 favorites]

For the record, Metafilter was ahead of the curve on evaluative lists for exceptional claims:

From 2007; Pastabagel's ground rules for claiming proof of extraordinary phenomena.
posted by quin at 1:33 PM on June 25, 2009 [1 favorite]

"one that had been validated" ... or something...
posted by saulgoodman at 1:33 PM on June 25, 2009

I like how Pastabagel's peer-review process seems to center around violence. I think that a lot of bogus science would fall by the wayside if that was more commonly implemented.

Sadly, his views on otters leave him somewhat lacking as a credible source on, well, anything.
posted by lekvar at 1:46 PM on June 25, 2009

I always like applying crackpot metrics to themselves. A lot of them don't fare very well— they're often just codifications of gut reactions and biases, which, albeit sometimes quite accurate depending on whose gut we're talking about, are not science.
posted by hattifattener at 1:56 PM on June 25, 2009

@saulgoodman

Shermer posted an explanation recently about why his economic views are what they are. I think he leaves his kit home on those days.
posted by jfrancis at 1:58 PM on June 25, 2009

Back when I was a lad, my Granddad had a little baloney detection kit that he introduced me to, a little book you might have heard of called The Bible.

He said "If its in this book, its baloney."
posted by Joey Michaels at 2:06 PM on June 25, 2009 [5 favorites]

I always like applying crackpot metrics to themselves. A lot of them don't fare very well— they're often just codifications of gut reactions and biases, which, albeit sometimes quite accurate depending on whose gut we're talking about, are not science.

Word! I really wish someone would write a version of a crackpot detection system moderated/improved/altered by the admission that logical positivism[1] is more than a little bit busted. IMO most of the annoying wrongness of the self-described skeptic community is their dogged, irrational attachment to deeply self-contradictory ideas of how knowability works. The fact that most of the people they're deploying their wrongness at are themselves much, much more wrong somehow makes the capital-S Skeptics' own wrongness even more frustrating.

[1]: This might not be the right term... I'm using it more or less as shorthand for "people who think empiricism proves its own rightness"
posted by You Can't Tip a Buick at 3:22 PM on June 25, 2009

Well, it's been pointed out before that induction from empirical information has its flaws as a way of understanding the world, but what alternatives do we have?
posted by hattifattener at 3:39 PM on June 25, 2009

"people who think empiricism proves its own rightness"

It isn't that empiricism proves its own rightness, it's just that it's the only source of information that comes from outside our own heads. If there's an objective reality (and there appears to be), empiricism alone gives us clues as to its nature.

But come on, be honest. People don't hate skeptics because they have a problem with empiricism. People hate skeptics because skeptics tell them that they can't ever be a vampire, that their alien abduction stories probably aren't true, and that they can't read each others' minds. The philosophical objections only get raised after the fact as a justification that doesn't sound as stupid as "But but but I really do have psychic powers guys, honest!"
posted by Pope Guilty at 4:57 PM on June 25, 2009 [4 favorites]

people who think empiricism proves its own rightness

If you can name even one other worldview that has put humans on the moon, I'll eat my hat.
posted by DU at 5:38 PM on June 25, 2009 [3 favorites]

If you can name even one other worldview that has put humans on the moon, I'll eat my hat.

Humans on the moon, eh? Prove it.

(tee hee)
posted by Sys Rq at 5:44 PM on June 25, 2009

Damn, I was hoping this would be something else entirely...
posted by mnb64 at 6:53 PM on June 25, 2009

Humans on the moon, eh? Prove it.

Humans, eh?
posted by fleetmouse at 7:43 PM on June 25, 2009

Flag on the moon. How'd it get there?
posted by Snyder at 8:36 PM on June 25, 2009

The moon: flag it and move on.
posted by lekvar at 9:47 PM on June 25, 2009 [2 favorites]

This list isn't exactly exhaustive, they missed at least one:

13. Does the person making the claims stand to gain financially or otherwise by convincing others?
posted by embrangled at 11:05 PM on June 25, 2009

Here, nevertheless, Shermer goes on working from the assumption that free market economic theory has a rational, scientific basis, seemingly oblivious to the fact that science actually isn't on his side.

On the Cato Institute's podcast, you can find a talk Shermer gave there. It's an unfortunate, bad talk, because Shermer plays up to the audience, repeatedly saying things like "Hey, if you guys can believe in evolution, maybe we can make liberals believe in capitalism, hahahha!" to snickers of laughter from the audience.

It's a revealing joke. For Shermer and his audience, the free market is as solid and obvious a truth as natural selection, and the skeptics - be they anti-capitalists or creationists - are all of the same cloth, foolish and contemptible. The blanket labeling of free market critics as "liberals" is also revealing.

This isn't to say that Shermer is thus worthless or unreliable. It's just that he - and any other critic or skeptic - has blind spots, places where their reasoning fails. We all do, and it is good to remember this.
posted by outlier at 2:18 AM on June 26, 2009 [2 favorites]

[i]13. Does the person making the claims stand to gain financially or otherwise by convincing others?[/i]

This is certainly a valid question to ask, but it can easily be twisted and abused. We've seen this over the past eight years when, whenever someone tried to expose some wrongdoing by the Bush administration, the first howl was, "He's got a book he's selling!" as though this automatically invalidated the claim.
posted by Legomancer at 7:08 AM on June 26, 2009

Very true, Legomancer. Then again, most of the items on the list are vulnerable to that problem, particularly this one:

10. Are personal beliefs driving the claim?

Pish posh, global warming? You're just obsessed with your your own personal beliefs. Plus, you have a book coming out. Textbook baloney!

Hence the list not being exhaustive. Baloney detection is just too complex a process to be summed up in bullet points. It's just not possible to assess a claim according to those criteria without your own personal ideologies coming into play and potentially polluting your judgement.

4. Does this fit with the way the world works?

Surely skeptics ought to know better than anyone that beliefs about "how the world works" vary wildly from person to person? Isn't that sort of the problem? Pose a question like that to any kind of religious fundamentalist and I very much doubt the answer would meet a skeptic's definition of baloney-free.
posted by embrangled at 7:36 AM on June 26, 2009

I'm annoyed that eight out of ten of these are "yes/no" questions; as if the questions themselves were rhetorical or perhaps as if these questions only apply to very particular cases which can be definitely decided. Furthermore most out-and-out fraudulent claims come with a battery of deceptive supporting material to answer simplistic stuff like this. Claims are hardly ever single entities, but systems with complex internal interactions and behaviors of their own.

I'm not going to say that the following questions are good baloney detectors, but I have found them useful and perhaps others will too. Adjust as necessary or desired.

What would prove this claim false?

(Many extraordinary claims are unfalsifiable. But there are shades of this: they may not be practically testable, they may undecidable in some other way, paradoxical, or tautological. A claim could be any of these but still useful one way or another too.)

What is the value of this claim? What are the costs of believing this claim?

How does the claimant benefit from making these claims? How would I benefit from believing in this?

Presuming no malice but not assuming altruism, how would you reward the claimant for this claim?

What other changes might believing in this claim inspire in my life?

What are my alternatives?


People are social and I expect that intelligence and inquiry are, if not social, perhaps our sociability is emergent from our curiosity. Anyway:

Who else believes this? What are they like? Why might I want to be like them?

What might my neighbors and friends think of me believing in this?

What worldview is this claim part of? What role does it play in the lives of those who believe in it?

What might be the benefit if everyone in the community believed this? What might be the harm?


Beliefs often seem intensely personal even when they are not. It's takes some work to untie the subjective from the objective (even grammatically). I think think these questions are a good start:

Why do I want this? Why am I afraid of that?
posted by wobh at 8:14 AM on June 26, 2009

It seems to me that the skeptics have a singular view of what science is and what it is not. They do not appear to take into account that scientific theories are not statements of how things are, but are particular ways of understanding how things are that are capable of answering some otherwise unanswered or previously unasked questions. They can be confirmed but not proven. A theory (the word shares its root with theatre) is actually a way of seeing that mar or may not be useful. But it can no more be a final theory than a poem can be a final poem after which no other poem need ever be written. And science is all about theories. Once upon a time there were laws but most of these have been repealed or at least revised, over the centuries.

Beware Scientism, a new religion much like all the others. But it is the deity worshipped by the Skeptics.
posted by donfactor at 9:07 AM on June 26, 2009 [1 favorite]

They do not appear to take into account that scientific theories are not statements of how things are, but are particular ways of understanding how things are that are capable of answering some otherwise unanswered or previously unasked questions.

Yes, skeptics say "the way the world is" rather than "the way that all evidence and experimentation and observation up to this point, as best we understand it, points to the underlying truth of the world most likely and in all probability working" because we worship the current scientific understanding. That's by far and away the most likely and least absurd reason.
posted by Pope Guilty at 9:38 AM on June 26, 2009

There's a frustrating subset of skeptics that are hard core libertarians. Despite the difficulty in determining empirical truth in the social sciences as compared to the "hard" ones, they view economics as proven science and go from there. Shermer and Penn Jillet, both of whom I respect deeply are afflicted with this blind spot. There have been attempts lately to separate both atheism and libertarianism from the Skeptical movement, attempts that I feel are long overdue, as it has the unfortunate tendency to cause people to remove themselves from the movement or not join it when they see things that they feel are not as well back up as most of the claims trumpeted by some of the big names.

That said, this list is quite good, if only a beginning.
posted by Hactar at 10:22 AM on June 26, 2009 [2 favorites]

Please tell me that I'm not the only one who read the link as "Joan Baez's crackpot scoring system" and wondered what in the world any of this had to do with folk music.

Nope; me too.
posted by ZenMasterThis at 7:55 PM on June 26, 2009

Neil Tyson on our lack of skepticism
posted by homunculus at 10:36 AM on June 30, 2009

John and Joan Baez are cousins.
posted by lukemeister at 11:06 AM on July 13, 2009