My wife and I thought this was a really awesome post!
August 24, 2011 1:37 PM Subscribe

Is that review a fake? A new paper from Cornell researchers proposes an algorithm for sussing out fake reviews from websites. Here's a summary of tell-tale signs.
posted by empath (71 comments total) 22 users marked this as a favorite

Uh yeah, of course as soon as this is published, fake reviewers will just adjust their techniques. Like when Bayesian spam filtering came out and spam got all surreal.
posted by delmoi at 1:40 PM on August 24, 2011 [4 favorites]

High verb use? Most of my sentences use verbs, sorry!
posted by axiom at 1:45 PM on August 24, 2011 [8 favorites]

I husband and I think this is all very silly!
posted by Windigo at 1:46 PM on August 24, 2011

The researchers used a machine learning approach to develop a classifier that works similarly to a spam-detecting email filter. It works on a probabilistic basis to come up with confidence scores. The researchers set a cutoff past which a review is classified as fake. I haven't read the paper, but I imagine that the classification works on word tokens tagged with the part of speech (rather than some sort of holistic measures), and that no one word or class of words is a useful predictor of fakeness on its own. Thus far, it seems a useful application of machine learning, but not particularly surprising from a scientific viewpoint.
posted by Nomyte at 1:47 PM on August 24, 2011 [2 favorites]

Deceptive Indicators
* Greater use of first-person singular: "I" or "me"
* Use of "!" and positive emotion
* High adverb use: "very" and "really"

I have apparently never read a real review on any website.
posted by 2bucksplus at 1:48 PM on August 24, 2011 [25 favorites]

This is why we can't have accurate things.
posted by The Card Cheat at 1:48 PM on August 24, 2011 [38 favorites]

High verb use? Most of my sentences use verbs, sorry!

I'd have been interested to see a read review for comparison. I wonder if they tend to be made up more of phrases: "Beautiful room, great view of the lake, good service. Very happy, thanks!"
posted by not that girl at 1:48 PM on August 24, 2011 [6 favorites]

I...wow. I've made exactly one review on Yelp. According to Cornell's tell-tale signs, I am fake. Maybe I ought to revise how I interact with the world. Yeesh. (...I mean, yeesh!)

Here is the review if you want to see for yourself how ridiculous I am.
posted by phunniemee at 1:49 PM on August 24, 2011 [4 favorites]

High verb use? Most of my sentences use verbs, sorry!

This sentence no verb. That not hard.
posted by tommasz at 1:49 PM on August 24, 2011 [17 favorites]

I have apparently never read a real review on any website.

This is probably true.
posted by empath at 1:53 PM on August 24, 2011

I accidentally the whole review.
posted by griphus at 1:53 PM on August 24, 2011 [7 favorites]

This is precisely why I read Metafilter, which is an online site where users share links and interesting conversation, which I love!
posted by Bunny Ultramod at 1:53 PM on August 24, 2011

In principle, review sites could address this problem fairly effectively by monitoring IP addresses, think spamhaus but secret.
posted by jeffburdges at 1:54 PM on August 24, 2011

My partner and I just read your awesome post on the front page of Metafilter.com, empath! We were very very pleased with our reading. It was highly informative, thoroughly enjoyable, and we had a wonderful time!

I hope that you will make many more excellent posts such as this one!

Okay, I think I have the surefire way of determining if a review is fake. If you can read it perfectly well as deadpan sarcasm, then it's probably fake.
posted by empath at 1:54 PM on August 24, 2011 [10 favorites]

I forget the show on PBS i saw, but it talked about how some people to simply go around the web with different aliases and "review" products, post on forums, etc. It was depressing watching them work, kind of worse than a bot, since bots tend to not seem "real" to me. Went beyond "fanboy" or "hater" and kind of made me not like humanity a bit there. Some other fact they pointed out was that people rarely bother reviewing things unless they have passionate dislike or love of the product, if they are simply satisfied, they won't bother. So you really only get people who have had troubles with it, adore it, but rarely in between.
posted by usagizero at 1:56 PM on August 24, 2011 [4 favorites]

This is excellent post. I am it very interesting. Get shoes latest fashions at low prices www.fashionshoes.ru
posted by TheWhiteSkull at 2:00 PM on August 24, 2011 [12 favorites]

I've never quite figured out what "knowledgeable staff" means. They'll help with your kid's algebra homework? They know what street the museum is on...in their own home town?
posted by Thorzdad at 2:01 PM on August 24, 2011

The bricks-and-mortar counterpart is retail stores that use customer surveys (the crap on your receipt that says you might win a $10,000 gift certficate by taking a brief phone survey) to gauge employee and/or department performance. Very, very few people take surveys if they are just "satisfied" (unless they're just trying to win the gift certificate) but get one asshole on a bad day and boy, watch your performance scores plummet.
posted by Curious Artificer at 2:03 PM on August 24, 2011

The academic paper is quite interesting even without getting into the technical details. For one thing, they conclude that "the detection of deceptive opinion spam is well beyond the capabilities of human judges."
posted by exogenous at 2:03 PM on August 24, 2011 [1 favorite]

This is fascinating. A review I posted on Yelp got filtered out once and I've wondered for months what caused that.

I had written it for a spa whose owner had gone truly out of her way to satisfy me on an issue that had initially made me unhappy. I decided to thank her by writing the positive review, focusing on the quality of customer service which was really outstanding.

I did mention the name of the business, used a lot of first person singular pronouns, a good number of verbs, and gave them five stars.
posted by Dragonness at 2:03 PM on August 24, 2011

Apparently I write in fake review voice when I write reviews! Who knew? These all strike me more as "person not used to writing from their own voice" and "stilted primary school writing style" more than fake. Which weirdly, I switch to that voice when writing a web review, because I KNOW no one is going to read it if I write it in my voice.
posted by strixus at 2:03 PM on August 24, 2011 [4 favorites]

That was in response to usagizero.
posted by Curious Artificer at 2:03 PM on August 24, 2011

Egads, this sort of article always convinces me that, not only do people think my reviews are fake, but that there's a reasonably good chance that I would fail the Turing test.
posted by Joey Michaels at 2:22 PM on August 24, 2011 [9 favorites]

I think this is a really great New York Times article and I enjoyed reading it very much. It was really informative. I like the fact that the New York Times writes amazing articles like this one! I learned a lot.

My husband and I and our entire family read this comment by griphus, and we all really laughed very hard! Griphus used sarcasm really well to make a humorous point about the techniques described in the article linked to in the FPP (which we have not had a chance to read yet, but we are sure that it is really insightful). Also, the minibar helped my husband and I write this.
posted by GenjiandProust at 2:27 PM on August 24, 2011 [6 favorites]

Very, very few people take surveys if they are just "satisfied" (unless they're just trying to win the gift certificate) but get one asshole on a bad day and boy, watch your performance scores plummet.

This is why you pay secret shoppers and not depend on self selected customer surveys. Any store that seriously graded you on the latter is just grossly incompetent.
posted by empath at 2:28 PM on August 24, 2011

Can you elaborate on that?

What makes you believe that there's a reasonably good chance that you would fail the Turing test?
posted by rkent at 2:30 PM on August 24, 2011 [5 favorites]

Uh yeah, of course as soon as this is published, fake reviewers will just adjust their techniques. Like when Bayesian spam filtering came out and spam got all surreal.

Oh man, those were the halcyon days... my inbox was just awash in these word-salad missives of utter poetic bizarreness. I remember catching one that was so beautiful in its spastic language-hemorrhaging that I inadvertently opened it. I was immediately disappointed, not because the reveal didn't clarify the mystery, but that there wasn't MORE of that fractal gibberish unfolding inside the body of the email. The only reason I don't try to reproduce some of the text from memory is that I'm afraid by forcing it into reality deliberately, I would crush the delicate wonder of its non-construction, like catching butterflies with a baseball mitt on.
posted by FatherDagon at 2:32 PM on August 24, 2011 [18 favorites]

The names on that stuff were great, too. I once got markov-spam from a Harold Q. Octagons.
posted by griphus at 2:34 PM on August 24, 2011 [14 favorites]

That looks exactly like any review I've ever written. Just because one uses better grammar does not make it a fake review.
posted by Malice at 2:37 PM on August 24, 2011 [1 favorite]

of course as soon as this is published, fake reviewers will just adjust their techniques. Like when Bayesian spam filtering came out

But if the paper is right and it can be turned into a generally applicable filter (each of which I'm more than a bit skeptical about), there's no easy way for lying review writers to adapt — the study's technique is based on the unconscious probabilistic language-use tendencies of humans who happen to be lying. Computers don't write the kind of fake reviews under discussion the way they write spam, because the point is that ultimately human readers should believe the reviews are genuine.
posted by RogerB at 2:39 PM on August 24, 2011

In principle, review sites could address this problem fairly effectively by monitoring IP addresses, think spamhaus but secret

it's a good thing that doesn't have any potential for abuse or anything

also, using exclamation points is cheesy if serious and grotesque if ironic.
posted by This, of course, alludes to you at 2:39 PM on August 24, 2011

"High verb use? Most of my sentences use verbs, sorry!"

Yeah, but are those verbs . . . high?
posted by Eideteker at 2:50 PM on August 24, 2011

Yeah, but are those verbs . . . high?

No, man, high is an adjective/adverb. And occasionally, sometimes, a noun.
posted by axiom at 2:58 PM on August 24, 2011

Editor: This is a joke, right? I mean this is the stupidest thing I've ever read!
Homer: What's wrong with it?
Editor: You keep using words like "Pasghetti" and "Momatoes" You make numerous threatening references to the UN and at the end you repeat the words "Screw Flanders" over and over again.
posted by blue_beetle at 2:58 PM on August 24, 2011 [2 favorites]

What makes you believe that there's a reasonably good chance that you would fail the Turing test?

Because I'm a robot.

Damnit! You got me again! Okay - start over. Ask me anything. Anything at all, human.
posted by It's Raining Florence Henderson at 3:05 PM on August 24, 2011 [3 favorites]

What if the fake review coincidentally is true.

(Sorry for blowing your mind.)
posted by found missing at 3:07 PM on August 24, 2011 [1 favorite]

Underrepresented terms in fake reviews:
"crap"
"shit"
"stinks"
"forgot"
"ruined"
"assholes"
"avenge"
"ground glass"
"fecal"
"projectile"
"undead"
"botulism"
"expensive"
"urinal cake"
"single pubic hair"
posted by benzenedream at 3:12 PM on August 24, 2011 [17 favorites]

The weak point of the paper is that they used Mechanical Turk to write all the fake reviews, while for the truthful opinions: "we mine all 6,977 reviews from the 20 most popular Chicago hotels on TripAdvisor."
Do they really believe everything on TripAdvisor is the truth? In fact by selecting only the most popular hotels that probably guarantees a higher number of fake reviews.

They eliminate non-5-star reviews; non-English reviews; and reviews with fewer than 150 characters but I rather doubt that guarantees that all the others are 100% truthful.
posted by Lanark at 3:13 PM on August 24, 2011 [2 favorites]

'!' -- fake review
'!! !!! !! !1! !' -- real deal
posted by mazola at 3:18 PM on August 24, 2011

non-English reviews

I've seen a few Italian-language TripAdvisor negative reviews, followed up shortly by the hotel owner/manager himself logging in to defend his honor, Italian style. Good times. Especially fun if you imagine them being shouted out loud by Sophia Loren and Marcello Mastroianni at each other.
posted by gimonca at 3:20 PM on August 24, 2011 [2 favorites]

Deceptive Indicators
* Greater use of first-person singular: "I" or "me"
* Use of "!" and positive emotion
* High adverb use: "very" and "really"

I'm not convinced these are indications something is fake. Either a good number of people actually write like this, or Facebook, gmail and hotmail, and my family are part of an elaborate Truman Show-like deception at my expense.
posted by Hoopo at 3:24 PM on August 24, 2011 [5 favorites]

I assure you, Hoopo - Facebook, gmail, hotmail, and your family are really very real! Believe me! Very real! Now step away from the sailboat, please. Remember how afraid you are of the water!
posted by It's Raining Florence Henderson at 3:27 PM on August 24, 2011 [6 favorites]

My wife and I read the article listed in the New York Times and were pleasantly surprised. It was informative and well written. While we didn’t get a chance to read the actual paper, it was really nice to have such a concise summary of a topic near and dear to my heart. My wife really enjoyed the awesome callouts on the infograph.

I'm sure I will be reading wonderful summary articles from them in the near future!
posted by Nanukthedog at 3:28 PM on August 24, 2011

...and it's been eight long years since Henry Raddick last reviewed something at Amazon.co.uk.
posted by rh at 3:41 PM on August 24, 2011 [1 favorite]

Windigo: "I husband and I think this is all very silly"

Rastafarian, are we?
posted by dunkadunc at 4:00 PM on August 24, 2011

Multiple humans 9 hours. Pillow, toilet present. No fatality. Acceptable outcome.
posted by vanar sena at 4:22 PM on August 24, 2011 [6 favorites]

Gah. So for what it's worth, the opposite of high verb use isn't "zero verb use," it's "low verb use." For instance, this sentence

Smiling, he turned.

is 66% verb, which is about as high as you can get in a complete declarative sentence. This one

Proper maintenance is the solution to many common automotive problems.

is 10% verb, which is pretty low.

Novels and memoirs tend to be verb-rich; so do human-interest-type news stories — and apparently so do fake reviews. Technical writing and most "hard" news stories tend to be verb-poor — and, apparently, so do real reviews.

These are all just stylistic tendencies and not absolute laws. But as stylistic markers go, verb rate is a pretty robust one.
posted by nebulawindphone at 4:28 PM on August 24, 2011 [3 favorites]

vanar sena, does "toilet present" mean they put a chocolate mint on the toilet seat? My husband and I would love that.
posted by taz at 4:35 PM on August 24, 2011 [2 favorites]

Pillow, toilet prescient. No fate. Acceptable outcome.
posted by It's Raining Florence Henderson at 4:39 PM on August 24, 2011 [1 favorite]

Dude, I have apparently been writing fake reviews for years. Where is my paycheck?
posted by Nattie at 4:48 PM on August 24, 2011

TRAPPED IN NIGERIA
posted by It's Raining Florence Henderson at 4:50 PM on August 24, 2011

In principle, review sites could address this problem fairly effectively by monitoring IP addresses, think spamhaus but secret.

False-review farms could respond by running everything through the hundreds of thousands of ip address available to a botnet.

I think people have been gaming reputation systems since before we've been people.
posted by sebastienbailard at 4:53 PM on August 24, 2011

I hear the Garden of Eden was way overrated.
posted by It's Raining Florence Henderson at 4:56 PM on August 24, 2011

Huh. Most of the things they describe have always struck me as, y'know, the usual review style. I envision a feedback loop where the volume of fake reviews causes real reviewers to write in a more fake-review-esque style. Drag.

Well, it's probably the stylistic difference between someone who's not a writer simply writing their own experiences and someone writing in a narrative style. A "professional" review in a newspaper would be written in a narrative style, but so would most fiction.
posted by delmoi at 5:13 PM on August 24, 2011 [1 favorite]

I hear the Garden of Eden was way overrated.

IP tracing points to a disgruntled Lilith.
posted by joe lisboa at 5:17 PM on August 24, 2011 [1 favorite]

The Disgruntled Lilith Fair was also oeuvrerated.
posted by It's Raining Florence Henderson at 5:19 PM on August 24, 2011 [1 favorite]

A pun-off is like Global Thermonuclear War: the only winning move is not to play.

How about a nice game of pun-off?
posted by joe lisboa at 5:33 PM on August 24, 2011

This reminds me of the time when me and my buddy were driving through the outskirts of Las Vegas and up ahead there was a little dingy motel with a sign outside. It's the typical motel sign that's got a white background with black block letters. Normally that's where you see the rate or "free cable" or whatever.

This sign had no price or list of tempting amenities. Instead it said "Recommended by Owner".
posted by storybored at 6:26 PM on August 24, 2011 [1 favorite]

Full hotel name + exclamation mark = dead giveaway in my books, as well. Also, oddly specific details only the hotel would be proud of, like what I happened to read earlier today: "quality linens!" right in the title.

I've never quite figured out what "knowledgeable staff" means.

That phrase always reminds me of something that happened to my dad. My parents (who are Japanese, so it wasn't a language problem) were at a hotel in Japan. They thought the man who checked them in was slightly odd, but thought nothing more of it. Later, my dad had a rather heavy box he needed to courier, so he carried it to the front desk and asked where the closest courier depot was. The odd man from earlier gave him directions, and off my dad went, on foot -- but found it closed when he got there. Somewhat frustrated, he hauled the box back to the hotel and, as the odd man was still there, told him the courier was closed. "Oh yes, it's closed on Sundays," said the man. Instead of asking why he hadn't mentioned this earlier, my dad asked, "Well, is there anywhere else that can take this box on a Sunday?" The odd man smiled brightly. "Yes! We offer a courier service here!" and whipped out a delivery form.
posted by (alice) at 7:36 PM on August 24, 2011 [7 favorites]

I've never quite figured out what "knowledgeable staff" means.

Heh. In a previous millennium I worked in a hotel in Toronto. I recall a guest from a certain large country to the south who had some very pointed opinions about Canadians in general, and the lack of knowledgeability of service sector employees in particular. I recall one conversation in particular when I was behind the front desk. I am not exaggerating this -- this is as close to verbatim as I can recall:

"Hey, I read something in my guidebook about a castle in Toronto. What is that about?"

"That would probably be Casa Loma. It is a massive Gothic Revival house, and it looks a lot like a castle."

"How long has it been around?"

"It was built just before World War I."

"Who built it?"

"The owner was named Sir Henry Pellatt and the architect was E.J. Lennox."

"How did this Pellatt guy afford it?"

"He was quite a wealthy businessman -- one of the richest men in Toronto at the turn of the century."

"How did he get rich?"

"He built the first hydro-electric plant at Niagara Falls, and helped bring electric light to Toronto."

"What happened to him?"

"Well, he had several business setbacks and died quite poor."

"When did he die?"

"1939."

"Was he living at Casa Loma then?"

"No, he had to give it up and move out in the twenties. He was actually living with his chauffeur when he died."

"So did it become a museum as soon as he moved out?"

"No, it was in private hands and was a nightspot and a luxury hotel for a time. The Kiwanis Club bought it in the thirties and opened it to the public in 1937."

"What does the name mean?"

"Er, Casa Loma is Spanish for 'Hill House'."

"Is it built on a hill?"

[gestures to tourist map on desk] "It is actually on this long ridge that runs across most of the city just above Davenport Road. The ridge is the ancient shore of the lake from just after the last Ice Age."

"Hmm. Where is this Casa Loma?"

"About three miles northwest from here." [marks location on aforementioned map]

"Can I get there by public transit?"

"Absolutely -- take the subway at the corner northbound to Dupont station -- from there it is just north; about a five-minute walk." [circles Dupont on map]

"How much is admission?"

"With tax, $20.55."

"What time does it close today?"

"5:00 PM, but I think the last admission is at 4:00 or so."

"You THINK? Don't you know anything about the place? Aren't you supposed to help guests who ask questions?"

" ... "
posted by ricochet biscuit at 8:32 PM on August 24, 2011 [28 favorites]

What about the anti fake review? Or is that the fake anti review?

"One of the waiters was clearly drunk, and vomited on a number of patrons - thankfully none on our table. When I ordered dessert, our server sniggered and said my creamy custard was "made with LOVE."

Dead set, that's what I would do as a business owner if I suspected competitors were bombing complaints sites.
posted by uncanny hengeman at 9:06 PM on August 24, 2011 [2 favorites]

ie. I'd give a few outrageous fake reviews to my own business. Just in case that comment above was a bit ambiguous.

Come to think of it, maybe I'd do that REGARDLESS of what was being said on review sites.

the next big marketing trend - you heard it here first, folks
posted by uncanny hengeman at 10:53 PM on August 24, 2011

taz: "vanar sena, does "toilet present" mean they put a chocolate mint on the toilet seat? My husband and I would love that."

Tip: toilet presents may vaguely resemble chocolate, but they are almost always not.
posted by vanar sena at 3:09 AM on August 25, 2011 [1 favorite]

* Greater use of first-person singular: "I" or "me"
* Use of "!" and positive emotion
* High adverb use: "very" and "really"
* High verb use

Hulk stay one night Holiday Inn!
Pillow too small for Hulk head!
But bedtime chocolate delightful!
Also Hulk smash fake review checker!
posted by Sutekh at 8:17 AM on August 25, 2011 [1 favorite]

The speech was disturbing, the food inedible and the gift bags... pretty frightening. And when G.O.B. realized he wouldn’t get tipped...
posted by haveanicesummer at 9:33 AM on August 25, 2011

I saw that ep for the first time about 3 days ago!

...Sad times. I only have about 5 left before I finish the whole series.

......But I've never seen an ep of 30 Rock, so there's always that to look forward to, going by internet buzz.
posted by uncanny hengeman at 7:42 PM on August 25, 2011

The only good thing about finishing Arrested Development is you now get the opportunity to watch it again. The show has so, so many riches that can only be found upon repeat viewings. I might be up around 10 times or more for some episodes and I STILL see things I never noticed before.
posted by haveanicesummer at 1:12 AM on August 27, 2011

Totally on board with that. I saw the ethics teacher/civics teacher episode on free-to-air TV last night. Can't believe how much stuff I missed first time around.

"George Michael, what are you doing at a high school dance?"
posted by uncanny hengeman at 11:19 PM on August 28, 2011

« Older What's it like to have your film flop at the box o... | Jerry's Map:... Newer »

This thread has been archived and is closed to new comments

My wife and I thought this was a really awesome post! August 24, 2011 1:37 PM Subscribe

My wife and I thought this was a really awesome post!
August 24, 2011 1:37 PM Subscribe