.org! Those Yahoos really Binged that Googling, Siri!
August 8, 2012 12:29 PM Subscribe
Researchers analyze what you get when you "page Dr Google".
[R]esearchers also categorized the results by the organization or group that had generated the website. Out of the 1300 websites identified by the searches, 246 (19%) were retail product review site websites and 250 (19%) were websites associated with specific companies or interest groups. Product review retail websites were also the ones which had the lowest level of medical accuracy (8.5%). On the other hand, government websites and websites of national organizations (as identified by URL ending in .org) had the highest level of accuracy (80.9% and 72.5%, respectively).Though this research only looked at typical phrases for a specific topic using one search engine, different measures seem to indicate that at this time, what search engine will work best can vary on what you ask, where you ask it, and how you ask it.
Since they gave Siri pre-programmed humor for questions like "What are you wearing?", I can't believe they didn't include "What's the answer to Life, the Universe, and Everything"?
What shabby caliber of nerds is Apple hiring over there?
posted by Egg Shen at 1:00 PM on August 8, 2012 [2 favorites]
What shabby caliber of nerds is Apple hiring over there?
posted by Egg Shen at 1:00 PM on August 8, 2012 [2 favorites]
This is pretty meaningless.
First, results from past the first page of Google are checked: but no-one ever looks at these. They search for something else.
Second, a page that doesn't answer the question (e.g. is a product page, or is behind a paywall) is counted as a failure. But if you're smart enough to use Google, you're smart enough to read "This article is available for $18" and hit your back button and try the next link, so that's a meaningless count.
Third, we aren't given the figures for the accuracy of medical professionals (the alternative) as a comparison: are they 80% right? 90%? 70%?
My wife is pregnant - hooray! - and has already spotted incorrect information from the assigned medical professional. I mean "information that is not in line with the official National Health Service policy on their website" not "information that is not in line with my personal woo anti-immunisation wackiness" stuff. Nothing major, of course, fingers crossed. And of course the NHS site is great.
Actually, that'd be a great study: medical professional versus NHS website - who wins? One of the academics I worked with did some work on computerisation in the 1970s (his PhD supervisor was Alan Turing!) and his anecdote was that the doctors at his test hospital had the expert system taken out because their students could look things up more easily than they could, undermining the whole idea of doctorial seniority. And we can't have that! (Again, this was an anecdote...)
posted by alasdair at 1:01 PM on August 8, 2012
First, results from past the first page of Google are checked: but no-one ever looks at these. They search for something else.
Second, a page that doesn't answer the question (e.g. is a product page, or is behind a paywall) is counted as a failure. But if you're smart enough to use Google, you're smart enough to read "This article is available for $18" and hit your back button and try the next link, so that's a meaningless count.
Third, we aren't given the figures for the accuracy of medical professionals (the alternative) as a comparison: are they 80% right? 90%? 70%?
My wife is pregnant - hooray! - and has already spotted incorrect information from the assigned medical professional. I mean "information that is not in line with the official National Health Service policy on their website" not "information that is not in line with my personal woo anti-immunisation wackiness" stuff. Nothing major, of course, fingers crossed. And of course the NHS site is great.
Actually, that'd be a great study: medical professional versus NHS website - who wins? One of the academics I worked with did some work on computerisation in the 1970s (his PhD supervisor was Alan Turing!) and his anecdote was that the doctors at his test hospital had the expert system taken out because their students could look things up more easily than they could, undermining the whole idea of doctorial seniority. And we can't have that! (Again, this was an anecdote...)
posted by alasdair at 1:01 PM on August 8, 2012
Third, we aren't given the figures for the accuracy of medical professionals (the alternative) as a comparison: are they 80% right? 90%? 70%?
Well, I'd very much hope they'd be more than 82% correct on infant cigarette smoking. There are plenty of people who can google but are not adept at working out that something is a retail site. Often because retail sites and drug companies work hard at disguising themselves as places where you are getting unbiased information. And that's before you start hitting issues like immunisation where the anit-immunisation crowd are very good at slathering their crap all over the internet.
posted by lesbiassparrow at 1:16 PM on August 8, 2012
Well, I'd very much hope they'd be more than 82% correct on infant cigarette smoking. There are plenty of people who can google but are not adept at working out that something is a retail site. Often because retail sites and drug companies work hard at disguising themselves as places where you are getting unbiased information. And that's before you start hitting issues like immunisation where the anit-immunisation crowd are very good at slathering their crap all over the internet.
posted by lesbiassparrow at 1:16 PM on August 8, 2012
"results from past the first page of Google are checked: but no-one ever looks at these"
Ah. That explains a lot. I've often wondered why people have difficulty tracking information that I find quite easy to uncover.
posted by lastobelus at 1:18 PM on August 8, 2012 [2 favorites]
Ah. That explains a lot. I've often wondered why people have difficulty tracking information that I find quite easy to uncover.
posted by lastobelus at 1:18 PM on August 8, 2012 [2 favorites]
First, results from past the first page of Google are checked: but no-one ever looks at these.
I think that would have made their results less accurate. The first page of most of our baby related questions was almost always BabyCenter, followed by various mommy forums and question/answer sites of various repute (it is nice to know Yahoo! Answers figured out how babby was formed). Government websites or places of actual medical repute are usually non-profit and don't have very good SEO.
posted by Gary at 1:35 PM on August 8, 2012
I think that would have made their results less accurate. The first page of most of our baby related questions was almost always BabyCenter, followed by various mommy forums and question/answer sites of various repute (it is nice to know Yahoo! Answers figured out how babby was formed). Government websites or places of actual medical repute are usually non-profit and don't have very good SEO.
posted by Gary at 1:35 PM on August 8, 2012
I was able to get the original article (here; it may be behind a paywall for some) and I think the results could have been skewed in a couple of ways. First, they looked only at advice involving infants (although this is understandable in a pediatric journal). As anyone who has had a baby knows, there is a huge surplus of advice on infants in the world, so there is a lot more bad information out there to be found. If they looked at something relatively few lay people have heard of, say, VATER syndrome, they might have gotten more good results. On the other hand, some of their questions were pretty obvious; as lesbiassparrow alluded to above, correct advice concerning smoking and babys should approach 100% (and I hope relatively few people would need to Google it to find out; although I do see a lot of parents who come in reeking of smoke with their infants. They don't strike me as having access to computers, though.).
If I link to sites with medical information, I generally limit myself to government sites such as the CDC, well-known medical organizations such as the Mayo Clinic, and occasionally a scholarly site such as the Cochrane Collaboration or a journal such as NEJM. I am very careful about medical information on sites by or sponsored by for-profit entities.
posted by TedW at 1:37 PM on August 8, 2012 [1 favorite]
If I link to sites with medical information, I generally limit myself to government sites such as the CDC, well-known medical organizations such as the Mayo Clinic, and occasionally a scholarly site such as the Cochrane Collaboration or a journal such as NEJM. I am very careful about medical information on sites by or sponsored by for-profit entities.
posted by TedW at 1:37 PM on August 8, 2012 [1 favorite]
Yeah, I stopped when I got to "Since most parents would probably read the first few pages of the Google search results, the researchers only analyzed the first 100 websites identified by each of the thirteen Google searches (total of 1300 websites)."
Nobody reads 1300 websites trying to get advice on something like what position their baby should sleep in. What you want to do here is you want to get a group of new parents, let's say 200 couples, and have them do a questionnaire in which they try to provide answers to a dozen or so common infant-health questions. Then you turn half of them loose on Google and have them research those questions for, say, an hour each. The other half sit down with a pediatrician (not one of the researchers, and not a pediatrician who had just been briefed on the official AAP recommendations, but a regular pediatrician -- you'd need a sample group of pediatricians too, say about 20 or so) and have a consultation session in which the doctor advises them on the topics of the questionnaire, spending an amount of time on each question commensurate to the amount of time normally spent in the course of typical post-natal checkups -- say, 20 minutes per item for a total of four hours' consultation time which is quite a bit of doctor time in this day and age.
After this you have all the couples go back and do the questionnaire again and see how the two groups stack up against each other in terms of how well their answers agree with the AAP recommendations for those issues. If you want to be really good, have them come back in a week later and do it again, and then a month later and do it a final time, to see if they retain that information.
Now that is a study that might produce some interesting results. This, though? Frankly all it produced was some interesting bollocks. It's a lot cheaper to just have a few grad students poke at Google for a few hours according to some arbitrary methodology and then analyze the results than it is to do a proper study on the matter, but it doesn't give you the quality of data that is needed for actually making policy decisions or adjusting guidelines. All it creates is some sciencey speculation and some nice PR in Scientific American. Humbug.
posted by Scientist at 2:49 PM on August 8, 2012 [4 favorites]
Nobody reads 1300 websites trying to get advice on something like what position their baby should sleep in. What you want to do here is you want to get a group of new parents, let's say 200 couples, and have them do a questionnaire in which they try to provide answers to a dozen or so common infant-health questions. Then you turn half of them loose on Google and have them research those questions for, say, an hour each. The other half sit down with a pediatrician (not one of the researchers, and not a pediatrician who had just been briefed on the official AAP recommendations, but a regular pediatrician -- you'd need a sample group of pediatricians too, say about 20 or so) and have a consultation session in which the doctor advises them on the topics of the questionnaire, spending an amount of time on each question commensurate to the amount of time normally spent in the course of typical post-natal checkups -- say, 20 minutes per item for a total of four hours' consultation time which is quite a bit of doctor time in this day and age.
After this you have all the couples go back and do the questionnaire again and see how the two groups stack up against each other in terms of how well their answers agree with the AAP recommendations for those issues. If you want to be really good, have them come back in a week later and do it again, and then a month later and do it a final time, to see if they retain that information.
Now that is a study that might produce some interesting results. This, though? Frankly all it produced was some interesting bollocks. It's a lot cheaper to just have a few grad students poke at Google for a few hours according to some arbitrary methodology and then analyze the results than it is to do a proper study on the matter, but it doesn't give you the quality of data that is needed for actually making policy decisions or adjusting guidelines. All it creates is some sciencey speculation and some nice PR in Scientific American. Humbug.
posted by Scientist at 2:49 PM on August 8, 2012 [4 favorites]
P.S. I'm sure Dr. Moon is aware of the shortcomings of this research and I bet she's annoyed at Scientific American for overstating the impact of her study's results. Hopefully Dr. Moon's study was just a pilot project intended to provide provocative preliminary data so that she can get a bigger grant to do a proper study on the issue. We certainly could use some solid research into the results of people researching medical issues online.
posted by Scientist at 2:53 PM on August 8, 2012 [2 favorites]
posted by Scientist at 2:53 PM on August 8, 2012 [2 favorites]
They also missed that a lot of people ask questions of Google like it's a benevolent ancestor spirit.
posted by Sebmojo at 2:55 PM on August 8, 2012 [2 favorites]
posted by Sebmojo at 2:55 PM on August 8, 2012 [2 favorites]
On the other hand, I've run into a rash of doctors over the past 5 years giving terrible recommendations, clearly motivated by 1. profit, and/or 2. ego. It sucks, but I don't trust as a given that a doctor has my best interests at heart anymore. Not to mention, some doctors are good, some are bad, and some are mediocre. And as a patient, I have no way to filter between them.
At least with Google, I can compare many peoples recommendations, peruse patients' results (both positive and negative), etc. It may well bite me someday, I trust my own bullshit detector more than I trust doctors. So far, this has saved me two major, unnecessary surgeries and prevented a staph infection that was misdiagnosed as fungal from spreading catastrophically. Thanks Google!
posted by LordSludge at 3:17 PM on August 8, 2012
At least with Google, I can compare many peoples recommendations, peruse patients' results (both positive and negative), etc. It may well bite me someday, I trust my own bullshit detector more than I trust doctors. So far, this has saved me two major, unnecessary surgeries and prevented a staph infection that was misdiagnosed as fungal from spreading catastrophically. Thanks Google!
posted by LordSludge at 3:17 PM on August 8, 2012
I originally read this as Dr. Giggles. And that's a completely different issue.
posted by Splunge at 3:25 PM on August 8, 2012
posted by Splunge at 3:25 PM on August 8, 2012
This is strangely relevant to my issues. This evening, I had an unusual and upsetting situation with my teeth, and I consulted Dr. Google. The most relevant possibility presented, taking into account all symptoms, was that I have trench mouth. I'm going to go ahead and get a second opinion from an actual dentist.
posted by Countess Elena at 5:56 PM on August 8, 2012 [1 favorite]
posted by Countess Elena at 5:56 PM on August 8, 2012 [1 favorite]
One of the things I have found most frustrating about online medical resources is the fact that they rarely tell you how important a given sign or symptom is to a diagnosis. Like with Trench Mouth, are you likely to have it if you have everything except "grayish film on the gums"? What about if you don't have a fever, bad breath, or painful gums but your gums are red and pockmarked with ulcers? What if your gums bleed in response to pressure, but not "profusely"?
Diagnosis is tricky stuff, it's not just a matter of having 75% of the official symptoms according to PubMed Health. That's really easy to forget if there's some mysterious thing that appears to be wrong with your body and you're trying to research it with no formal training and maybe just a touch of hypochondria. Not that I'm saying that that describes you, Countess Elena; I'm speaking in generalities there.
Even when you're dealing with a fairly authoritative source like PubMed or the CDC or the AAP or whatever, it's often very hard to figure out how to effectively use the information provided.
posted by Scientist at 10:08 PM on August 8, 2012 [1 favorite]
Diagnosis is tricky stuff, it's not just a matter of having 75% of the official symptoms according to PubMed Health. That's really easy to forget if there's some mysterious thing that appears to be wrong with your body and you're trying to research it with no formal training and maybe just a touch of hypochondria. Not that I'm saying that that describes you, Countess Elena; I'm speaking in generalities there.
Even when you're dealing with a fairly authoritative source like PubMed or the CDC or the AAP or whatever, it's often very hard to figure out how to effectively use the information provided.
posted by Scientist at 10:08 PM on August 8, 2012 [1 favorite]
& my minor commentary in the title was not meant derisively at Dr Moon now that I'm rereading the finished fpp and first comments; was trying to limit the editorializing of the limited scope & weird (to me) testing method. But if it's a precursor to real world testing .... Then yay.
Side notes: they mentioned .org but not .gov (& I assumed they limited it to US centric sites but I'll see if I can get the study to read). .gov is hugely impossible to get, whereas I've never seen significant validation of requirements/met for registry of .org sites.
I guess I'm a weirdo in going to more than one page of results. But sometimes I'll just look at the first page & rewrite my query based on the results, to pick a related phrase or exclude spurious results by querying -[$results clutter].
posted by tilde at 11:15 PM on August 8, 2012
Side notes: they mentioned .org but not .gov (& I assumed they limited it to US centric sites but I'll see if I can get the study to read). .gov is hugely impossible to get, whereas I've never seen significant validation of requirements/met for registry of .org sites.
I guess I'm a weirdo in going to more than one page of results. But sometimes I'll just look at the first page & rewrite my query based on the results, to pick a related phrase or exclude spurious results by querying -[$results clutter].
posted by tilde at 11:15 PM on August 8, 2012
TedW thanks for the link. Looks like they just didn't spell out .gov but they were looked at.
posted by tilde at 11:20 PM on August 8, 2012
posted by tilde at 11:20 PM on August 8, 2012
« Older Ishi, last of the Yahi: of two worlds, and in... | "Industrial Revolution Nightmare Dream House" Newer »
This thread has been archived and is closed to new comments
When is the next Haley's comment? Responded "You have no meetings matching Haley's"
I think you can hardly fault Siri for not answering the question you didn't ask.
posted by axiom at 12:41 PM on August 8, 2012 [3 favorites]