"medicine is broken" - Ben Goldacre, co-founder of AllTrials

February 4, 2013 2:06 PM

Ben Goldacre has been talked about here before. This year, following on from his book Bad Pharma - where he described a culture of badly-done medical trials on unsuitable subjects (sometimes with horrific but, tragically, potentially preventable outcomes), where swathes of results that don't reflect well on the drug in question go unpublished and even, in some cases, hidden - he has co-founded the project AllTrials, which campaigns for all medical trials to be recorded and reported.

Goldacre claims that reporting all trials that take place would mean that corporations can no longer skew published results in their favour by hiding away the less favourable ones. This means that the drugs which come to market and are prescribed most often are the best possible ones for the job.

But what about competition? Surely there's a market for variety and some kinds might be better than others? Yes, but if a company tells a doctor that their drug is the best, when it cures 6/10 people, and another drug is available that cures 8/10 - then the doctor who is swayed by false results could be missing an opportunity to cure two more people out of every ten.

Furthermore, missing data can lead to potentially crucial data not being published, such as the desperately sad case of a new anti-arrhythmic drug, which was prescribed to people who had had a heart attack as a preventative measure for further attacks. "well over 100,000 people died unnecessarily before it was realised that the fine balance between benefit and risk was completely different for patients without a proven abnormal heart rhythm." (quote taken from excerpt lower down page). This could have been prevented if a 1980 study reporting the issue, although writing it off at the time as an anomaly, had been published. It was subsequently published "with a mea culpa, realising the harm that had been done by not publishing it sooner" but sadly, the damage had been done.

posted by greenish (33 comments total)

As much as this is needed, I can't help but feel this will be used by quacks to 'prove' that Big Pharma is hiding the truth about cancer being a fungus or vaccines causing autism.
posted by Charlemagne In Sweatpants at 2:11 PM on February 4 [6 favorites]

Maybe, but nobody has been a bitter critic of quacks misusing studies, so I suspect he'll do as much as possible to avoid that. That said, quacks are alwyas going to be quacks.
posted by Artw at 2:14 PM on February 4 [4 favorites]

Speaking of quacks, it would be interesting to read all of Burzynski's abandoned antineoplaston trials.
posted by benzenedream at 2:28 PM on February 4 [1 favorite]

Artw: "Maybe, but nobody has been a bitter critic of quacks misusing studies, so I suspect he'll do as much as possible to avoid that. That said, quacks are alwyas going to be quacks."

True on both counts. Goldacre more than adequately established his anti-quackery credentials (for me, anyway) in his first book, Bad Science. Quacks gonna quack, but AllTrials is a much-needed campaign against some horrifyingly poor practices. I hope it gains momentum.

I highly recommend Bad Pharma, by the way.
posted by Zeinab Badawi's Twenty Hotels at 2:48 PM on February 4 [6 favorites]

If you do 20 studies of an ineffective drug with a significance level of 95%, you should expect one of them to show a statistically significant effect.
posted by Salvor Hardin at 2:52 PM on February 4 [7 favorites]

I think Goldacre's point is exactly that, the problem being that the pharmaceutical company then publishes the single study, suppresses as many as possible of the other 19 and makes it all look as rosy as can be.

Then they mess with the regulator so that nothing ever gets done to force them to show that there were 19 other trials which suggest that the drug is useless.

Then they advertise the single result with a massive campaign of advertising so it displaces some other drug, which might have measurable benefits.

Then they advertise it some more so that it is used for things it was barely even tested for.
posted by biffa at 3:00 PM on February 4 [7 favorites]

Full disclosure: I saw Goldacre speak at the LSE in London this evening, and I can tell you, his quack-dar is still on full power.
posted by greenish at 3:04 PM on February 4 [4 favorites]

I can't help but feel this will be used by quacks

I can't help but feel that the last thing a quack will do is willingly acquaint a potential mark with Ben Goldacre.
posted by George_Spiggott at 3:18 PM on February 4 [16 favorites]

Cherry picking data/studies for a meta-analysis is obviously hugely unethical.

Forcing everyone to do so by only publishing the studies with the results you like isn't any better, and has the added bonus of making it impossible for other academics to truly effectively peer-review the study.
posted by jaduncan at 3:22 PM on February 4 [2 favorites]

"Medicine is broken" heh. If only medicine research. There is fraud running into the billions for all kinds of research, with folks scamming the government grant system for duplicate grants and so forth. I was going to do an FPP on this, but there's always the contingent of defenders, so it's like tilting at windmills. The fact is that it is not only medicine research that's broken, but the entire research paradigm, both private and government. Here's a glimpse:

Big Data computation at the Virginia Bioinformatics Institute at Virginia Tech reveals that over the past two decades funding agencies may have awarded millions and possibly billions of dollars to scientists who submitted the same grant request multiple times -- and accepted duplicate funding.

Meanwhile, the multi-decade "war on cancer" continues to show such staggering lack of progress that an "exhaustive national report" found little to cheer on that front. When lay organizations express frustration and call for reform, they get dismissed as cranks by the vested interests. But even high-profile people have spoken out, not that it will help:

DNA pioneer James Watson takes aim at "cancer establishments"

"A day after an exhaustive national report on cancer found the United States is making only slow progress against the disease, one of the country's most iconic - and iconoclastic - scientists weighed in on "the war against cancer." And he does not like what he sees."

Watson is of course an unfortunate spokesman in view of his racist remarks, but he is still involved in ongoing cancer research:

Nobel Laureate James Watson Puts Forth Novel Hypothesis On Curing Late-Stage Cancers

He is of course not the only critic, but he has the kind of prominence that can gather attention. Not that anything is going to change anytime soon, but boy, is the whole shebang broken, medicine research especially, cancer in particular. We need to scrap the whole thing and start over again, because what's happening right now is not getting us anywhere (see the report on abysmal progress in the "war on cancer"). The extensive fraud is not encouraging either.
posted by VikingSword at 3:23 PM on February 4 [1 favorite]

Well; if you think this sort of thing has broken cancer treatment, wait til you see what it has done to psychiatry.
posted by hobo gitano de queretaro at 3:28 PM on February 4 [2 favorites]

As a doctor I share Goldacre's frustration that our broken medical system often makes it hard for me to figure out how to best help patients. Better data is needed.

The depth of my profession's problems used to be touted by only a few brave souls (like Nortin Hadler or Gil Welch ). But now the Instutute of Medicine and the American College of Physicians publically state a third of all medical care is useless, wasting $750 billion annually. Despite spending about twice as much on care Americans have shorter, sicker lives than people in the rest of the industrialized world. American doctors (and patients too) have really screwed up, and anyone who looks closely can see so.

Admitting our faults (on many levels) is important. Unlike quawkery good science can learn from its mistakes. Marty Makary (from my alma mater, although I was before his time) thinks transparency is the best way to save hospitals.

About half of what I do now has scientific evidence of efficacy. American doctors lifted ourselves out of our own 19th century snake-oil qauckery by using rigorous science directed at patient benefit. In the middle of the last century we were inventing real life saving breakthroughs like antibiotics.

Now the best selling class of drugs is atypical anti-psychotics. There aren't that many schizophrenics but there are millions of stressed out depressed people that fail their first antiidepressant so they can be prescribed these powerful and expensive drugs. Today industry comes up with many borderline effective but highly profitable mood drugs marketed with all the dirty tricks Goldacre describes and more.

Sloppy science in the service of profit is shameful. We need to do rigorous science for patients' benefit again if we want to save our profession again.

I advise patients to ask their doctors to stick to giving them the best evidence based treatments.
posted by ed1897 at 3:32 PM on February 4 [11 favorites]

The fact is that it is not only medicine research that's broken...

This is certainly an issue, but the lack of transparency in medical research can definitely kill you.


Transparency--what a novel thought.
posted by BlueHorse at 3:36 PM on February 4

""Medicine is broken" heh. If only medicine research. There is fraud running into the billions for all kinds of research, with folks scamming the government grant system for duplicate grants and so forth. I was going to do an FPP on this, but there's always the contingent of defenders, so it's like tilting at windmills. The fact is that it is not only medicine research that's broken, but the entire research paradigm, both private and government. Here's a glimpse:

Big Data computation at the Virginia Bioinformatics Institute at Virginia Tech reveals that over the past two decades funding agencies may have awarded millions and possibly billions of dollars to scientists who submitted the same grant request multiple times -- and accepted duplicate funding."

That's a good headline, but the actual text is that — assuming that the grants are true duplicates (not similar, overlaying projects with multiple modes), and assuming that the number is consistent with the amount of academic plagiarism discovered with a similar "big data" application (the comparison to plagiarism is only supported by a supposition that the number of grant duplicates missed by the analysis is similar to the number of plagiarisms missed), i.e. the most damning interpretation, that still amounts to only about 2.5 percent of total grant funding. It's a lot of money and a reasonable target for concerns on waste, but to use it as a claim that medical research is "broken" is to go all in on a pair of threes.
posted by klangklangston at 3:37 PM on February 4 [2 favorites]

I'm a doc, and try to be pharm-free (i.e. no lunches, pens, dinners) but you get enough exposure to the industry to know how they operate. It's about resource maximization and harping your product even if it's not the best. Usually, if the FDA has let it out, class differences are rather minor.

Take statins, which I prescribe regularly and without reservation for the prevention of a heart attack in someone whose never had a heart attack: the big trial showed at 4yrs, 5.1% of people on a statin had died from any cause, 5.7% of people on placebo had died from any cause. The absolute risk reduction is 0.6%.

Billions of dollars rest upon differences that small and this one for a drug universally accepted as a great thing.

Imagine something less clear, less specific....yikes....

The prettiest people I see around hospitals are drug reps and even though I've stopped talking to them too much, when I do engage them, they won't get into arguments of this sort with me. They'll only use the term "absolute risk reduction" if it helps them.

Thanks for the info though, never thought of quantifying actual mortality from failed drugs. Wow.
posted by skepticallypleased at 3:43 PM on February 4 [5 favorites]

When the first big push for EBM came about I was very optimistic. Then I woke up. In time, it showed that the problem is that the "evidence" itself is frequently fraudulent to begin with, so what is a doctor supposed to do, when s/he can't trust the "evidence". Hard to practice evidence based medicine that way. To wit, take the shameful story of hip replacement metal on metal joints, where the company making the joints not only knew about the high failure rates, but basically released false data about durability which convinced thousands upon thousands of people to go ahead and get these poison releasers. But nobody is going to jail. So when you get false data, how are you to trust "evidence"? EBM is great in theory, but failing in practice, as a result of garbage in garbage out. I really wonder how physicians feel about trusting claims from pharma about medication, devices and treatments, when there is this level of fraud going on. And of course, the FDA is toothless and underfunded.
posted by VikingSword at 3:44 PM on February 4 [3 favorites]

"In the US, a law was passed requiring trial results to be posted within one year on the website clinicaltrials.gov, and in 2012, it was shown that this law has been ignored in four out of every five trials."

Can someone smarter than me help me out here? Shouldn't this kind of thing (whether a trial had been conducted, and whether the results had been posted) be reasonably easy to track? What would be the logistics involved in monitoring trials and levying escalating fines against Pharma companies for every unpublished result? I know regulatory fines are more often than not mere drops in a vast bucket of corporate capital budgets, but if the cost of withholding information about a failed clininal trial that will never reap benefits for the company is high enough, surely there's got to be incentives somewhere?
posted by Phire at 3:58 PM on February 4 [1 favorite]

re: Anti-quackery, the quote Ben Goldacre gives is: "The existence of flaws in aeroplanes doesn't prove the existence if magic carpets.", which I think is a lovely simile.

He also wrote an NY Times op-ed this week, which US mefites will be pleased to know is the first time a newspaper has fact-checked him after he's submitted an article. In the UK, the author submits the article and it gets published. No problems when it's someone like him writing, a big problem most of the time.
posted by ambrosen at 4:21 PM on February 4 [2 favorites]

This stuff is always so hard to decipher because you have to separate the legitimate concerns about drug marketing (if any) from the knee-jerk mental-illness-denialist "Prozac Nation" garbage.

To wit:

Now the best selling class of drugs is atypical anti-psychotics. There aren't that many schizophrenics but there are millions of stressed out depressed people that fail their first antiidepressant so they can be prescribed these powerful and expensive drugs. Today industry comes up with many borderline effective but highly profitable mood drugs marketed with all the dirty tricks Goldacre describes and more.

Really? People with mild depression are being given powerful anti-psychotics? That's certainly news to me, and to anyone with even a passing familiarity with anything about mental health.

So there's the thing. No one denies cancer or AIDS are real illnesses, or claims that new drugs to treat them, even when imperfect, are "marketing tricks" devised solely for "profit." It all comes down to this stone-aged denial of mental illness as real illness, or at least it's always so closely coupled that the reader is tempted to throw the whole thing out as garbage.
posted by drjimmy11 at 4:52 PM on February 4

I'm not sure if you work in mental health but I certainly can vouch for the fact that both Seroquel (quetiapine) and Abilify (aripiprazole) are given out like skittles in outpatient settings for mild depression, and are even FDA approved for treating mania or non-psychotic bipolar adjuncts. At this point expecting someone to have breakthrough mania on these supposedly anti-manic drugs is more common than not.

It's not just bad psychiatry manifesting as polypharmacy. From the inpatient patient records I've been looking at over the last few months, plenty of nurse practitioners and family docs seem to be riding the wave of big pharm propaganda.

So somebody is telling primary care providers that these anti-psychotic drugs, which carry significant side effects and cost an average of 1k / month, are indicated for bolstering SSRI therapy .....because that's how they are being used now.
posted by hobo gitano de queretaro at 5:13 PM on February 4 [2 favorites]

People with mild depression are being given powerful anti-psychotics? That's certainly news to me, and to anyone with even a passing familiarity with anything about mental health.

*cough*

They aren't merely being prescribed. They've been actively promoted by the manufacturers for off label use:

"The company’s pharmaceuticals subsidiary, Janssen, has been under scrutiny for years over its promotion of Risperdal, which treats symptoms of bipolar mania and schizophrenia. State and federal authorities have said that Janssen promoted the drug for uses it did not have approval for, including dementia in elderly patients, bipolar disorder in children and adolescents, depression and anxiety."

And there were similar lawsuits against Eli Lilly in 2009 for their Zyprexa marketing.
posted by Wemmick at 7:19 PM on February 4 [1 favorite]

I think the reactions here might have gone a little further than may be productive.

When I first heard about clinical trials that only included "treatable patients" I was highly skeptical about the validity of the studies.

Thinking about it further - why treat someone who won't respond to the treatment? If research groups can identify subsets of individuals who will respond well to a novel treatment, that's *fantastic*! You screen patients who will respond to this new medication (better than current offerings, either better amelioration or lesser side effects) and you only give this new medication to the patients who will respond. The patients who don't have a predicted high response rate should be given the standard treatment.

Individualized treatment should be the way to go, but to do it well within the current drug approval system is seemingly fruitless (in terms of FDA approval practices, corporate ROI, and the current for-profit medicine model practiced in the USA). There's *some* basic research funding along these lines but the programs that I've heard about are funded are pie-in-the-sky crap that isn't funded enough to let them bring anything to reality.

There are a ton of FDA rejected medications for numerous conditions. If there was a mechanism to fund researching biomarkers or something to determine why they were effective for a subset of individuals, we'd have a lot more options to treat illnesses and disease.

q.v. drug effectiveness efficacies between people of different ethnic backgrounds, esp. Americans of African descent.
posted by porpoise at 7:31 PM on February 4 [2 favorites]

The NYTimes op-ed that ambrosen linked is excellent. Check it out.
posted by marble at 7:32 PM on February 4

Really? People with mild depression are being given powerful anti-psychotics? That's certainly news to me, and to anyone with even a passing familiarity with anything about mental health.

It's an anecdote, but I know someone who got a Seroquel prescription for insomnia. I'm not in any way a medical professional, but some of the side effects of that drug are scary.
posted by cosmic.osmo at 10:10 PM on February 4

It's an anecdote, but I know someone who got a Seroquel prescription for insomnia. I'm not in any way a medical professional, but some of the side effects of that drug are scary.

I got prescribed Seroquel and suggested horse tranquilizer for my mild depression (and maybe to shut me up) but two bad shrinks in no way suggests that we should stop treating depression with anti-depressants. I think it's more helpful to encourage people to take medication, since there's so much stigma and so many well-funded forces acting against it.
posted by Charlemagne In Sweatpants at 10:22 PM on February 4

Charlemagne - it might be easier to encourage people to take medication if we were more confident that it was the best available for the problem in question! Hence my admiration for Goldacre's initiative :)
posted by greenish at 1:54 AM on February 5

Did Ben Goldacre sneak a masturbation reference ("pocket billiards") into his New York Timed op-ed?
posted by wilberforce at 6:14 AM on February 5

I'm all for AllTrials in theory, but as someone very interested in pharma research (in particular fields at least), I'm not sure what I'd do with that data if I had access to it. It's an unfortunate fact that basic scientific literacy isn't enough to make sense of large quantities of data, and if you can't tell what you're reading, all you're doing is switching which authorities you trust.
posted by immlass at 8:11 AM on February 5 [2 favorites]

That's a really good point, immlass, but I think even if you can't personally vet the data, it will reassuring to know that the data is out there for someone to crunch if they want to, and if multiple sufficiently-nerdy-and-interested people with no third-party affiliations crunch the data and come to a different conclusion than the big Pharma folk, then that's pretty damning evidence.

It's like how the Freedom of Information Act (insofar as it's still being applied, these days) may not matter much to your day-to-day life because there's a deluge of data that you can look at that you likely never will, but the fact that you can in theory get access to something is a pretty big boon for transparency.
posted by Phire at 8:32 AM on February 5 [1 favorite]

I was amazed to see GlaxoSmithKline come out in favour earlier today.

Still not 100% certain that's not a Yes Men account or something ...
posted by alby at 9:03 AM on February 5

immlass: My (limited) understanding is that it would matter because someone (and there are institutions and researchers that do this) would be able to carry out properly informed meta studies of the data and come up with meaningful comparisons between drugs. At the moment drugs are being prescribed based on partial information with the result that (1) they may end up on something which is ineffective, less effective than the alternative or even actually damaging and (2) that they/we/health providers are paying more for these less effective drugs, which means less money in budgets for other items, which also is likely to negatively impact on health.
posted by biffa at 9:28 AM on February 5 [1 favorite]

I'd definitely rather have the data out there for other people (statisticians & scientists) to crunch and learn from than not. But just putting the data out there is going to be a limited help, especially in the early stages before the data can actually be crunched by someone who knows what they're doing. And then I as a consumer still have to trust that whoever is doing the meta studies doesn't have an agenda.

The increase in transparency is a boon, but it's not a magic bullet. And I say that as someone with a vested interest in seeing improvements in drug effectiveness.
posted by immlass at 9:59 AM on February 5

No, its not a magic bullet. As Goldacre makes clear in his book (why yes, yes I have just fiunished reading it and am now pretending to some expertise) there are masses more things wrong with the pharmaceutical sector to worry about. But good information is hugely important, it could literally save hundreds of thousands of lives. The science of medicine is a pretty slow business, getting good data and identifying what works best now will have an impact for literally decades on prescribing behaviours, just as not having good data will.

If you want to get some tips on things you can do yourself then his book has plenty of ideas.
posted by biffa at 10:23 AM on February 5