Pandemic science is out of control
April 28, 2020 1:49 PM Subscribe
A toxic legacy of poor-quality research, media hype, lax regulatory oversight, and vicious partisanship has come home to roost in the search for effective treatments for COVID-19.
Thank you for this. The amount of garbage studies and data is depressing.
I am going to trying and ignore it for a few weeks until the methods get better validation.
But you cannot read a covid thread w/o it turning into battle over studies.
posted by KaizenSoze at 2:04 PM on April 28, 2020 [7 favorites]
I am going to trying and ignore it for a few weeks until the methods get better validation.
But you cannot read a covid thread w/o it turning into battle over studies.
posted by KaizenSoze at 2:04 PM on April 28, 2020 [7 favorites]
Josh Marshall's list of experts to follow
posted by a snickering nuthatch at 2:06 PM on April 28, 2020 [4 favorites]
posted by a snickering nuthatch at 2:06 PM on April 28, 2020 [4 favorites]
I have a new appreciation for the moderation done by scientific journals. Someone on the blue posted a link to nih.gov's aggregator of studies. And some of them are out there..
Scroll through the pages and you eventually stumble upon, a rather scientific sounding study, "Coronavirus (COVID-19), First Indication of Efficacy of Gene-Eden-VIR/Novirin in SARS-CoV-2 Infections", and when you click on the "Full Text" button, you come across this
posted by ecco at 2:18 PM on April 28, 2020 [6 favorites]
Scroll through the pages and you eventually stumble upon, a rather scientific sounding study, "Coronavirus (COVID-19), First Indication of Efficacy of Gene-Eden-VIR/Novirin in SARS-CoV-2 Infections", and when you click on the "Full Text" button, you come across this
"Its formula includes five ingredients, comprising a 100 mg extract of quercetin, a 150 mg extract of green tea, a 50 mg extract of cinnamon, a 25 mg extract of liquorice and 100 µg of selenium."I have no idea how it would mitigate a viral infection. But I like joking with my girlfriend that she's been self treating all this time.
posted by ecco at 2:18 PM on April 28, 2020 [6 favorites]
Medlife Crisis (Dr. Rohin Francis) is a great medical youtube channel which is super entertaining but maybe a little more serious now that he's treating covid-19 patients and a month ago had (presumptive) covid-19 himself. His most recent video was about ventilators, Dunning-Krueger, and Elon Musk.
posted by allegedly at 2:35 PM on April 28, 2020 [8 favorites]
posted by allegedly at 2:35 PM on April 28, 2020 [8 favorites]
That paper that ecco posted gets better. In the "Declaration of Competing Interest" section:
But enough derailing onto weird papers. Thanks, latkes, for posting the original link. Our response to COVID-19 has me thinking a lot about what "evidence-based policy" means, what to do when we don't have that evidence, and the quality of data sets. I'm not trained in these fields at all, and I don't have the answers. I don't know if anyone does, but I'm lucky to live in a place where the government seems to be handling this crisis fairly competently, and I will continue to follow their directives.
posted by invokeuse at 2:36 PM on April 28, 2020 [4 favorites]
HP [first author] is the inventor of the Gene-Eden-VIR/Novirin formula.
But enough derailing onto weird papers. Thanks, latkes, for posting the original link. Our response to COVID-19 has me thinking a lot about what "evidence-based policy" means, what to do when we don't have that evidence, and the quality of data sets. I'm not trained in these fields at all, and I don't have the answers. I don't know if anyone does, but I'm lucky to live in a place where the government seems to be handling this crisis fairly competently, and I will continue to follow their directives.
posted by invokeuse at 2:36 PM on April 28, 2020 [4 favorites]
Along similar lines: Against pandemic research exceptionalism — Crises are no excuse for lowering scientific standards
posted by tonycpsu at 2:37 PM on April 28, 2020 [14 favorites]
posted by tonycpsu at 2:37 PM on April 28, 2020 [14 favorites]
Lancet: A real-time dashboard of clinical trials for COVID-19 - Kristian Thorlund, Louis Dron, Jay Park, Grace Hsu, Jamie I Forrest, Edward J Mills
COVID-19 Clinical Trials Registry (with dashboard)
posted by ZeusHumms at 2:43 PM on April 28, 2020 [5 favorites]
COVID-19 Clinical Trials Registry (with dashboard)
posted by ZeusHumms at 2:43 PM on April 28, 2020 [5 favorites]
The sources I have been using:
Chemical & Engineering News tries to break down the science to that which your average chemist can understand (block cookies to bypass the paywall), and honestly, I suspect most non-chemists can follow 90% of the articles. Leans a bit into industry news and press releases at times, but also some REALLY good articles.
In the Pipeline by Derek Lowe one of the great chemistry blogs, Derek has been working in drug discovery for decades and summarizes the news for us non-drug chemists. He leans a little technical, but his conclusions are in plain, blunt, English.
I also get some papers directly from Science, since I have to keep up with its headlines for work, but they tend to be rather hard to get into and are usually summarized elsewhere.
posted by Canageek at 2:54 PM on April 28, 2020 [12 favorites]
Chemical & Engineering News tries to break down the science to that which your average chemist can understand (block cookies to bypass the paywall), and honestly, I suspect most non-chemists can follow 90% of the articles. Leans a bit into industry news and press releases at times, but also some REALLY good articles.
In the Pipeline by Derek Lowe one of the great chemistry blogs, Derek has been working in drug discovery for decades and summarizes the news for us non-drug chemists. He leans a little technical, but his conclusions are in plain, blunt, English.
I also get some papers directly from Science, since I have to keep up with its headlines for work, but they tend to be rather hard to get into and are usually summarized elsewhere.
posted by Canageek at 2:54 PM on April 28, 2020 [12 favorites]
I think this is overall a good essay, but misses or misstates an important point about the nature of science-based medicine. Science is, by nature, an error-filled process. That is in fact how it progresses, by trying lots of ideas, most of which are wrong, and then methodically evaluating them to find out which ones are the wrong ones and which ones are right, or at least sufficiently less-wrong to be useful until something better comes along. So this:
The problem with science-based medicine in a novel pandemic is that this process of self-correction takes time, and time is exactly what we don't have. So with this quote:
However. Science-based medicine also means that doctors can approach new situations with relevant knowledge from similar, previous cases. In the case of hydroxychloroquine, this drug has been around for a while. Its risks are known in patients without COVID, and there's no reason to think those risks would be any less in those with it. Conversely, while there is a known mechanism by which this drug modulates the immune system in autoimmune diseases, and both in vitro and in vivo evidence from laboratory studies that it can interfere with the replication of human coronaviruses, I've seen no evidence that this drug was ever adequately tested in clinical trials for treating coronaviruses prior to the COVID-19 outbreak. So while there is some reason to think that hydroxychloroquine could be effective against SARS-CoV-2, the drumbeat of politicians and political hacks pushing this drug was premised on nothing stronger than this relatively thin evidence.
So should a doctor treating a COVID patient use hydroxychloroquine, or other similar unproven treatments? Honestly, I'm not sure scientific medicine can answer that question yet. I'm definitely not an infectious disease specialist, so it may be that there has been evidence that I haven't seen, but if it exists it certainly hasn't entered the public discussion. The article is absolutely right that the quality of studies like those of Raoult and Zelenko is abysmally poor for addressing either safety or efficacy, and those studies should not be used to inform anyone's decision making. Certainly it is clear that hydroxychloroquine is not the magic bullet that the feedback whine between Trump and Fox News claimed it was, and the drug is sufficiently dangerous that the answer to Trump's question "what have you got to lose?" was always "quite a lot, actually." But for a front-line physician trying to treat a very sick patient, evaluating the known risks of the drug against its uncertain benefits, choosing hydroxychloroquine may be a reasonable best choice from a set of bad, or at least highly uncertain, options. Or maybe not; perhaps the known risks are simply too great and the potential benefits too slight for it to ever be a good decision. Those kinds of difficult choices are what makes being a doctor so incredibly difficult, and I don't envy them. Hopefully, soon, we will have good science to make that choice easier, but right now we don't, and there is a limit to how quickly we can get that data.
However. The fact that front-line physicians may have to make decisions to use unproven treatments before properly controlled studies can be conducted does not mean those studies are not still needed. The individual cases of recovery or death that doctors will see in the clinic don't add up to a controlled trial, and we should not delude ourselves into thinking that we've learned whether a drug really works from those cases. At best they provide the kind of creative mess of ideas that later studies will winnow down; at worst they will convince clinicians that they know something they don't actually know. We need to keep firmly in mind the difference between scientific confidence and risk-based medical decision making.
posted by biogeo at 3:01 PM on April 28, 2020 [32 favorites]
Indeed, over the past 40 years, many of the procedures, tests, drugs, and medical devices that doctors once believed to be valuable have been abandoned as they proved useless or even deadly.is not evidence for any problem in the science of finding medical treatments, it is the system working as intended. The problem occurs when flawed procedures, tests, drugs, and medical devices don't get abandoned. Certainly there are plenty of cases of such flawed treatments persisting for much longer than they should either because they are not adequately investigated early on or because the evidence against them is ignored, but as a whole the system works remarkably well at self-correcting.
The problem with science-based medicine in a novel pandemic is that this process of self-correction takes time, and time is exactly what we don't have. So with this quote:
On an emergency medicine email list, one physician said, “By the time everyone gets their evidence together in a nice controlled study COVID will have packed its bags and left town with thousands dead.”I'm actually sympathetic to the anonymous ER doc. Well-controlled studies take more time than we have, and doctors operating on the front line have to select treatments from a menu of options available right now, many of which are insufficiently tested for the physician to have confidence in their safety or efficacy. Treating the patient in front of them is their job, and if they wait until all options have been adequately tested in a properly controlled study, that patient may well be dead before they act.
However. Science-based medicine also means that doctors can approach new situations with relevant knowledge from similar, previous cases. In the case of hydroxychloroquine, this drug has been around for a while. Its risks are known in patients without COVID, and there's no reason to think those risks would be any less in those with it. Conversely, while there is a known mechanism by which this drug modulates the immune system in autoimmune diseases, and both in vitro and in vivo evidence from laboratory studies that it can interfere with the replication of human coronaviruses, I've seen no evidence that this drug was ever adequately tested in clinical trials for treating coronaviruses prior to the COVID-19 outbreak. So while there is some reason to think that hydroxychloroquine could be effective against SARS-CoV-2, the drumbeat of politicians and political hacks pushing this drug was premised on nothing stronger than this relatively thin evidence.
So should a doctor treating a COVID patient use hydroxychloroquine, or other similar unproven treatments? Honestly, I'm not sure scientific medicine can answer that question yet. I'm definitely not an infectious disease specialist, so it may be that there has been evidence that I haven't seen, but if it exists it certainly hasn't entered the public discussion. The article is absolutely right that the quality of studies like those of Raoult and Zelenko is abysmally poor for addressing either safety or efficacy, and those studies should not be used to inform anyone's decision making. Certainly it is clear that hydroxychloroquine is not the magic bullet that the feedback whine between Trump and Fox News claimed it was, and the drug is sufficiently dangerous that the answer to Trump's question "what have you got to lose?" was always "quite a lot, actually." But for a front-line physician trying to treat a very sick patient, evaluating the known risks of the drug against its uncertain benefits, choosing hydroxychloroquine may be a reasonable best choice from a set of bad, or at least highly uncertain, options. Or maybe not; perhaps the known risks are simply too great and the potential benefits too slight for it to ever be a good decision. Those kinds of difficult choices are what makes being a doctor so incredibly difficult, and I don't envy them. Hopefully, soon, we will have good science to make that choice easier, but right now we don't, and there is a limit to how quickly we can get that data.
However. The fact that front-line physicians may have to make decisions to use unproven treatments before properly controlled studies can be conducted does not mean those studies are not still needed. The individual cases of recovery or death that doctors will see in the clinic don't add up to a controlled trial, and we should not delude ourselves into thinking that we've learned whether a drug really works from those cases. At best they provide the kind of creative mess of ideas that later studies will winnow down; at worst they will convince clinicians that they know something they don't actually know. We need to keep firmly in mind the difference between scientific confidence and risk-based medical decision making.
posted by biogeo at 3:01 PM on April 28, 2020 [32 favorites]
The problem with science-based medicine in a novel pandemic is that this process of self-correction takes time, and time is exactly what we don't have.
I would argue that the real problem we're grappling with here is that we have a significant percentage of the population so stupid and/or authoritarian-minded that they literally cannot cope with the recognized uncertainty and inevitable errors of the iterative scientific research process. By this I don't mean that they find it difficult and painful to watch medicine grope for treatments for a deadly pandemic; we all do. I mean that they want only diktats at maximum volume, and nothing else. The diktats can even change as long as no one admits error. They regard any entity's self-recognized inability to be infallible as a fatal blow to any credibility, when in fact it merely reflects a responsible engagement with reality rather than an infinite recursion of bullshit. (In the advanced form, they know the diktats are bullshit, too, but they also consider everything bullshit, so they only want the form of certainty in a direction they like.)
These are people fundamentally unfitted to be citizens of a democracy in the postmodern era, much less in a time of crisis like this that requires difficult policy judgments in real time based on insufficient scientific evidence, and I don't know how we engage them in a democracy that is even vaguely in touch with reality.
posted by praemunire at 3:18 PM on April 28, 2020 [77 favorites]
I would argue that the real problem we're grappling with here is that we have a significant percentage of the population so stupid and/or authoritarian-minded that they literally cannot cope with the recognized uncertainty and inevitable errors of the iterative scientific research process. By this I don't mean that they find it difficult and painful to watch medicine grope for treatments for a deadly pandemic; we all do. I mean that they want only diktats at maximum volume, and nothing else. The diktats can even change as long as no one admits error. They regard any entity's self-recognized inability to be infallible as a fatal blow to any credibility, when in fact it merely reflects a responsible engagement with reality rather than an infinite recursion of bullshit. (In the advanced form, they know the diktats are bullshit, too, but they also consider everything bullshit, so they only want the form of certainty in a direction they like.)
These are people fundamentally unfitted to be citizens of a democracy in the postmodern era, much less in a time of crisis like this that requires difficult policy judgments in real time based on insufficient scientific evidence, and I don't know how we engage them in a democracy that is even vaguely in touch with reality.
posted by praemunire at 3:18 PM on April 28, 2020 [77 favorites]
So maybe think twice about signing up for a "challenge trial" .
I saw a document out of Harvard outlining legitimate approaches to increasing the speed and efficiency of drug trials, it seemed to make a lot of sense but can't find the link right now. One issue that I've read in discussions is that a major time sink in medical trials is paperwork and getting the permissions to conduct is a extremely slow process. It just seems like all the non-science slowdowns should be minimized, and with a bunch of vaccines already started some in phase 2, that's actually happening. But yeah first try not kill the test subjects.
posted by sammyo at 3:35 PM on April 28, 2020 [1 favorite]
I saw a document out of Harvard outlining legitimate approaches to increasing the speed and efficiency of drug trials, it seemed to make a lot of sense but can't find the link right now. One issue that I've read in discussions is that a major time sink in medical trials is paperwork and getting the permissions to conduct is a extremely slow process. It just seems like all the non-science slowdowns should be minimized, and with a bunch of vaccines already started some in phase 2, that's actually happening. But yeah first try not kill the test subjects.
posted by sammyo at 3:35 PM on April 28, 2020 [1 favorite]
I mostly agree with you, praemunire, but I think we're talking about two different, though interrelated, problems, both of which need to be solved.
posted by biogeo at 3:52 PM on April 28, 2020 [3 favorites]
posted by biogeo at 3:52 PM on April 28, 2020 [3 favorites]
I have been following most developments on reuters.com - and I think their coverage has been superb.
They are now tracking the various treatments and possible pathways - again excellent.
Another resource https://www.doherty.edu.au/news-events/setting-it-straight
Peter Doherty won his Nobel prize for his research into immune responses - so the articles he is writing are the perfect introduction to the technical issues
posted by Barbara Spitzer at 4:39 PM on April 28, 2020 [3 favorites]
They are now tracking the various treatments and possible pathways - again excellent.
Another resource https://www.doherty.edu.au/news-events/setting-it-straight
Peter Doherty won his Nobel prize for his research into immune responses - so the articles he is writing are the perfect introduction to the technical issues
posted by Barbara Spitzer at 4:39 PM on April 28, 2020 [3 favorites]
As opposed to herbal treatments, homeopathy, and Goop?
posted by Ideefixe at 4:53 PM on April 28, 2020 [1 favorite]
posted by Ideefixe at 4:53 PM on April 28, 2020 [1 favorite]
I really love the book Bad Science by Gary Taubes, which details the frenzy over Pons and Fleischman's cold science debacle. There wasn't the same level of incompetence from the President and his cronies, but the pressure to generate papers ASAP about THE MOST IMPORTANT THING EVAR was the same. One of the takeaways from the book was that meaningful, well controlled studies and papers take time which means the worst papers are front-loaded during the initial flurry of research activity.
posted by benzenedream at 5:27 PM on April 28, 2020 [6 favorites]
posted by benzenedream at 5:27 PM on April 28, 2020 [6 favorites]
A video on why most attempts to quickly build inexpensive ventilators won't work and is more difficult that people realize.
posted by ShooBoo at 6:03 PM on April 28, 2020 [7 favorites]
posted by ShooBoo at 6:03 PM on April 28, 2020 [7 favorites]
In the case of hydroxychloroquine, potential side effects are not trivial. The drug can affect heart rhythm, potentially triggering a rare condition known as torsades de pointes, which can end in sudden cardiac death.
As I understand, lots of drugs are actually capable of causing some degree of QT prolongation (the EKG change associated with an increased risk of torsades de pointes). It tends to become an active concern when a patient is taking a high dose, or more than one such drug. The trouble with the (hydroxy)chloroquine/azithromycin combo is they are both such drugs, plus SARS-CoV-2 seems to affect the heart in its own right in some cases which may also heighten the potential for bad interactions. I think the other big uncertainty is that HCQ is known to act to suppress some aspects of immune response, and we don't really know whether that's harmful or helpful to COVID-19 patients (or it could even vary case-by-case).
I agree with biogeo though that this article is oddly dismissive of the ethical questions around controlled medical trials:
Unfortunately, that study also did not include a placebo arm on the grounds that the researchers thought it would be “unethical” to deprive study participants access to the drug. But given that there is no idea whether the drug causes more harm than benefit, this appeal to ethics might itself be considered unethical.
The question of what the line is where it becomes unethical to give placebo instead of treatment is historically a pretty fucking big deal in medical science!
posted by atoxyl at 6:18 PM on April 28, 2020 [10 favorites]
As I understand, lots of drugs are actually capable of causing some degree of QT prolongation (the EKG change associated with an increased risk of torsades de pointes). It tends to become an active concern when a patient is taking a high dose, or more than one such drug. The trouble with the (hydroxy)chloroquine/azithromycin combo is they are both such drugs, plus SARS-CoV-2 seems to affect the heart in its own right in some cases which may also heighten the potential for bad interactions. I think the other big uncertainty is that HCQ is known to act to suppress some aspects of immune response, and we don't really know whether that's harmful or helpful to COVID-19 patients (or it could even vary case-by-case).
I agree with biogeo though that this article is oddly dismissive of the ethical questions around controlled medical trials:
Unfortunately, that study also did not include a placebo arm on the grounds that the researchers thought it would be “unethical” to deprive study participants access to the drug. But given that there is no idea whether the drug causes more harm than benefit, this appeal to ethics might itself be considered unethical.
The question of what the line is where it becomes unethical to give placebo instead of treatment is historically a pretty fucking big deal in medical science!
posted by atoxyl at 6:18 PM on April 28, 2020 [10 favorites]
'There is no absolute truth': an infectious disease expert on Covid-19, misinformation and 'bullshit' (Guardian)
Carl Bergstrom’s two disparate areas of expertise merged as reports of a mysterious respiratory illness emerged in January
posted by katra at 6:23 PM on April 28, 2020 [12 favorites]
Carl Bergstrom’s two disparate areas of expertise merged as reports of a mysterious respiratory illness emerged in January
The following interview has been edited for length and clarity. [...]And I try to regularly check the Harvard T.H. Chan School of Public Health news link, because it offers an ongoing selection of links to articles that seem to take a more critical approach (and happen to include Harvard T.H. Chan School of Public Health experts speaking to a variety of media outlets).
Are there bullshit narratives about the coronavirus that you are concerned about right now?
What’s happened with this pandemic that we’re not accustomed to in the epidemiology community is that it’s been really heavily politicized. Even when scientists are very well-intentioned and not trying to support any side of the narrative, when they do work and release a paper it gets picked up by actors with political agendas.
[...] one of the biggest things that people [in the media] could do to improve would be to recognize that scientific studies, especially in a fast-moving situation like this, are provisional. That’s the nature of science. Anything can be corrected. There’s no absolute truth there. Each model, each finding is just adding to a weight of evidence in one direction or another.
posted by katra at 6:23 PM on April 28, 2020 [12 favorites]
Even when scientists are very well-intentioned and not trying to support any side of the narrative, when they do work and release a paper it gets picked up by actors with political agendas.
I've really come to realize these past few weeks how even if you think you're just collecting and reporting raw data, there's an enormous number of gaps and judgement calls that can quickly take on a political dimension. Think basic facts everyone should be able to know: how many people got tested for the virus today; how many of those tests were positive? That doesn't sound subjective, let alone political.
But then you dig into it. Some jurisdictions report the number of specimens tested, not the number of people tested (people can be tested multiple times). Figuring out the number of people tested requires a time-consuming case matching process, and not everyone does that or does it quickly. But then what happens if a state backtracks and goes from reporting people tested to specimens tested? It suddenly looks like their testing capacity increased. Even a simple question suddenly becomes a matter of judgement, and that decision has political implications.
Georgia's new data dashboard is a particularly interesting example (that link won't display any data unless you disable your ad blocker; I never got all the way to the bottom of the rabbit hole of why embeds from sas.com are on blocklists). Scroll down to COVID-19 Cases Over Time, which is the sort of chart you'd want to look at to make decisions about reopening businesses, which is either a health decision or a political decision depending on your agenda. It looks like a beautiful downward sloping curve perfect for politicians to declare a grand reopening. But then you look at the fine print: "* 14-day window – Confirmed cases over the last 14 days may not be accounted for due to illnesses yet to be reported or test results may still be pending." And significantly, they also say that they backdate cases to the date of symptom onset where that's known.
Is that an invalid thing to do? Nope; looking at the date of symptom onset is a potentially useful way of organizing the data to help understand what's happening. But it's also not what other states are doing, yet the chart invites comparisons to make Georgia look good. The most recent entries at the right of the chart will always be low because they're purposely going back in time to attribute cases to previous days. If you ignore the most recent 14 days, the part of the data they say isn't valid yet, cases are at their peak, but if you just look at the chart, you could easily think the problem is solved and we can all go back to normal. Which is how we get two views of the same state: one where a model projects that they'll max out on ICU capacity and one where it looks like there were just a handful of new cases today and we're well past the peak. I'll at least hand it to whoever built this dashboard for trying to add the caveats necessary to explain the reporting situation to the graph. But it's a prime example of how what seems like a really objective health question—make a chart showing how many people are testing positive for the virus—suddenly becomes a political question depending on what story you want to tell.
Similarly, should epidemic curves and states be reported per capita? It's complicated. You can be sincerely well-intentioned and just want to make a useful chart, and there are valid reasons to do it either way, yet that question has significant political implications and can easily be used to mislead.
It's not enough to just put out accurate data right now. It takes a bunch of extra effort to go out of your way to try to ensure that data isn't likely to be easily misleading because someone, quite possibly someone with an agenda, will be tempted to use it to compare things that aren't comparable or extrapolate something that isn't extrapolatable.
posted by zachlipton at 7:49 PM on April 28, 2020 [19 favorites]
I've really come to realize these past few weeks how even if you think you're just collecting and reporting raw data, there's an enormous number of gaps and judgement calls that can quickly take on a political dimension. Think basic facts everyone should be able to know: how many people got tested for the virus today; how many of those tests were positive? That doesn't sound subjective, let alone political.
But then you dig into it. Some jurisdictions report the number of specimens tested, not the number of people tested (people can be tested multiple times). Figuring out the number of people tested requires a time-consuming case matching process, and not everyone does that or does it quickly. But then what happens if a state backtracks and goes from reporting people tested to specimens tested? It suddenly looks like their testing capacity increased. Even a simple question suddenly becomes a matter of judgement, and that decision has political implications.
Georgia's new data dashboard is a particularly interesting example (that link won't display any data unless you disable your ad blocker; I never got all the way to the bottom of the rabbit hole of why embeds from sas.com are on blocklists). Scroll down to COVID-19 Cases Over Time, which is the sort of chart you'd want to look at to make decisions about reopening businesses, which is either a health decision or a political decision depending on your agenda. It looks like a beautiful downward sloping curve perfect for politicians to declare a grand reopening. But then you look at the fine print: "* 14-day window – Confirmed cases over the last 14 days may not be accounted for due to illnesses yet to be reported or test results may still be pending." And significantly, they also say that they backdate cases to the date of symptom onset where that's known.
Is that an invalid thing to do? Nope; looking at the date of symptom onset is a potentially useful way of organizing the data to help understand what's happening. But it's also not what other states are doing, yet the chart invites comparisons to make Georgia look good. The most recent entries at the right of the chart will always be low because they're purposely going back in time to attribute cases to previous days. If you ignore the most recent 14 days, the part of the data they say isn't valid yet, cases are at their peak, but if you just look at the chart, you could easily think the problem is solved and we can all go back to normal. Which is how we get two views of the same state: one where a model projects that they'll max out on ICU capacity and one where it looks like there were just a handful of new cases today and we're well past the peak. I'll at least hand it to whoever built this dashboard for trying to add the caveats necessary to explain the reporting situation to the graph. But it's a prime example of how what seems like a really objective health question—make a chart showing how many people are testing positive for the virus—suddenly becomes a political question depending on what story you want to tell.
Similarly, should epidemic curves and states be reported per capita? It's complicated. You can be sincerely well-intentioned and just want to make a useful chart, and there are valid reasons to do it either way, yet that question has significant political implications and can easily be used to mislead.
It's not enough to just put out accurate data right now. It takes a bunch of extra effort to go out of your way to try to ensure that data isn't likely to be easily misleading because someone, quite possibly someone with an agenda, will be tempted to use it to compare things that aren't comparable or extrapolate something that isn't extrapolatable.
posted by zachlipton at 7:49 PM on April 28, 2020 [19 favorites]
Folks that I check on Twitter (adding to latkes list!)
Dr. Tara C. Smith @aetiology infectious disease epidemiologist
Dr. Maia Majumder @maiamajumder Computational health and epidemiology
Dr. Ian Mackay @MackayIM virologist
Dr. Saskia Popescu @SaskiaPopescu Biodefense, infectious disease epidemiologist
Dr. Angela Rasmussen @angie_rasmussen virologist
Dr. Emma Hodcroft @firefoxx66 Phylogenetics (virus genome)
posted by spamandkimchi at 8:27 PM on April 28, 2020 [6 favorites]
Dr. Tara C. Smith @aetiology infectious disease epidemiologist
Dr. Maia Majumder @maiamajumder Computational health and epidemiology
Dr. Ian Mackay @MackayIM virologist
Dr. Saskia Popescu @SaskiaPopescu Biodefense, infectious disease epidemiologist
Dr. Angela Rasmussen @angie_rasmussen virologist
Dr. Emma Hodcroft @firefoxx66 Phylogenetics (virus genome)
posted by spamandkimchi at 8:27 PM on April 28, 2020 [6 favorites]
daniel griffin addresses evidence-based, eminence-based (and otherwise based) medicine, and a perilous emergence of "careful observation and experience based medicine" in the context of doctors' anxiousness to do anything instead of nothing for covid patients during the opening minutes of latest this week in virology, episode 606.
posted by 20 year lurk at 8:40 PM on April 28, 2020 [2 favorites]
posted by 20 year lurk at 8:40 PM on April 28, 2020 [2 favorites]
I could write a long screed about how no one knows anything--in fact I did and then deleted it--but there's not much point. Cutting edge science is hard enough in normal times with the peer review and years to have grad students do extra experiments to please some reviewer with a bone to pick. And now? All the experts are forced to try and make deductions based on rushed preprints. It doesn't matter how well we vet our lists, any statement that is not primarily bullshit detection is more likely than not to turn out to be wrong.
The wealth of twitter feeds, the fake specificity that come from looking at maps of confirmed cases, or trying to guess about testing rates and serology data, are all fine if you want a front seat on science as it is done live at a sub-peer review level. Just remember that's what it is. It's not quite the same as being "informed." Which is, as stated, essentially impossible since we lack the information.
posted by mark k at 9:14 PM on April 28, 2020 [12 favorites]
The wealth of twitter feeds, the fake specificity that come from looking at maps of confirmed cases, or trying to guess about testing rates and serology data, are all fine if you want a front seat on science as it is done live at a sub-peer review level. Just remember that's what it is. It's not quite the same as being "informed." Which is, as stated, essentially impossible since we lack the information.
posted by mark k at 9:14 PM on April 28, 2020 [12 favorites]
It's not enough to just put out accurate data right now. It takes a bunch of extra effort to go out of your way to try to ensure that data isn't likely to be easily misleading because someone, quite possibly someone with an agenda, will be tempted to use it to compare things that aren't comparable or extrapolate something that isn't extrapolatable.
Is this even possible? Even here, every bit of info has the tendency to bring out the armchair epidemiologist in all of us, compounded with group dynamics to express passionately how utterly dire the situation is, it becomes impossible to escape short term bad interpretations and extrapolations even with the best of intentions.
Just reading "A toxic legacy of poor-quality research, media hype, lax regulatory oversight, and vicious partisanship has come home to roost in the search for effective treatments for COVID-19" seems pretty dramatic considering the world has only known of covid for a few months. Even without such "toxic legacy", how much could possibly be known in such short time? Almost nothing can be said about it right now with certainty. The problems in research today existed a year ago, and the year before. Yet does anyone deny that the bulk of knowledge inevitably marches on, in starts and fits and stumbles, pretty much as it usually does? Despite all the noise, I have little doubt that a covid vaccine won't arrive as soon as it possibly can.
posted by 2N2222 at 12:47 AM on April 29, 2020 [4 favorites]
What’s happened with this pandemic that we’re not accustomed to in the epidemiology community is that it’s been really heavily politicized.
Taken by itself, that's kind of a stunning remark, given the history of how HIV has been handled?
Possibly the professor being interviewed means that people in power are demanding action to fit their agendas today, rather than actively ignoring the situation as was the case in the 80s (but that would be me putting words in their mouth).
posted by gimonca at 4:01 AM on April 29, 2020 [5 favorites]
Taken by itself, that's kind of a stunning remark, given the history of how HIV has been handled?
Possibly the professor being interviewed means that people in power are demanding action to fit their agendas today, rather than actively ignoring the situation as was the case in the 80s (but that would be me putting words in their mouth).
posted by gimonca at 4:01 AM on April 29, 2020 [5 favorites]
Local media here did a decent job in this article: Why Minnesota's COVID-19 Models Are So Different, at least of illustrating some of the issues--and the dramatic differences in the models themselves.
posted by gimonca at 4:10 AM on April 29, 2020 [3 favorites]
posted by gimonca at 4:10 AM on April 29, 2020 [3 favorites]
I was hoping there would be some discussion of one of my favorite hate issues: that when politicians try to to direct research, by throwing money at a field, they create perverse incentives and bad research. I was pretty shocked when one of the first days of response here in Denmark, they put the equivalent of all the year's public medical research budget into corona-virus research. We don't even have enough qualified people to use all of those money. Which means someone will get grants in spite of not being highly qualified. (I don't know if what they did was redirect existing money, if so the effect will be even worse).
I'm about to review a big bunch of applications for a private foundation, and it'll be interesting to see how many hastily made up corona applications are in there. I'm not the medical expert, and I can't judge what is good or bad, but I trust my colleague, a leading anesthesiologist. We don't have a lot of money, but often young researchers use a grant from us as seed money for bigger applications, and we see the trends quite clearly.
posted by mumimor at 5:28 AM on April 29, 2020 [8 favorites]
I'm about to review a big bunch of applications for a private foundation, and it'll be interesting to see how many hastily made up corona applications are in there. I'm not the medical expert, and I can't judge what is good or bad, but I trust my colleague, a leading anesthesiologist. We don't have a lot of money, but often young researchers use a grant from us as seed money for bigger applications, and we see the trends quite clearly.
posted by mumimor at 5:28 AM on April 29, 2020 [8 favorites]
I went to a talk by Henry Brem, one of the people who developed the Gliadel wafer which is used for treatment of some brain tumors (GBM); I think 'success' is defined as 'person lives longer than six months'. He told the story of how at one talk, somebody told him that people were dying! It was unethical to run a control group when you thought you could help somebody. And apparently somebody else stood up and told that guy "That's why nobody knows if your treatments work or not."
posted by Comrade_robot at 7:12 AM on April 29, 2020 [5 favorites]
posted by Comrade_robot at 7:12 AM on April 29, 2020 [5 favorites]
Science Has an Ugly, Complicated Dark Side. And the Coronavirus Is Bringing It Out. • Jackie Flynn Mogensen, Mother Jones, 4/28/2020 • "Experts say the pandemic is letting bad science slip through the cracks."
Lengthy overview of challenges posed to science by the current pandemic.
Lengthy overview of challenges posed to science by the current pandemic.
“In a normal news cycle, science gets like less than 1 percent of the coverage,” [Maciej Boni, an associate biology professor specializing in mathematical modeling and epidemiology at Penn State University] says. “But in today’s news cycle, the only thing in the news is science. So everybody, every economist, every physicist, every tech guru, everybody wants a piece of the attention if they think they can link themselves or link their own pastime, their background, to something that’ll be covered.”posted by ZeusHumms at 12:32 PM on April 29, 2020 [3 favorites]
I used to be a life coach on YouTube, but now I'm a covid expert for clicks!
posted by benzenedream at 12:57 PM on April 29, 2020 [5 favorites]
posted by benzenedream at 12:57 PM on April 29, 2020 [5 favorites]
Here's another sharp, link-filled look at the uncertainties surrounding the pandemic: Why the Coronavirus Is So Confusing, looking at the flawed way some scientists and journalists have been presenting information about it. Lots of smart folks quoted. From The Atlantic.
posted by mediareport at 2:12 PM on April 29, 2020 [2 favorites]
posted by mediareport at 2:12 PM on April 29, 2020 [2 favorites]
Does anyone know anything about this site? It sounds like poppycock wrapped in pseudoscience & laced with swift rosy outcomes, but who can say?
posted by chavenet at 3:32 PM on April 29, 2020
posted by chavenet at 3:32 PM on April 29, 2020
Does anyone know anything about this site? It sounds like poppycock wrapped in pseudoscience & laced with swift rosy outcomes, but who can say?
Based on the description of their methodology they posted, it looks like the projections assume mitigation measures (e.g. school closures, gathering size limits, general social distancing) remain in place until the "end" of the pandemic in a country. Because countries are relaxing and tightening those measures all the time, I don't think you can put much stock in the predicted endpoint of the model on any particular day. The most you can conclude is the likely date when new cases would drop below a certain level if mitigation measures remained the same.
A link to the write-up appears below.
Writeup
posted by eagles123 at 5:00 PM on April 29, 2020 [1 favorite]
"Experts say the pandemic is letting bad science slip through the cracks."
Keep your eye on the so called biopsychosocial movement in medicine (really just old fashioned behaviourists & moralists in an unconvincing disguise), in particular their claims about the sequelae of COVID.
They are already moving to claim this turf like zombie vultures on pharmaceutical grade crank.
It ain't gonna be pretty.
posted by Pouteria at 6:23 PM on April 29, 2020 [2 favorites]
Keep your eye on the so called biopsychosocial movement in medicine (really just old fashioned behaviourists & moralists in an unconvincing disguise), in particular their claims about the sequelae of COVID.
They are already moving to claim this turf like zombie vultures on pharmaceutical grade crank.
It ain't gonna be pretty.
posted by Pouteria at 6:23 PM on April 29, 2020 [2 favorites]
I went ahead and made a twitter list of all the users listed in this thread by latkes and spamandkimchi for convenience.
Thanks for pulling those together. Hope we can add more.
posted by JoeXIII007 at 7:07 PM on April 29, 2020 [3 favorites]
Thanks for pulling those together. Hope we can add more.
posted by JoeXIII007 at 7:07 PM on April 29, 2020 [3 favorites]
@chavenet: The site is nonsense, or perhaps more generously not distinguishable from speculation. It's an exercise in curve fitting. Sampling data that is trending solely upwards and extrapolating a peak, as they are willing to do for various countries, is both easy to do and generally unsound. An R^2 value on this sort of curve shape is essentially meaningless. And discarding bad fits as they do in a case like this isn't (as they imply) being cautious, it's ignoring data that shows your model is not generally applicable. You could add a reasonable assumption or change a parameter and end up with equally good fits that drew completely different conclusions.
Skimming the write up I don't think it's motivated by anything except the ability to do some basic stats operations and set up a web site; i.e., not an attempt to deceive or mislead or even grab a minor bit of fame.
posted by mark k at 10:22 PM on April 29, 2020 [1 favorite]
Skimming the write up I don't think it's motivated by anything except the ability to do some basic stats operations and set up a web site; i.e., not an attempt to deceive or mislead or even grab a minor bit of fame.
posted by mark k at 10:22 PM on April 29, 2020 [1 favorite]
Here's a good site with clear explanations of developing stories by experts: Roundups and Rapid Reactions by the Science Media Centre. It doesn't have every development and it's pretty UK-centric, but it really cuts through the B.S. about COVID-19.
posted by Harvey Kilobit at 9:29 PM on May 1, 2020 [4 favorites]
posted by Harvey Kilobit at 9:29 PM on May 1, 2020 [4 favorites]
Here are summaries of the "pick of the coronavirus papers", aka Nature wades through the literature on COVID-19 so you don’t have to
posted by Harvey Kilobit at 12:18 PM on May 3, 2020 [6 favorites]
posted by Harvey Kilobit at 12:18 PM on May 3, 2020 [6 favorites]
« Older “When these six destinies entwine, their world... | Helping your children process COVID-19 emotions. Newer »
This thread has been archived and is closed to new comments
Physicians
Vinay Prasad
Seth Trueger
Darrel Francis
Kim Narain
Muge Cevik
Adam C Lake
uché blackstock
Epidemiologists & related specialists
Natalie E. Dean
Bethany Hedt-Gauthier
Kosali Simon
Carl T. Bergstrom
Usama Bilal
Marc Lipsitch
Trevor Bedford
posted by latkes at 1:57 PM on April 28, 2020 [40 favorites]