Measurement error and the replication crisis
February 11, 2017 12:17 PM   Subscribe

Why traditional statistics are often “counterproductive to research in the human sciences” A Retraction Watch interview with Andrew Gelman.

Loken and Gelman's related Science article here.
posted by MisantropicPainforest (7 comments total) 39 users marked this as a favorite
Kind of a click-baity way of framing a reasonable point - that you need to be much more statistically and analytically rigourous when dealing with noisy data. It's not so much that "traditional statistics" doesn't work in the human sciences, more that it's just more difficult to do things that are relatively easy to get right (or right enough) in the physical sciences.
posted by howfar at 1:26 PM on February 11, 2017 [9 favorites]

I agree with howfar. There is also not sufficiently rigorous training in social sciences (in many graduate programs) for the use of analytical tools beyond traditional inferential tests (like analysis of variance) and the data one collects when doing research in the social sciences is - I agree - must messier and difficult than the physical sciences. noisy noisy noisy.
posted by bluesky43 at 2:09 PM on February 11, 2017 [2 favorites]

Or social science is more difficult because the "noise" is actually relevant information.

At least, that's how it is in biology. It's only "noise" because the analytical tools taught in statistics classes were developed for bench science, and not harder sciences.
posted by eustatic at 4:33 PM on February 11, 2017 [6 favorites]

Part of the confusion (in research as well as in the press) is what is meant by "traditional statistics." The closer you get to "Statistics" as an academic field, the more you realize that it's a hotbed of intense contention over foundational issues.

What I think the article is really getting at is "introductory statistics" while trying not to sound patronizing. Or, to be more precise, the real problem are the simplifying assumptions that are made at introductory levels, which often don't map onto how the social sciences (A) measure variables and/or (B) traditionally undertake the research process. The assumption, for example, that observations are "independent and identically distributed" ("iid") is generally something that is baked in to all the formal procedures that students learn in their first statistics course or two. As you relax the iid assumption, the ground under your feet becomes treacherous and a whole new landscape of complex procedures reveals itself.

Even people who are themselves quite statistically sophisticated can read the signs wrong. Take, for example, the dispute between FivethirtyEight and the Princeton Election Consortium over forecasting the 2016 election. Everyone favored Clinton, but FiveThirtyEight gave Trump a much larger probability of winning than the PEC (which rated his chances as being less than 1%). Both were running high-powered Bayesian models, the likes of which are beyond the reach of many social scientists. So why the big discrepancy? It really boiled down to how much to relax iid. As the PEC put it:

The failure was in the general election – and even there, polls told us clearly about just how close the race was. The mistake was mine, in July: when I set up the model, my estimate of the home-stretch correlated error (also known as the systematic uncertainty) was too low. To be honest, it seemed like a minor parameter at the time. But in the final weeks, this parameter became important.

In order words, PEC assumed that the polls in the various states were more independent that FiveThirtyEight was willing to assume, and the consequence was that their forecast completely missed the mark.

Social scientists routinely make the same sort of mistake. They assume that their observations are somehow independent while ignoring major sources of interdependence (e.g. "all these scores were collected in the same lab setting"), and the consequence of ignoring these details renders the statistical exercise of comparing outcome to expectation badly biased. There are many other mistakes that are commonly made, but this is a serious one that can undermine whole research programs.

"Rigor" is an easy word to throw around, but sophistication is not enough. What is needed is an appreciation of the right kinds of rigorous statistical practice, and that sort of recognition has not traditionally been taught in social science curriculums.
posted by belarius at 4:35 PM on February 11, 2017 [30 favorites]

To be honest, the kind of statistical sophistication that one needs in order to do social science well is incredibly demanding, and I wonder whether it's even realistic to expect it of most social scientists without either seriously restructuring social science curriculums or making collaborations with statisticians the normal rule.

For example, I took a year-long, intensive statistics course as a PhD student. It was more demanding than any other graduate course I took - despite having a mathematics degree (including some statistics and probability) going in. At the end of the course, I felt like I knew just enough to know that I don't know anything. The first time I had to look at some complicated data, I made my best guess as to how to approach it - and was totally wrong.

(Luckily, I had no confidence, and actually did consult with a statistician first.)

But it's not possible to devote a significant amount of time to advanced statistical training in my program, or in many others. Tight funds mean they want you out in five years, but the actual work expected of you means that the average is more like six. Get a research position later and you'll still have to struggle with finding the time for continuing your statistical training.

If you add more statistical training you'd have to cut training in other, expected areas of expertise, or extend the years of the program - and the latter is not happening any time soon in this environment.
posted by Kutsuwamushi at 6:32 PM on February 11, 2017 [8 favorites]

I'm currently in a PhD program in applied statistics within the social sciences (i.e. we are the number/stats people who teach everything statistical that social science people take at my institution) and I'm pretty frustrated with the level of instruction I have gotten in terms of actually dealing with real data and research.

I haven't had one single course that hasn't started with "this is all about what to do with cleaned-up data you already have; getting that data is outside the reach of this course". We have the opportunity to work as research assistants in a consulting office for professors who need outside assistance with quantitative work; great experience on one level but we recieve basically NO guidance on our actual work. I do not actually know if any of the output/reports I produced in that job actually used the right approach or were written up correctly. I hope they were and overall I think they were but I have no idea because again, no guidance.

It's scary to be 1-2 years from graduation and realize that I'm at a level where I'm going into this field as one of the supposedly more knowledgeable people... and I know basically nothing. And not in an impostor syndrome sense, I know those feels! In a real, holy shit basically no one knows what they're doing and when is this whole thing gonna come crashing down sense.
posted by augustimagination at 8:13 PM on February 11, 2017 [12 favorites]

augustimagination Quoted For Truth: 'In a real, holy shit basically no one knows what they're doing and when is this whole thing gonna come crashing down sense.'

I only took a year of biostatistics at the undergraduate level-- and my performance was not exceptional. I now have to interface daily with heavy, heavy stats people, and what's really, really scary is how often the phrase 'listen, we really don't know what's going on here' or 'whooopsie' comes up.

Data is hard, statistics is hard, talking to real statisticians and you'll hear that constantly.

I really admire the social sciences, but just a cursory reading of papers [coming from a more quantitative discipline] and I literally become pale all time: 'Uh, no.' 'No again, and that's wrong.' 'And that's wrong, and that's wrong, and that's wrong.' And this is just from an 'interested layman' perspective.
posted by mrdaneri at 7:32 AM on February 12, 2017 [6 favorites]

« Older Dammit, I gotta pee!   |   Fighting Gerrymandering with Mathematics Newer »

This thread has been archived and is closed to new comments