What Actually Helps Poor Students? Human Beings

May 16, 2017 9:16 AM

Fredrik deBoer at the ANOVA explains a new meta-analysis on Academic Interventions for Elementary and Middle School Students With Low Socioeconomic Status.
posted by a snickering nuthatch (32 comments total)

So, Freddie realizes what critics of the education "reform" movement have been saying for years. Color me shocked.
posted by NoxAeternum at 9:21 AM on May 16, 2017 [5 favorites]

I think everyone is aware smaller class sizes help, they just don't have the teachers or funding to make it so.
posted by corb at 9:24 AM on May 16, 2017 [2 favorites]

So, Freddie realizes what critics of the education "reform" movement have been saying for years. Color me shocked.

While I understand the annoyance of studies and reports repeating what is considered common knowledge among certain groups, I also see the value of a formal study verifying what was previously anecdotal, even if based on hundreds if not thousands of anecdotes, because the formality gives weight to the answer, which can then be used to leverage for more funding and support.
posted by filthy light thief at 9:54 AM on May 16, 2017 [16 favorites]

I think that the article is saying more than "more teachers, smaller classes", despite what the summary from deBoer is glibly claiming. Here's a few lines from their conclusions section,

The examination of intervention components showed that tutoring, feedback
and progress monitoring, and cooperative learning have comparatively large and
robust average effect sizes.
...
Studies of feedback and progress monitoring in our material mainly give teachers
more knowledge about student progression to allow teachers to adjust material
and instruction appropriately. It should be emphasized that there are only five
studies using this component, and in all cases this component is combined with
another component. [With further explanation of recent studies past the cut-off date that
show improved data analysis tools in the hands of admin. and teachers can
assist with this].
...
Incentive programs, after-school programs,
summer programs, coaching and mentoring of students, psychological/
behavioral interventions, personnel development, and computer-assisted instruction
programs were all positive but not significant. However, there are examples of
effective interventions in many of these categories, and most of these components
have been examined by only a few studies. Therefore, we want to emphasize that
our results do not imply that incentive programs cannot be effective, or that there
are no incentive programs with positive effects (there are such examples)
...
[From the 'implications' sub-section] Our results indicate that it is possible for schools and local stakeholders
to substantially improve the educational achievement of low SES students. As such,
they provide motivation for action. We hope that the review may provide inspiration
for educational decision makers at all levels who are looking for ways to
improve the educational achievement of low SES students. When it comes to the
specific interventions a school or a teacher should choose, our results provide
less guidance. Even in our large sample of studies, it was not possible to fully
explain why some interventions worked better than others. We also believe that
the impact of an intervention depends on the local context

So... yeah, this is massively different than the ultimate conclusion of deBoer, "Yet how often do you hear about tutoring from ed reformers? How often does it pop up at The Atlantic or The New Republic? Compare that to computer-mediated instruction, which is a topic of absolute obsession in our ed debate, the digital Godot we’re all waiting for to swoop in and save our students. No matter how often we get the same result, technology retains its undeserved reputation as the key to fixing our system."

This is wrong on a few levels: first, a lot of the high effect interventions are technologically mediated (although it's technology as an assistive tool instead of being the sole basis of the intervention); secondly, it's misrepresenting the extremely carefully measured conclusions of the studies which indicate that in specific contexts the more technologically centered intervention can be effective; and third, it's a misrepresentation of most ed-tech researchers that I've read, most of whom would find 'you need people in addition to technology' to be a non-controversial (and really, obvious to the point of banality) claim.

This is a valuable study, but it deserves better than a simplistic "more teachers, smaller classes, no technology" gloss.
posted by codacorolla at 10:06 AM on May 16, 2017 [16 favorites]

Was hoping for more analysis of variance
posted by clockzero at 10:36 AM on May 16, 2017 [5 favorites]

'you need people in addition to technology'

I used to work in educational research involving technology and yes, this is completely non-controversial in academic ed tech research. I'm now in a position of supporting already-implemented ed tech so get solicited a lot with ed tech start ups and they are insufferable with the pie in the sky nonsense about technology solving everyone's problems everywhere. But that's kind of their job. They are selling something. They and their claims should all be studiously ignored.
posted by soren_lorensen at 11:33 AM on May 16, 2017 [3 favorites]

Number one: the "why didn't you write hundreds more words to conform to my whims as a reader" attitude is only ever found among those who have never published anything and have no idea about what constraints any meaningful definition of concision. I was already above 2000 words. There are limits to what people will read.

Number two: I am drawing my conclusions about the (utter lack of) value of educational technology not just from this study but from a vast swath of studies that show no meaningful learning gains. I was not drawing the conclusion that this study functions in and of itself as a refutation of ed tech but using the most impressive finding (itself quite small in context) about tutoring and how little you hear about it from ed "reformers." The obvious reason for that is that there is no currently-existing well-moneyed firms that exist to extract precious public resources through tutoring, but I'm sure they're working on that. If you had bothered to follow a link to another study I looked at, where I refer to a broad base of research that you are fully welcome to investigate yourself, you wouldn't make the mistake of thinking that my conclusions were drawn only from this study. If you're going to engage, read carefully, and please, do more to fix your own ignorance of the policy and research context.

Number three: When you say "everyone knows" that we need small class sizes, or that it's obvious that we need people in addition to technology, you're simply talking out of ignorance, again. Technology is represented as a REPLACEMENT for personnel in ed policy discussion constantly. Things like Khan Academy have been represented over and over again as financially valuable precisely because they supposedly can cut down on labor costs. You haven't heard those arguments? I'm sorry, that's not my job to fix. You are not the cosmos; your solipsism isn't my problem. Today, right now, in places that matter and where power is accrued, people are making precisely the arguments that you say aren't made. I engage with them all the time. Your misconception is your problem and not mine. To say nothing of the fact that we're already, say, spending $1 billion to bring useless iPads into Los Angeles classrooms and not spending anything like similar money there to hire tutors. Again: learn first, then speak.

Your gloss on my "gloss" is inaccurate, and you're demonstrating profound ignorance about the policy context and conversations in which this discussion is happening. Do your homework.
posted by Freddie deBoer at 11:36 AM on May 16, 2017 [21 favorites]

Are you writing an article about education politics, or an article about education research? OK, I know the answer-- an article painstakingly describing elementary statistical methods in order to then jump to a political conclusion (that I happen to agree with) in the last paragraph.

This article was not concise in any meaningful way, unless the goal was to explain how to carefully understand evidentiary arguments and then ignore them.

I am a teaching assistant and hugely invested in the value of one-on-one tutoring and teaching as a social process (and the uselessness of extraneous technology, even when teaching technology, which is what I do), but I think this is a substandard statistical analysis with no particular audience. It's obvious and elementary to people in statistics or education research, and uselessly misleading and tangential to a political audience that wants to understand the conclusions of education research more deeply. So you could always consider writing a better article. Or a more persuasive piece that demonstrates your legendary mastery of the field of education research.
posted by stoneandstar at 12:19 PM on May 16, 2017 [12 favorites]

Hey, Freddie, welcome to Metafilter, hope you stick around.
posted by AceRock at 12:34 PM on May 16, 2017 [2 favorites]

Yeah, I have several friends who are getting out of teaching right about now, aland they all say the same thing: their districts are spending money on everything but personnel. One said to me recently, and I agree, that even an older textbook in the hands of a smart, experienced teacher who's up-to-date in their field and has adequate support staff is preferable to brand new textbooks in the hands of teaching and support staff who are inadequate in skill, training, experience, and number. If you can only afford a computer or a teacher, your money is better spent on the teacher.
posted by The Underpants Monster at 12:50 PM on May 16, 2017 [5 favorites]

Genes accounted for nearly 50% of the variation in mental ability of high SES children but only a negligible share of low SES children’s variation, indicating that the latter are not reaching their full cognitive potential.

I sure wish the authors had cited this. It's quite a claim. Perhaps it's meant to be attributed to Andreassen & Fletcher, 2007? The word "gene" doesn't appear in their paper, and it doesn't look relevant. The following in-line citation, Rhemtulla and Tucker-Drob, 2012, is closer to the topic, but a cursory reading makes that statistic hard to justify. The highest value for their fitted "genes" variance on the math test, which showed the largest result, just barely reaches 50% for the very highest SES cases on the farthest edge of the plot. Their reading test instead shows the opposite slope, and they give us no reason to believe that's any less valid a metric. Combining the two and claiming you're measuring something seems. . . naive at best.

Which isn't to say that metastudies aren't useful, or that this one is wrong. Just that their introduction invites skepticism.
posted by eotvos at 12:52 PM on May 16, 2017 [2 favorites]

Hey Freddie. I read a good chunk of what you blog and often find it interesting and worth the time, although I have noticed that often I would rather not follow the threads that lead to more internet fighting, of which I consume entirely too much already. I think this thread will probably go better if it goes less fighty.

(Please? Everybody?)
posted by brennen at 12:52 PM on May 16, 2017 [8 favorites]

Freddie: You make good points in your comment, but we generally try to be less snarky with each other here while still having a vigorous discussion. Yes, I know... we often don't succeed. But we try. :-)

That said, I found your blog post interesting and informative, and it's good to have you in the discussion with us here. I'm curious about the wide spread in small-group instruction results. I'm used to seeing that large of a spread when n is small, but you seem to be suggesting that it's instead because the underlying studies were all over the place. Since "small class sizes" is always a hot topic in education discussions, I was wondering if you could talk more about what's going on with those results. Are the underlying studies large enough that we should be seeing more consistent results? Is there any question that there may have been agenda-driven results in the underlying studies, i.e. funders out to prove that small class sizes do/don't make a difference?
posted by clawsoon at 1:06 PM on May 16, 2017 [4 favorites]

Given that NoxAeternum opened the comments with a crappy snark dump, it seems a bit unfair to single out FdB for being snarky in his response above.
posted by pharm at 1:15 PM on May 16, 2017 [7 favorites]

I bet if the OP had said "Mefi's own", we wouldn't have started out so snarky. :->
posted by clawsoon at 1:20 PM on May 16, 2017 [3 favorites]

If you know Freddie from previous ventures and other venues you know he does have a bit of a reputation for getting in fights - but that first comment seems to project on him some kind of previous views that I'm not sure he ever espoused so that's not really a great foot to start off on. I'm finding the topic of the blog and the direction he's approaching it from interesting so I hope we can have a good-faith discussion all sides.
posted by atoxyl at 1:53 PM on May 16, 2017 [1 favorite]

My experience in education - my whole career, first in the Educational Talent Search arm of the Trio programs, now at community college - offers lots of anecdotes congruent with the metastudy's conclusions. Students from low-income, first-generation backgrounds thrive best when supported by rich, repeated contact with well-trained professionals. Seems obvious, but still needs better support than my stories, so thank you.

About investing in Chromebooks, etc: the students I work with use them for all kinds of important things, like:
+ searching and applying for jobs
+searching and applying for scholarships
+ researching colleges, majors, outreach and summer programs
+ submitting college applications

And that's just the stuff I document for my school's outreach activities. Many poor kids have a phone at least marginally capable of doing those things (but many lack data plans); they prefer to use their Chromebooks every time. Especially since their phones are often cracked.

It's probably analogous to the benefits of preschool/daycare for younger children - the developmental advantages may diminish as the child ages, but the benefit to the family is real and significant: it frees the parent to do things like work a job. The school-supplied Chromebook frees money to do other things, and the tech support available through the school means the machines actually work.
posted by Caxton1476 at 2:01 PM on May 16, 2017 [11 favorites]

The problem there is that the education "reform" movement doesn't operate in good faith, given all the horror stories we keep seeing come out of it (hence why reform is in quotation marks above.) From what I've seen, the movement is a joint effort between the usual "privatize everything" grifters and the Silicon Valley set to achieve two goals:

One, to turn public education funding into the next public kitty open to looting by the public sector; and

Two, to continue to prop up the "STEM crisis" myth that seems to be at the heart of a lot of bad hiring policies in the Valley.

The problem is that this movement has gained a lot of undeserved credibility, most likely due to the money that it tossed around, and a few "success stories" that have been since called into question. There's finally been some pushback, but the problem is that you have a bad faith actor that nevertheless gets invited to make serious policy proposals.
posted by NoxAeternum at 2:11 PM on May 16, 2017 [4 favorites]

(What I meant about your initial comment is that it read like you were assuming the author was a member of said education "reform" movement, which I don't think he is or was.)
posted by atoxyl at 2:29 PM on May 16, 2017

Nice to meet you Fred. Here're some of my own notes for you: if you're going to present conclusions about an article that you yourself are offering to summarize for your audience, then you should:

• Realize that your work will likely be read in the context of a single link, and that your anonymous Internet audience (who I assume you are writing for given that it's a public facing blog) most likely doesn't care to click through to your other writing,
• and therefore, you should take steps to distinguish between a summary of the work of others, and your own opinions built from that work. In addition, you should probably not assume your audience will be well versed in your other arguments.
• And, therefore, add a ~300 word transition paragraph that essentially states, "The article concludes with these discussion points, I both agree and disagree with this because [whatever] as I've stated in previous work [summarize here]. Therefore, I think [my own interpretations of those findings]."

The final section of your post contains no such transition.

Nowhere do you mention the ways that the authors contextualize their findings through their discussion section. In fact, the way that you have written your last section makes it seem as though the authors of the original study are making your points (and they are not). This is done in such a way that it makes me suspicious of the post's framing, because you have offered a summary of an article, and then left out elements of that article that disagree with your larger point.

The discussion section of the original article (which I quoted from in my original comment) is important to the conclusions reached by the authors, and despite how you might disagree with that contextualization, if you're presenting a summary of the work of other authors you should take care to separate your interpretations of their work from the interpretations that they have provided.

Here's another point of style for free: jumping into your trackback links and starting a personal fight is not a good look, my man.
posted by codacorolla at 2:43 PM on May 16, 2017 [13 favorites]

Given this author's penchant for sneering at "identity politics" (aka civil rights), it seems fairly gross (if typical) that this summary about issues facing students of low SES doesn't mention any of the most prominent factors that might be linked to the lack of resources for said students, aka systematic racism and the decades long crusade of white supremacists to choke off all educational funding for majority PoC schools.

Seriously, can you imagine the fury of our nation's racists (aka our government) if low SES schools were openly talking about about decreasing class size and increasing access to tutoring? These people don't even want poor students to have access to lead-free water, air conditioning in 100 degree summer weather, or books. The problems facing these students, like most things in the news these days, are overwhelmingly about institutional racism and the attendant income inequality. Pretending that it is about the people who run these schools being fixated on technology is utterly disingenuous.

The authors of the original paper are all from Denmark, so I’m not surprised that they don’t focus on this aspect of the research. But when you look at the studies they are studying, the US reports are overwhelmingly telling the story of systematic racism. For FdB to ignore that aspect while drawing conclusions about the tendencies of the US education system borders on the ridiculous.
posted by a fiendish thingy at 3:29 PM on May 16, 2017 [6 favorites]

Given this author's penchant for sneering at "identity politics" (aka civil rights), it seems fairly gross (if typical) that this summary about issues facing students of low SES doesn't mention any of the most prominent factors that might be linked to the lack of resources for said students, aka systematic racism and the decades long crusade of white supremacists to choke off all educational funding for majority PoC schools.

I don't think the piece was intended to be a "summary about issues facing students of low SES" rather than an inquiry about what aspects of specifically education policy and pedagogy are revealed to be most effective in a meta-analysis of various academic interventions.

Nevertheless, he does find time to observe socioeconomic constraints which create persistent inequalities, acknowledge the racialized effect of those inequalities and note that these may be fixable via direct socioeconomic policy (read: redistribution and hierarchy leveling)

Your suggestion that he is blind to the role of systematic racism in creating the problems in education strikes me as overstated.
posted by layceepee at 4:11 PM on May 16, 2017

What Actually Helps Poor Students? Human Beings

Allow me to go one step further. You know what really helps all students? Educational decision-makers, policy-writers, study creators, study analysts, meta-study analysts and other so-called pundits who have actually worked as classroom teachers.

Teachers find it horrifying and galling as hell that people who have literally never spent ONE MINUTE actually teaching children do all of these studies and then write papers as though they have a single bloody clue as to what schools are all about.
posted by yes I said yes I will Yes at 4:34 PM on May 16, 2017 [5 favorites]

Given this author's penchant for sneering at "identity politics" (aka civil rights), it seems fairly gross (if typical)

I really was not a fan of the "everybody else is Doing Activism Wrong" version of Freddie, but "against civil rights" is a really unfair conflation, and going after him for writing a post about something that isn't racism on a blog where he also has posts that talk about racism is kind of absurd. This post isn't "a summary of issues facing students with low SES," let alone a definitive explanation of unequal educational outcomes. It's a comparison of particular kinds of programs that are floated as responses to unequal educational outcomes. And bullshit reformers pushing flashy technological solutions that happen to enrich their corporate benefactors is a real thing and a real problem - as Nox said closely related to the whole charter school scam. The people who support that stuff deserve a lot more blame for distracting from issues of systemic racism than people who are writing summaries of studies debunking it.

Again I'm familiar with FdB and I know there are things about his personality and past behavior that haven't done him any favors as far as attracting a negative response. But I think a couple people here are going well out of their way to find problems with this specific thing of his that don't have much to do with what the thing is about.
posted by atoxyl at 4:38 PM on May 16, 2017 [4 favorites]

If anyone wants to know how to improve schools, come out for a drink with any group of teachers after school on any Friday. We can tell you in one hour how to fix education.
posted by yes I said yes I will Yes at 4:42 PM on May 16, 2017 [1 favorite]

My take-away from the post is the specific idea that educational interventions exhibit a trade-off that is fundamentally parallel to the lightpost problem that philosophers of science know well. This is a methodology of education studies topic; a meta-level idea.

That's actually kind of interesting, and also is conceptually different than arguing issues about technocratic education, or teacher-student ratio. Those are obviously important but I don't see those as the main point being discussed; they are used/referenced as specific non-meta, actual examples of education methodology.
posted by polymodus at 4:43 PM on May 16, 2017

Things like Khan Academy have been represented over and over again as financially valuable precisely because they supposedly can cut down on labor costs.

Technology is tools, and tools can always be used well or poorly. In my time as a math/science tutor (hello), I saw Khan Academy used to replace not the teacher, but the worksheets. The teacher doesn't have to grade the homework, but can still use it to see where more work is needed. I dare say that does cut down on labor costs... but not on classroom time. The cost savings would be in freeing the teachers to do other things than routine grading.

So in the one area FbD discusses with which I am most familiar, I disagree with his implication.
posted by clauclauclaudia at 5:15 PM on May 16, 2017 [1 favorite]

I bet if the OP had said "Mefi's own", we wouldn't have started out so snarky. :->

Embarrassingly, I didn't think to look. As an aside, I've long thought that we should try to comment constructively enough that the responses look largely the same whether we're talking about someone "in the room" or not.
posted by a snickering nuthatch at 6:24 PM on May 16, 2017 [3 favorites]

I guess it depends your sense of the education policy discourse. In my experience, technology is pushed as a solution to problems far more often than less sexy solutions that involve hiring more people and taking steps to ensure they are well trained. Based on that, I thought it was a good article.
posted by eagles123 at 6:47 PM on May 16, 2017 [2 favorites]

polymodus: My take-away from the post is the specific idea that educational interventions exhibit a trade-off that is fundamentally parallel to the lightpost problem that philosophers of science know well. This is a methodology of education studies topic; a meta-level idea.

I missed that in the post, or I'm misunderstanding you. Could you expand? What unstudied interventions outside the circle of easy-to-search light are you thinking of?
posted by clawsoon at 8:54 PM on May 16, 2017

The unfortunate part of the conclusions is that it basically confirms that there's no saving education from Baumol's Cost Disease, and that the cost of education is probably going to increase steadily and unstoppably over time.

One of the reasons there's been so much interest in throwing technology at the problem of education in the last few decades is that technology at least provides some theoretical relief from the ever-tightening economic vise. It's a way out of what otherwise looks like a very grim situation, so it should be unsurprising that so many people have grasped at it, even when it has continually underdelivered.

If automation and computerization isn't going to actually work—if there's no way to effectively leverage technology (which tends to get cheaper, in real terms, over time)—then the other options are all various flavors of "very hard" or "very bad".

On the "very hard" side, you have the task of convincing voters that they should pay an increasing amount of money towards education, either for themselves (the higher-ed problem, in the US anyway), or for others (the public education problem), and it's not clear how successful that is going to be. It's not like people haven't been making that argument for a while now, and if you step aside from the arguments themselves, I don't think that they are suddenly going to get more effective at convincing voters tomorrow. But maybe I'm wrong, and there's some magic rhetorical argument out there waiting to be deployed that will convince voters to tax themselves significantly higher so that they can maintain very high staffing levels even in an environment where human labor becomes increasingly priced over time.

On the "very bad" side, we could just accept worse outcomes than in the past, because the inputs have gotten more expensive and the budget allotted to those inputs hasn't changed. This is what has happened, traditionally, to lots of things subject to the same forces—e.g. my house, like most modern houses, has interior walls and carpentry that are much shoddier than a house built in 1920, because I can't afford the skilled labor than a 1920-style house (with plaster-and-lath walls and the rest) would cost. But we've decided as a society that slapping up sheets of taped drywall is just fine, and (perhaps importantly?) even people who can afford to do better, typically don't bother.
posted by Kadin2048 at 9:47 PM on May 16, 2017 [12 favorites]

As a professor at an open access college, the conclusion that human contact is the solution is one that I agree with because I see it in action every day. We have small class sizes, purposeful mentoring of every student by an assigned faculty member, a tutoring center, and a peer supplemental instruction program. I personally have witnessed many many students make it through tough stuff that might have caused them to quit college because one of these interventions made the difference.

As an educator of people of color in the southeastern US, I also have unsurprising opinions about how to properly fund our PK-12 public schools to allow application of this solution. We need real, honest to goodness integration, redrawing individual school assignment lines and, yes, entire school district lines to ensure that everyone goes to school with a broad representation of the people who live in their city/county/metro area. That is the only way to ensure that funding of all sorts, federal, state, local, and donations, is actually equitable. And we need to stop funding schools with local property taxes and instead fund them with (local, state, and/or federal, I'm not picky) income taxes, which need to be higher on the wealthiest among us.

I understand that both of those changes are complete political nonstarters in our racist, classist society where people claim to care about "the children" but really only mean "my child and screw everyone who isn't lucky enough to have a parent like me", but that doesn't mean that it's not worth discussing. We do have a solution, we just don't care enough about other human beings to actually implement it.
posted by hydropsyche at 4:18 AM on May 17, 2017 [10 favorites]