Inside the multimillion-dollar essay-scoring business
March 7, 2011 8:56 AM Subscribe

With the institution of No Child Left Behind, educational testing in the US boomed. Now, some of the low paid temp workers hired to score these tests are speaking out about the behind the scenes manipulation that goes on to ensure test scores are in line with "customer expectations".
posted by reenum (142 comments total) 37 users marked this as a favorite

Another reason to hate these tests.
posted by The Devil Tesla at 8:57 AM on March 7, 2011 [3 favorites]

Daily Show: Diane Ravitch believes education reform should focus on getting children out of poverty, not finding the bad teachers.
posted by homunculus at 9:01 AM on March 7, 2011 [31 favorites]

It fits with one of the underlying themes of "The Wire," when you start focusing too much on the numbers, the goal just becomes gaming the numbers.
posted by drezdn at 9:01 AM on March 7, 2011 [47 favorites]

The very instant I heard about essay questions on these tests my first question was "How on earth are they going to score these fairly? Or at all?"

I hadn't considered "low paid temp workers". The uniquely American solution to, and cause of, all our problems.
posted by DU at 9:01 AM on March 7, 2011 [20 favorites]

We're creating jobs! Yaaaaaayyyy!
posted by RobotVoodooPower at 9:03 AM on March 7, 2011 [2 favorites]

My surprised face: :|
posted by Mental Wimp at 9:06 AM on March 7, 2011 [11 favorites]

Same here, Mental Wimp.
posted by Xoebe at 9:07 AM on March 7, 2011

Oh, and conservatards? This is what happens when you outsource the task of educating our children to profit makers: costs will be cut until the product is worthless.
posted by Mental Wimp at 9:07 AM on March 7, 2011 [50 favorites]

When I was in high school, our standardized graduation test included an essay about "Peer Pressure." Being the smartass I was (and am), I wrote a lengthy, grammatically correct essay on the pressure to be a good bowler. How all the cool kids bowl and the pressure to get the best bowling shoes...crap like that. On the surface, it was a well-written, entirely correct, wiseass response. What I didn't know is that our essays were being read for content, not just grammar. I had to sit with a counselor and explain that, no, I didn't have thoughts of suicide every time I saw a 7/10 split.

Let this be a lesson, kids. Asshattery will get you nowhere, but a one-way ticket to even more asshattery.
posted by ColdChef at 9:11 AM on March 7, 2011 [31 favorites]

This is astoundingly fucked up and it doesn't surprise me at all.
posted by two or three cars parked under the stars at 9:12 AM on March 7, 2011 [4 favorites]

Because those things seem quite different to me.

But both useless. They are supposed to be teaching until understanding dawns. It isn't about making a more versatile Elizachild.
posted by DU at 9:13 AM on March 7, 2011 [4 favorites]

As a teacher, I will always maintain that standardized testing is a complete and utter waste of time and resources. It makes me indescribably angry that they are becoming so important. I don't have a huge issue with things like the SAT, GMAT, LSAT, etc.--if students want to go on to post-secondary education better they should learn about how the system works and how to maximize their chances. Testing kids in primary school and high school is crap, however. I've got kids sitting in my grade 10 math class who apparently scored a level 3 in the Ontario EQAO grade 9 math test, and they can't add two numbers together. Literally. I also have mathematical geniuses who only managed a level 1. Standardized testing is too often wholly unrelated to a child's actual abilities, and usually only measures how well a child can write a test. Our education dollars would be far FAR better spent on classroom resources, teacher training, and extra-curricular programming.
posted by Go Banana at 9:16 AM on March 7, 2011 [10 favorites]

It isn't about making a more versatile Elizachild.

Why, are you satisfied with the current line of Elizachildren™?
posted by Lord Chancellor at 9:16 AM on March 7, 2011 [2 favorites]

When teachers "teach to the test",

In high school, my US History class teacher was also the basketball coach. He had his own approach to test preparation, I suspect devised specifically for all his athletes to at least have one A per term. On the day before a test, he would go through and write out the exact questions and answers for the test on the board. Nine times out of ten, he would 'forget' to erase the board until halfway through the test the next day.
posted by nomisxid at 9:20 AM on March 7, 2011 [5 favorites]

I used to score essays for a company quite similar to what the article describes, except this was during the pre-NCLB days and thus the corruption was not quite so bald-faced.

Nevertheless, as someone with experience doing actual essay grading, I used to look around and think to myself, "I'm sure glad these particular standardized tests don't significantly affect these kids' futures."

Then NCLB happened and I foresaw this article eventually being written.

(The job sucks, by the way—at least if you have any love for the written word.)
posted by AugieAugustus at 9:22 AM on March 7, 2011 [2 favorites]

I actually support most of the intentions of NCLB. I had a few teachers in high school who were utterly unqualified, and a few others who were fine but were put in impossible situations (math class had 45 students in a room that was packed with 30). Some form of testing could have alerted administrators to the situation.

And I think it's possible for a test to be well designed. I took a bunch of AP tests and I thought they were a great way to show that I knew (eg) European history even though I didn't get in to the one section of that class that was offered that year.

But the tests have to be good tests, which means they should be managed by the federal government and people trained in psychometrics. Not low bidder testing companies selected by each state.

My favorite test question on my state's implementation of NCLB had a picture of a butterfly on graph paper and said "draw a reflection of the butterfly," with absolutely no indication which axis to reflect it around. I still have no idea what the right answer was supposed to be on that one, or what skill it was supposed to test.
posted by miyabo at 9:24 AM on March 7, 2011 [5 favorites]

Oh, and conservatards?

Could you not do this, please?
posted by hermitosis at 9:26 AM on March 7, 2011 [41 favorites]

Public education has become a sort of bizarre combination of spiraling failure and near-religiosity. If you say "Public education is a failure," then people on all sides of the political spectrum agree, but have different explanations of why: students just aren't as smart as they used to be, teachers' unions allow incompetents to teach, funding isn't high enough.

What if it just comes down to the idea that the system of public education as practiced in the U.S. simply is not good? Why can't we model our education systems on more successful systems in Europe and Asia?

"Asian schools don't teach their students to be creative thinkers," people will say. "They just memorize."

Well, which part of the U.S. prison system for children aged 5–18 is encouraging "creative thinking?" At least Chinese, Japanese, Korean, Vietnamese, and Indonesian schools produce students who can add and read and speak foreign languages. You know, the verbs in service of which education is pursued.
posted by sonic meat machine at 9:27 AM on March 7, 2011 [18 favorites]

At this point I just assume there is an active and vested interest in destroying all public education and support systems so they can be turned over to faith-based sources.
posted by The Whelk at 9:29 AM on March 7, 2011 [27 favorites]

The standard tests are not designed to produce good students capable of independent thought. They're designed to produce a minimally capable consumer drone capable of signing on the dotted line for a lifetime of student loans, credit cards and subprime mortgages and not asking too many questions about making a minimal salary with no benefits.
posted by T.D. Strange at 9:29 AM on March 7, 2011 [28 favorites]

I always assumed this meant that the tests can only cover a certain range of topics and so teachers only spend time teaching those topics and not going over other information that may be just as or more worthwhile.

As the spouse of a teacher this is pretty much my understanding of the problem. The focus becomes completely what is on the test, and other valuable things get tossed becuase they are not tested on them or there is no time. Things like music, art, and sports.
posted by Big_B at 9:32 AM on March 7, 2011 [4 favorites]

The most depressing part about this is how utterly predictable it all is; offer money to the school based on the ability to produce high scores, and act shocked when it's discovered that the scores have been kept artificially high.

This shouldn't come as a surprise to anyone.
posted by quin at 9:33 AM on March 7, 2011 [2 favorites]

I have a question about these tests. When teachers "teach to the test", are they teaching the kids to memorize answers to specific questions which will be repeated verbatim in the actual test? Or are they teaching kids questions that are similar to what is in the test, with the intent that kids will be able to adapt their answers to the novel questions? Because those things seem quite different to me.

Having been a child who was "taught to the test" you find it winds up being a little of both. A lot of teachers have to cover a generalized set of topics, with very specific points written in. IMO it allows for no variance on teaching style from the state school board's governed curriculum. There was a good chunk of questions that you could assume would be there, and many teachers simply teach the answer on those. This method continued literally from about the 2nd grade to the MCAT when it came to standardized tests. Note: I was/still am really good at taking standardized tests.

The part that has always been a joke on the standardized tests has been the writing component. I am not a particularly great writer, and my handwriting is notoriously difficult to read*. I've always gotten the equivalent to "slightly above average." Every writing test. The ridiculous part is that the writing component always counts for jack. In the MEAP (Michigan's test) it didn't really matter. In the ACT it didn't exist. In the MCAT it was a letter tagged along with the score. Unless you got the bottoms on the writing, no one gave a crap if you had a 32O or a 32R, it was the 32 that mattered.

* - I had one teacher, in the fourth grade, Ms. Hughes (A teacher who taught to the test, poorly). She did not like me because I, along with five other students, were significantly smarter than our peers, and segregated into gifted programs for half of the day. She actually had the gonads to tell my parents that my handwriting was too poor and I'd have to repeat the fourth grade as a result. I was at that parent-teacher conference, when my dad ever so politely told her to go fuck herself and to stop being a rabid bitch. Thanks, Dad.
posted by Mister Fabulous at 9:35 AM on March 7, 2011 [4 favorites]

I literally had to fight back tears watching that Daily Show interview, because she's so right about everything and it won't matter at all.
posted by absalom at 9:36 AM on March 7, 2011 [9 favorites]

Maybe this is a dumb question, but why aren't the teachers themselves grading these tests, instead of having them sent off to AnonyCorp International? I suppose it's cost & lack of time (teachers already have enough to do as it is. and of course earn way too much, too!), but is it also a requirement of NCLB to have these tests graded elsewhere?
posted by slater at 9:36 AM on March 7, 2011 [1 favorite]

"The legitimacy of testing is being taken for granted," he says. "It's a farce."

Money quote. This is absolutely true IMO, and the more we invest in producing flawed "metrics," the less we invest in actual classroom education. But it's big money, and the politics are challenging. Politician types want an easy executive-level report on eduction outcomes. The reality is the problem is inherently too complex and multi-dimensional for that kind of top-down, private-sector style management and decision support model. But most modern executives--public or private sectors--aren't really deep thinkers and tend to overvalue the superficial appearance of decisiveness because, I guess, they only ever learned about being leaders from watching TV or something, so there you go.

At this point I just assume there is an active and vested interest in destroying all public education and support systems so they can be turned over to faith-based sources.

That's not all that far off from what seems to be the endgame in Florida. There's talk here of replacing some public schools with virtual school programs (online education) and in letting parents opt to get a refund of the tax money that would have been spent on their child's education each year to direct to home schooling or whatever educational option they see fit (in other words, offering mom and dad some cigarette and beer money in return for not educating their kids in public schools).
posted by saulgoodman at 9:38 AM on March 7, 2011 [15 favorites]

sonic meat machine, I know this will send us down another endless Mefi Discusses Education thread, but my understanding is that part of the reason for high scores in many other countries vs. the US is that low-scoring students are not put in special programs that attempt to improve achievement, but simply tracked or weeded out. By the time kids get to college level, they are a much more selective group.

Which is something we find problematic here in the US, where we have committed to giving every child, no matter how disadvantaged, as much education as they can absorb. I would appreciate correction if I am wrong, but I believe most Asian countries and even some European ones do not approach education in this way, with this mission.

So then one question becomes; do we value the scores more, or do we value the mission more?

And before the thread spirals out into endless assertions about US education, I do want to put in that it is far from a monolith and varies wildly by region, and that we should not approach this topic as though each of our personal educational experiences was representative of the whole.

Having said all that, there is plenty of evidence that tying test results to funding creates massive incentive for system-gaming and so this article saddens but doesn't surprise me.
posted by emjaybee at 9:39 AM on March 7, 2011 [5 favorites]

why aren't the teachers themselves grading these tests, instead of having them sent off to AnonyCorp International? I suppose it's cost & lack of time (teachers already have enough to do as it is. and of course earn way too much, too!), but is it also a requirement of NCLB to have these tests graded elsewhere?

What? You want to trust those no-account Teachers with grading their own students' performance? Don't be silly: These tests are used for grading teachers.
posted by saulgoodman at 9:39 AM on March 7, 2011 [3 favorites]

When teachers "teach to the test", are they teaching the kids to memorize answers to specific questions which will be repeated verbatim in the actual test?

"Teaching to the test", as I understand it, is to take away any/all understanding from what the question means, and just tells the kids how to answer these questions.

For instance, I was exposed to this in my high school days when I was in an AP Calculus course. I was "pretty good" at Calculus (which, for an American kid in a high school means that you understand next to nothing, but you're awesome at cranking out ridiculous computations for no reason). Our teacher did a good job of preparing us for the AP exam. Why? Because we did things like this:

Question: Find the maximum and minimum values of the function f(x) = 3 - x^2 on the interval [-3,1].

How to attack: Find f'. Find the values of x that make f'(x) = 0. Plug those numbers into f. Plug in the endpoints of the interval into f. Choose the largest and smallest of these values.

I can't say that I really understood what I was doing. I just knew that was how I was supposed to answer those kinds of questions. The problem with this is while I could answer any Calculus question you reasonably might have thrown at my 17-year-old self, I couldn't explain why any of that crap worked to anyone.

It wasn't until much, much later, when I was in graduate school and had to teach Calculus that I realized what a disservice is being done by "teaching to the test". I had to go back and really understand the mean value theorem, the fundamental theorem of calculus, etc. Once I did, I realized how awesome the subject can really be sometimes, and how cool it is when everything fits together at the end. I go to great lengths myself now to try and avoid "here's how you solve problems like this" type lectures, and hope to ask questions that focus on understanding of topics rather than menial calculations that we should be using computers/calculators for.
posted by King Bee at 9:39 AM on March 7, 2011 [15 favorites]

That's not all that far off from what seems to be the endgame in Florida. There's talk here of replacing some public schools with virtual school programs (online education) and in letting parents opt to get a refund of the tax money that would have been spent on their child's education each year to direct to home schooling or whatever educational option they see fit (in other words, offering mom and dad some cigarette and beer money in return for not educating their kids in public schools).

Thank you I needed an excuse to start drinking at lunch.
posted by The Whelk at 9:39 AM on March 7, 2011 [19 favorites]

is it also a requirement of NCLB to have these tests graded elsewhere?

No profit motive.
posted by T.D. Strange at 9:40 AM on March 7, 2011

There will NEVER be enough good teachers for all the students in school. You know how I know that? Because there NEVER has been. Most people have memories of one or two great teachers out of, how many, 30 or 40 in 12 years of primary education? You're lucky. I never had one (and my parents moved me from public schools to private halfway through... I knew at the age of 12 that "profit based education" was a fraud). What we need is a way to spread the wisdom of the top 1% of teachers to all the students, "class size" be damned. Think that may be something the Internet might help with?
posted by oneswellfoop at 9:41 AM on March 7, 2011 [2 favorites]

Thank you I needed an excuse to start drinking at lunch.

How do you think I keep going?

(Note: I do not actually drink at work. Really.)
posted by saulgoodman at 9:41 AM on March 7, 2011

I wish I wasn't leading review sessions all day, so I have a *lot* more to say than the five minutes before my lunch is over will allow, so I'll bullet some major issues with the high-stakes testing

1. Schedule Disruption: for everyone one day of standardized testing, there are at least 3-4 more days of test prep - mock tests, etc - which essentially shut down the entire school. In my own experience, of 180 days of instruction, 20 or more are going to be testing, test prep, etc. In some districts it is as high as 40. Where I work, 2 of every 5 days in english are spent prepping for the ACT, because it's been decided by the State Legislature that the ACT is one of our benchmarks.

2. The tests are not standardized, but rather the definition of "proficient" and "advanced" are determined by state legislatures, so "proficient" in Tennessee might be considered "non-proficient" in California and "advanced" in Mississippi.

Ah, so much more I could ramble about, but I try not to break down at work and I only have about 120 more seconds to go pee before my next class.
posted by absalom at 9:44 AM on March 7, 2011 [3 favorites]

Think that may be something the Internet might help with?

Maybe. I think the best kinds of teachers are the ones who make the students do all the work, though. It's hard to have a discussion or inquiry-based class when the teacher never sees any of the students in person.
posted by King Bee at 9:44 AM on March 7, 2011

Metafilter: Thank you I needed an excuse to start drinking at lunch.
posted by josher71 at 9:46 AM on March 7, 2011 [1 favorite]

Maybe this is a dumb question, but why aren't the teachers themselves grading these tests, instead of having them sent off to AnonyCorp International?

Same reason you don't write your own annual review. There's well documented evidence that high stakes testing leads to cheating. If you down the path of measuring teachers and schools by test improvements, anything less than full proctoring is failure proof.

I know school administrations pay attention -- when only the gifted kids sick on testing day are pulled in for make up tests, it's pretty clear why. One shitty argument in favor of high stakes for teachers testing is that they've got less influence over the testing process than the administration and apparently legislature.
posted by pwnguin at 9:49 AM on March 7, 2011 [1 favorite]

I used to score tests for ETS, not even one of the outsourced companies that now does a lot of standardized testing. I had good and bad things to say about the experience, but I quit when they actually gave us all something like 20% pay cuts. The thing is, it's all about metrics. All they wanted, when I was scoring the CAHSEE and a few other tests was repeatable results. That different people, using the same rubric, would assign an essay the same score. In fact they often joked that what they'd really like is for us to be replaceable with machines, but you really can't tell anyone with confidence that a robot is scoring their child's essay. So you need people who act like robots.

The thing about the CAHSEE that was so heartbreaking is that a lot of these kids knew they were sort of under the gun, they knew the test was important but they also knew they couldn't probably pass it. We had instructions to mark as "blank" essays that were 100% blank, but a page that had almost nothing on it, maybe a word or two, was judged to be failing. I would see essays that would say simply "I no english" or "help me" These kids failed. And it was pretty clear that these kids had been failed by the system well before they had gotten to the test.

And really, a five paragraph essay using some decent words and segues between paragraphs would pretty much always get you a decent score. So while we'd tell people outwardly that it didn't matter how long your essay was, realistically, you could tell by waving a piece of paper in front of your face roughly how well that essay would do. Okay handwriting, some decent paragraph delineations, at least a page long. You'd do fine. You could see the kids who had taken Princeton Review or Kaplan classes because they would always toss a Thomas Jefferson quote in there, even for an essay about dress codes.

Did I say paper? I meant GIFs. We got the papers in GIFs in some sort of awful web-based scoring software that timed us and allowed for multiple scores on the same papers, etc. I think ti was designed so you couldn't copy papers to your hard drive, but really, anyone who is smart enough to view source could manage this. And at the time, which was maybe six or seven years ago, these people also couldn't use technology. You needed dial-up at the minimum to do this system, but people were always having terrible tech support problems and being threatened with being fired if you couldn't meet basic quota [they wanted us to grade 28 papers an hour if I recall correctly]. We got emailed with a huge email that had 800 email addresses on the "to" line of an email [no lie, I counted]. I signed some scary NDA so I'm being a little vague here, but if there was one thing, one thing in my life that points me towards the absolute bankruptcy of the NCLB prograqm [which they pronounced Nickleby and I wanted to kill them] was the year or so I spent doing this work. Shame on everyone, this program is an embarassment and worse than the challenging educational problems it was designed to combat.
posted by jessamyn at 9:49 AM on March 7, 2011 [72 favorites]

That was depressingly unsurprising. I was surprised, however, by the link at the bottom of each page that said "More ads". Has anyone in the history of the internet clicked on a link that says "More ads"?
posted by Horace Rumpole at 9:51 AM on March 7, 2011 [5 favorites]

While I agree wholeheartedly with the sentiment that the writing and scoring of these standardized tests leaves much to be desired, I would like to point out some of the good that has come out of them.

Standardized tests are not metrics that tell you much about whether a person can think creatively, although some of the math questions on our state test do require some inventiveness. We should not imagine that doing well on standardized tests is the end-all of education.

However, the tests in my state do a decent job of measuring which students are able to do a basically okay job of reading and solving math problems. And we are never going to be able to educate really successful thinkers if they can't read or do basic math.

And sad to say, without a standardized metric that is publicly scrutinized, the low achievement of many students of color and low-income students would never be on the radar. It would be ignored the way it was ignored for decades. I think it is being ignored a bit less now and I think NCLB deserves part of the credit for that, however terrible it is on the whole.

What I am angry about now is that states all over the country, faced with the dawning realization that getting absolutely every student to read and do math and understand writing and science successfully is actually a really tall order, are not diving into that work but rather cutting much-needed resources from the education system.
posted by mai at 9:52 AM on March 7, 2011 [3 favorites]

It wasn't called standardized testing way back... but I drew pictures with the dots on the punchcards instead of answering the questions. When asked why I said because the test was stupid. The powers that be made me take all advanced courses (which I skipped most of the time)

Fast forward to "now" my son and daughter take the tests and then make fun of the tests, which I reward them for but they aren't allowed to skip.
posted by mrgroweler at 9:52 AM on March 7, 2011 [2 favorites]

I know someone who was kind of an egghead as a young child -- to the point that he was taking IQ tests as early as Kindergarten, and his first SAT test came when he was in third grade and his teacher sent him to take it just to see what would happen. (He took so many IQ tests that he actually started to recognize the questions -- "hey, there's that question with the parallelogram again.") Teachers eventually started grooming him for these tests, he says, as part of an overall "let's get this kid in a really good school or admit him to college early" program. Because of a variety of crap-luck life interruptions, though, and his parents recognizing that his social skills weren't advanced enough to skip any grades, he ended up with a fairly standard school career after all.

But he thinks that the overload on standardized testing kind of forced him into being an intellectual "sprinter", and admits he has trouble thinking through things in the long term. He's still brilliant, but wishes he'd had a very, very different approach.

....And for the flip side of things -- for some reason I'm reminded of, and am grateful for, my fifth-grade science teacher, Mr. Bartlett. One day in class, he had a couple of people from the Board of Ed doing a random observation on our class, and he was pulling out more bells and whistles in class than he usually did -- showed a brief filmstrip, read through ten pages in the textbook with us, showed us a little model of whatever he was talking about. I vaguely had the feeling that something about Mr. Bartlett's method felt different than usual, but I didn't think anything else of it.

Then the next day, when we all got into Mr. Bartlett's class, we found him sitting on top of his desk, with absolutely no other bells-and-whistles kind of things. He started off class by apologizing to us -- because the principal had pressured him to "show off" to the Board of Ed people, and that's why he was rushing through things the day before like that. But he was afraid that some of us hadn't been able to follow along, so he wanted to give us all a chance to ask him to explain anything we'd been confused about the day before; and that was all we did that day, was a review of the previous day in his own -- successful -- teaching style, rather than the style the Board of Ed was used to.

Some teachers have different methods, but those methods also work.
posted by EmpressCallipygos at 9:53 AM on March 7, 2011 [10 favorites]

I have had friends who did this, both in Georgia where the state actually hires its own temps to grade tests, and here in Durham where one of the most prominent businesses in our downtown is a for-profit test grading company.

As in the article, my friends who have done this are quite intelligent people with college degrees who desperately needed the money. And the stories that they can tell (carefully obliterating all identifying information of course) are like the ones in the article.

I also have friends who are public school teachers. They are equally intelligent people with college degrees. The main difference between them and my friends who grade tests is that the teachers are passionate about their jobs and care about their students. They are not simply in it for the paycheck, obviously, or they wouldn't be public school teachers in "right to work" states in the South.

I don't have anything new to add to the story, just that I've been watching this phenomenon for the past 10 years or so and it's absolutely amazing who holds kids' (and teachers') futures in their hands, how much teachers care about their jobs, and how much we are letting down both the kids and the teachers.
posted by hydropsyche at 9:53 AM on March 7, 2011 [1 favorite]

What we need is a way to spread the wisdom of the top 1% of teachers to all the students, "class size" be damned.

"He who can destroy a thing controls the thing. I will take one hundred of your finest teachers and train them. This one hundred will train the thousands that remain."

-Paul Muad'Dib, educational consultant, as interviewed in Waiting for the Kwisatz Haderach
posted by Your Disapproving Father at 9:53 AM on March 7, 2011 [11 favorites]

sonic meat machine, I know this will send us down another endless Mefi Discusses Education thread, but my understanding is that part of the reason for high scores in many other countries vs. the US is that low-scoring students are not put in special programs that attempt to improve achievement, but simply tracked or weeded out. By the time kids get to college level, they are a much more selective group.

Which is something we find problematic here in the US, where we have committed to giving every child, no matter how disadvantaged, as much education as they can absorb. I would appreciate correction if I am wrong, but I believe most Asian countries and even some European ones do not approach education in this way, with this mission.

This is part of what I mean by the "pseudo-religiosity" of education discussions. That's not a knock on you, but it does reveal the "received thinking" about education here. U.S. school systems aren't giving students "everything they can absorb;" instead, they're (theoretically) giving every student what the dimmest bulb in the class can absorb.

"Tracking" is given a bad name, but it generally ends up with people tracked toward something. Yes, people end up being plumbers and roofers and car mechanics, but there are a lot of people who end up being tradespeople who do well and who are happier – much happier than if you attempt to force them to be an actuary or paralegal.

The people who can learn the most do, and don't have to waste their time with unchallenging crap for twelve years. Throughout my grade school curriculum, I could never have imagined studying for a test, or making a C on one; it was just that easy. When we're not serving smart students or the ones who can't read or add by the time they graduate, it's time to look up from the ideology and see whether someone else is doing better.
posted by sonic meat machine at 10:00 AM on March 7, 2011 [6 favorites]

Of course, we could always have a well-funded, trained, and experienced state department of education both administer and grade the tests. Not only would you take out the profit motive, but you would also put the authority squarely where it already rests.
posted by Lord Chancellor at 10:01 AM on March 7, 2011 [1 favorite]

Can someone help Detroit out with rigging those test scores? Because clearly we do not know how to do it (but we are broke, so maybe we just can't pay enough to have them changed.)
http://www.detnews.com/article/20110225/SCHOOLS/102250376/1026/SCHOOLS/Detroit-Public-Schools-fails-on-science-test
posted by Kokopuff at 10:03 AM on March 7, 2011

This story really resonated with me because I spent a couple of weeks at one of NCS Pearson's grading warehouses back in 2001.

When I got there, it was just as was described in the article: I had a one day orientation and then was given sample essays to grade to see that I could follow the rubric.

Once in the grading center, we were told that at minimum, we needed to grade 8 packets of tests. Each packet had 18-25 essays, give or take. For the first couple of days, I really bore down and read through each essay and tried to justify my grade. I was consistently off on my grades.

So, I decided to judge the tests based solely on the first sentence, the last sentence, and how many paragraphs were in between. Voila! My eight packets were usually done before lunch, and commendations were being handed to me for my speed and accuracy. Most days, I sat around with a packet after lunch and took the whole afternoon to grade it while I wrote out outlines for short stories on a notepad. On the day I decided to see how many packets I could get done if I applied myself, I got through twenty two of them.

At the time, it was all a big joke. But, I now see that I did those kids a hell of a lot more harm than good, and that these tests are all about money, and nothing else.
posted by reenum at 10:03 AM on March 7, 2011 [10 favorites]

For instance, I was exposed to this in my high school days when I was in an AP Calculus course. I was "pretty good" at Calculus (which, for an American kid in a high school means that you understand next to nothing, but you're awesome at cranking out ridiculous computations for no reason).

I loved my HS AP calc class for the opposite reason: Our teacher made sure to explain why you differentiate and integrate, kind of as Mister Fabulous mentions. And we all got 4 & 5's on the exam.

wish I wasn't leading review sessions all day, so I have a *lot* more to say than the five minutes before my lunch is over will allow, so I'll bullet some major issues with the high-stakes testing

Another point I'll add from the technology side is the straight it can put on technology resources. Due to how our state administers testing, they all have to be set up with proxies and another annoying settings to prevent possible cheating. As a consequence, our entire computer lab is out of commission until the end of the year. And this is for a small school district where each student takes at least one day, or 2 morning to finish their testing. On top of that, there is a very finite window to perform this testing. What school districts who have thousands of students and not enough computers to do all this testing will do next year when certain portions must be completed online, I haven't a clue. With all the funding cuts, reality is some schools may not be able to afford the requisite technology assets needed for online testing.

And on top of that, the software is buggier than hell. The software is supposed to properly save responses for a student's next test, but that is not always the case, causing the test program to bug out. And if a student somehow manages to make the testing software think they moved the open window, the software will crash on you.
posted by jmd82 at 10:06 AM on March 7, 2011

But guys but guys but guys.....

There has to be some independent measure of how well our schools are doing. We can't trust the local school boards, the parents don't always know what Johnny should have mastered by grade level, and standardized multiple-choice bubble tests cannot measure some very important things, beginning with a child's ability to write.

And yet we absolutely need some way of determining what is working and what isn't in our public schools. I don't have the answer either, but all this "Standardized tests suck, yadda yadda yadda...." isn't getting us to the solution either.
posted by LarryC at 10:06 AM on March 7, 2011 [4 favorites]

ColdChef: Let this be a lesson, kids. Asshattery will get you nowhere, but a one-way ticket to even more asshattery.

Is that a good thing or a bad thing? Conclusion unclear, minus 5 points.
posted by filthy light thief at 10:13 AM on March 7, 2011 [1 favorite]

Schedule Disruption: for everyone one day of standardized testing, there are at least 3-4 more days of test prep - mock tests, etc - which essentially shut down the entire school.

Re: teaching to the test: I always assumed this meant that the tests can only cover a certain range of topics and so teachers only spend time teaching those topics and not going over other information that may be just as or more worthwhile.

This was the experience with my recently graduated American public high school student.

There has to be some independent measure of how well our schools are doing

I blame computers and human desire for control for this. We live in a world where everything can be counted and measured, so why not teachers and students too?
posted by Brandon Blatcher at 10:15 AM on March 7, 2011

Oh, and conservatards? This is what happens when you outsource the task of educating our children to profit makers: costs will be cut until the product is worthless.

Pardon me if this was said already, but THEY KNOW THIS....THIS IS THEIR GOAL. Because then we will ALL be indentured servants, thus lowering wages back to industrial age standards for all of us.
posted by spicynuts at 10:15 AM on March 7, 2011 [3 favorites]

homunculus: "Daily Show: Diane Ravitch believes education reform should focus on getting children out of poverty, not finding the bad teachers."

Before Ravitch gets considered canonical in this thread, you should be well aware of the poor reputation for becoming vehemently against all the things she was vehemently for under Bush I (lots of which has manifested in some strange behavior, like blocking people on Twitter).

Making tests is really difficult, and grading tests is even moreso. The unfortunate reality is that a school system that routinely fails to give kids basic literacy and numeracy skills has to be held accountable somehow. Fortunately, it turns out taking tests isn't a terrible way to learn things, so long as the tests comprehensively cover things students need to know and include a variety of tasks that measure critical thinking and writing skills. The best charter schools, increasingly, are demonstrating that music and art make kids more capable of the information processing that good tests measure - excellence tends to breed excellence.

There's no doubt the field's got a lot of problems, most of them starting with the poor way many tests are graded. But the key to fixing those problems is investing more time and attention to making tests better, prepping scorers better and (critically) developing a real national curriculm with real national standards that can use the power of scale to make good tests. Making an audience of upper middle class white people in their early 20s at the Daily Show boo because they didn't like bubble sheets doesn't do that.
posted by l33tpolicywonk at 10:17 AM on March 7, 2011 [7 favorites]

pwnguin / saulgoodman: So instead of having the teachers grade the tests (and open it up to teachers giving their own students glowing reviews, resulting in their schools hopefully getting accolades & cash), we have the current situation where low-pay temp workers basically make up the grades?

I wouldn't go quite as far as that characterization. It's not that they make them up--they just ignore the fact that they aren't necessarily measuring what they claim to be measuring, don't always do a lot of the hard analysis to look for unintended biasing effects in the results, and there's a tendency to use the results punitively as mechanisms for grading and evaluating teacher performance, even when the data isn't there.

For example, in Florida, the FCAT is administered only in the 4th, 8th and 10th grades. And yet, lawmakers here have tried to tie all teacher performance evaluations to standardized test scores. How is it fair to evaluate a 9th grade teacher, in part, based on a students' learning gains on a standardized test administered a year before the student ever entered their classroom? Well, it isn't. Not to even mention how you account for the fact that not all classroom teachers teach the subjects tested on standardized tests and their impact on each area of learning gain would need to be weighted relative to their expected contributions--currently, there isn't even anything approaching attention to the finer-grained problems; the big glaring ones aren't even getting a whole lot of attention. I once helped spec out a data mart project that looked at teacher performance in relation to standardized test results. It's literally impossible in practical terms to pull together the performance metrics the politicians would prefer to see currently, and I don't see any way you ever could sensibly approach what's claimed to be the ideal set of metrics without massive systemic changes across the board that would significantly increase the costs of public education overall by requiring massive new administrative investments. And in the current political climate, no one is going to get elected on a promise to spend more on education bureaucracy, whether that's what it would take or not to get the kind of sound, long-term performance metrics everyone just assumes we already have somehow.
posted by saulgoodman at 10:20 AM on March 7, 2011 [1 favorite]

And yet we absolutely need some way of determining what is working and what isn't in our public schools. I don't have the answer either, but all this "Standardized tests suck, yadda yadda yadda...." isn't getting us to the solution either.

It's not my area of expertise at all, but if the goal is to rate a particular school (or teachers) effectiveness, wouldn't the best strategy be to:

1. define metrics (ie: get very clear on what exactly is being rated)
2. RANDOMLY choose a statistically valid number of kids from a certain grade
3. have these randomly chosen kids write a bunch of tests on a bunch of topics (obviously in such a way that it counts as regular class time for them)
4. mark these tests in some blind way (ie: those marking and those supervising them have no idea which kids from which schools are actually being graded).
5. use the results of these marks as a means to open investigation and discussion into what is (or isn't) working at particular schools
posted by philip-random at 10:21 AM on March 7, 2011 [1 favorite]

and (critically) developing a real national curriculm with real national standards that can use the power of scale to make good tests.

But, but, what about state's rights?!
posted by Brandon Blatcher at 10:21 AM on March 7, 2011 [1 favorite]

Ach--posted that too soon; meant to correct something. It's only the writing part of the FCAT that's administered in those time periods I mentioned; I guess we're up to doing the FCAT in all grades 3--11 now, with the exception of the writing. Still, there are big gaps in the data.
posted by saulgoodman at 10:22 AM on March 7, 2011

Sorry. Please note this correction to this comment.

"The writing portion of the FCAT is administered only in the 4th, 8th and 10th grades"
posted by saulgoodman at 10:23 AM on March 7, 2011

saulgoodman: "For example, in Florida, the FCAT is administered only in the 4th, 8th and 10th grades. And yet, lawmakers here have tried to tie all teacher performance evaluations to standardized test scores. How is it fair to evaluate a 9th grade teacher, in part, based on a students' learning gains on a standardized test administered a year before the student ever entered their classroom?"

The performance pay proposal before the legislature requires districts to develop assessments for grades in which FCAT isn't used. The bill, btw, only requires that 50% of the evaluation be based on scores, which is the highest percentage I've ever seen a mainstream ed reform proposal push for.
posted by l33tpolicywonk at 10:29 AM on March 7, 2011

Brandon: yes, lol, but...

Is there any reason why the Feds can't open their own 'charter' schools? The feds buy the land, implement their own curriculum, and take as many students as choose to attend given space constraints.

They could start with inner city school districts, which are both populous and awful.

The teachers would be highly educated and go through a comprehensive hiring process, on par with the Foreign Service, and be paid commensurate with their abilities.

The Feds would receive student $ from the state the same as any other charter school, and would expand as interest grows.

Let's have the Dept of Education actually do some educating.
posted by leotrotsky at 10:29 AM on March 7, 2011 [2 favorites]

For all the criticism being leveled at standardized testing in this thread, it seems to me that this article is really about one central problem: the testers' incentive to score consistently with some 'ideal' score distribution and with each other, rather than scoring according to the desired rubric.

Assuming that we want accurate results from these tests (and I realize this is a big assumption, especially in the light of the southeastern state's Department of Education rep telling the already-fudging scorers to change the rubric so that their students would be scored better) the solution seems fairly straightforward: spot checking by the Department of Education by scoring some sample of these tests by independent scorers who have the time and dedication and necessary incentives to score accurately according to the rubric, additionally they could provide feedback to the outsourced scorers so they could modify their process in whatever way was necessary to provide accurate results

I'd assume this sort of process would be fundamental to the design of a testing system. If it isn't I'd say that its the real problem here. I think accurate, well designed testing could be a useful feedback mechanism to students, teachers, parents, school district administrators, various concerned parts of the government, and the community as a whole.

I think the goal behind standardized testing is a good one: we want to know through some reasonable, objective criteria how our students are performing on various basic skills, and where we need to improve. I think perhaps the real question is: are we willing to do what is necessary to really find out. Mai's comment hits the nail on the head about the potential value of these tests for at least drawing attention to where our educational problems lie.
posted by Reverend John at 10:34 AM on March 7, 2011 [2 favorites]

There's no doubt the field's got a lot of problems, most of them starting with the poor way many tests are graded. But the key to fixing those problems is investing more time and attention to making tests better, prepping scorers better and (critically) developing a real national curriculm with real national standards that can use the power of scale to make good tests. Making an audience of upper middle class white people in their early 20s at the Daily Show boo because they didn't like bubble sheets doesn't do that.
posted by l33tpolicywonk at 1:17 PM on March 7 [+] [!]

eponysterical.
posted by ennui.bz at 10:42 AM on March 7, 2011 [1 favorite]

The performance pay proposal before the legislature requires districts to develop assessments for grades in which FCAT isn't used. The bill, btw, only requires that 50% of the evaluation be based on scores, which is the highest percentage I've ever seen a mainstream ed reform proposal push for.

While it doesn't directly impact teacher-compensation, the DOE's Education Data Warehouse has also been tasked, at times, with reporting and providing dashboards on this stuff for executive-level consumption, to guide policy analysis and decision-making. That's really where I first worked on these issues a while back, although I'm also involved in the voluntary prekindergarten (VPK) readiness assessment processes used to evaluate funding-levels for VPK programs throughout the state. I can tell you anecdotally, the main interest from the politicos when it comes to performance metrics always seems to be in figuring out ways to evaluate teacher performance, rather than on improving education outcomes for students.

At any rate, from what I've seen, the data needed to really do this kind of thing properly isn't there now, hasn't ever been there before, and couldn't possibly be there without someone, somewhere spending a lot more money than anyone wants to spend on the administrative side of education these days.
posted by saulgoodman at 10:43 AM on March 7, 2011 [1 favorite]

I have a question about these tests. When teachers "teach to the test", are they teaching the kids to memorize answers to specific questions which will be repeated verbatim in the actual test?

It's both, with them getting as close to the latter as possible. At least sometimes.

I just read an interesting book about this: Tested: One American School Struggles to Make the Grade, which follows a school's staff and students for one year and really shows how stressful the testing is, and how it distorts the whole educational process.
posted by not that girl at 10:43 AM on March 7, 2011

Some of you seem to be under the impression the powers that be desire a well educated populace. Interesting.
posted by notreally at 10:45 AM on March 7, 2011 [3 favorites]

Re: teaching to the test - my wife is a high school math teacher, and from what she's said, there's a two-part approach. First, the teachers know the topics to be covered on the standardized tests, and they (hopefully) work together to figure out a timeline to cover all the topics in time for the test. Second, the students are taught how to take a test. Some of this applies to the NCLB tests, but some is broader and helps kids in all future tests, so the program isn't completely flawed (just mostly).

Apparently No Child Left Behind is on the way out, but not anytime soon.

Under the federal act, every child is supposed to test on grade level in reading and math by 2014. "That can't happen," said Morton. "You have too many variables and you have too many scenarios, and everybody knows that would never happen."

That's right, every child. Special needs, English language learners, EVERYONE. Each year, the percentage of children testing at grade increases, so if your school missed the mark last year, there's always next year to do better, but the goal is higher.
posted by filthy light thief at 10:45 AM on March 7, 2011

also, ex-calculus teacher here... those grades you got in calculus, they have very little statistical validity. i'd guesstimate that, scientifically speaking, the best we can do with any validity for graded assesment is grad-school style grades: excellent, acceptable, and unacceptable.

and another thing: grading is about 80% social engineering. it's both carrot and stick, and the purpose isn't really to assess but to motivate and to also give students a sense they have done something in a class. (even if that calculus class sucked and taught you very little, your grade at the end made you feel like you actually had accomplished somethingh, at least long enough that you were graduated out the door)
posted by ennui.bz at 10:48 AM on March 7, 2011 [1 favorite]

saulgoodman: "I can tell you anecdotally, the main interest from the politicos when it comes to performance metrics always seems to be in figuring out ways to evaluate teacher performance, rather than on improving education outcomes for students. "

I know we've parted ways on this before, Saul, but these are largely the same thing. There's a barrage of studies that suggest teachers are the number one most important factor driving student outcomes and that teacher quality is low particularly in poor communities of color. Despite all of that, 97-99% of teachers get rated effective in any given year, and in almost every state, the only or the major basis for deciding who keeps a teaching job and who doesn't is how long they've been teaching.

I and most of the ed reformers I know don't hate teachers. Many of us are married to one, were a teacher, are a teacher or had some awesome teachers and wouldn't be in the field if that wasn't true. Still, we want the stakes to be high for teachers because the stakes for their graduates, particularly those born in poverty, are the highest imaginable: they're the difference between meaningful work and generational poverty. Teaching should be an elite occupation - one of the most elite ones we have. But elite occupations share two important qualities: they pay a lot of money, and lots of folks don't make the cut. We have to be moving towards a system where both of those things are true.
posted by l33tpolicywonk at 10:50 AM on March 7, 2011 [2 favorites]

This is part of what I mean by the "pseudo-religiosity" of education discussions. That's not a knock on you, but it does reveal the "received thinking" about education here. U.S. school systems aren't giving students "everything they can absorb;" instead, they're (theoretically) giving every student what the dimmest bulb in the class can absorb.

"Tracking" is given a bad name, but it generally ends up with people tracked toward something. Yes, people end up being plumbers and roofers and car mechanics, but there are a lot of people who end up being tradespeople who do well and who are happier – much happier than if you attempt to force them to be an actuary or paralegal.

Nah, I wasn't saying it was my received wisdom, simply that if you are comparing the US to other countries, you have to account for different goals and challenges.

I'm on board with not making college the end goal for every child, so long as we also aren't consigning non-college students to low wage and low status jobs. In a society as stratified as ours, it's not surprising that trying to help every child to thrive has proved difficult. I don't agree that letting the devil take the hindmost is the ethical response, though.
posted by emjaybee at 10:53 AM on March 7, 2011

But elite occupations share two important qualities: they pay a lot of money, and lots of folks don't make the cut. We have to be moving towards a system where both of those things are true.

I'm reminded of the excellent montage of clips which Jon Stewart ran through on Thursday's episode of The Daily Show, showing talking head pundit types speaking out of both sides of their mouths about job performance and compensation in the context of Wall Street and teacher pay.

It was pretty great to see the EXACT SAME PERSON who in one clip was saying that $50K plus benefits was too much money for a teacher to earn and they need to suck it up and take less, and then in the next clip say that $250K was nearly poverty level wages for a family with children, and that capping salaries for bank CEOs would result in all the best talent leaving the industry.
posted by hippybear at 10:57 AM on March 7, 2011 [32 favorites]

Still, we want the stakes to be high for teachers because the stakes for their graduates, particularly those born in poverty, are the highest imaginable: they're the difference between meaningful work and generational poverty. Teaching should be an elite occupation - one of the most elite ones we have. But elite occupations share two important qualities: they pay a lot of money, and lots of folks don't make the cut. We have to be moving towards a system where both of those things are true.

Then what you need is a completely new para-educational administrative support system that operates side-by-side with whatever existing education system we have to evaluate and rate teachers. You simply can't expect to build an entirely new parallel administrative system for evaluating, rating and ranking teacher performance without spending a lot of money on education over and above existing spending, not in lieu of it. But all of these efforts are instead draining money out of a finite pot of money that's already shrinking due to state level financial and revenue capture problems. You can't expect to get all this new information about teacher performance from a system that was never designed that way for nothing.

There's an unbridgeable gap, in the current political environment, between the ideal and the possible, and all this relentless pushing for the impossible is harming the system more than helping currently, IMO. If you want the ability to really connect student classroom performance to teacher performance in a strictly non-subjective, measurable way, then you're going to have to solve some more fundamental epistemological problems and political problems first. As a population, we aren't even comfortable with supporting education at current funding levels, and there's no way we'd be willing to pay what it would cost to achieve the kind of comprehensive, meta-educational educator evaluation system we're trying to boot-strap onto what was originally meant to be a system for educating and evaluating students. If education were professional sports, it would be like setting up a whole new meta-refereeing system to oversee the referees, without expecting it to cost us anything.
posted by saulgoodman at 11:04 AM on March 7, 2011 [1 favorite]

saulgoodman: "You simply can't expect to build an entirely new parallel administrative system for evaluating, rating and ranking teacher performance without spending a lot of money on education over and above existing spending, not in lieu of it"

I don't expect to do that, nor do most left-leaning folks on this side of the debate. But the paradigm you set up creates a pretty much impossible trap: only after the political muscle is overwhelming for funding schools can we talk about what's worth spending the money on.

Wisconsin has unfortunately taken a debate full of strawmen and just tossed more straw on the pile. There are an incredible number of shades of gray somewhere between hippybear's talking heads who want to destroy unions and collective bargaining rights and people who believe nothing about the elements of the education system involving adults should change. Lots of those shades include people who think we should spend more money on programs that work and programs that are worth experimenting on. That's what's happening when parents in Compton fight for the right to give their kids a school that isn't failing. That's what's happening when Michelle Rhee raises private money to supplement public funds so she can pay the best teachers $100,000 a year and when great charter networks like KIPP and SEED and Green Dot raise the same funds to compensate for states that don't give them the same allocations they give traditional public schools. That's what's happening when the president and the secretary of education pass and implement the largest discretionary program in the history of the Department of Education.
posted by l33tpolicywonk at 11:37 AM on March 7, 2011

I had a fairly naive view on test scores and teacher evaluation until I got deeper into the value-added literature. There are somewhat intractable problems in the basic idea. First are the endogeneity problems: students are not randomly assigned to teachers; teachers are not randomly assigned to classes or schools. In most adjustment strategies, residual confounding to the unobserved determinants of academic and life success is the rule rather than the exception. Second, the measurement problems are immense. You end up comparing the trajectories of students across multiple years and classrooms, but of course at each year the tests are quite different and may not reflect the same underlying concepts at all. This NYT piece is a good example of that: a randomized trial found that the impact on test scores of good teachers in kindergarten disappeared by middle school and came back in middle-age income; the tests in middle school and later had stopped "seeing" something important that the elementary ones did. Third, the statistical problem is quite difficult. The test-retest validity of a given instrument may be quite high, but the variability in the intellectual growth of the student is huge, adding a bunch of noise to the system. The effect of a teacher may take several years to really show, since test items require several skills which a student may pick up only with time. The effect of teachers will also be quite heterogeneous across students due to teacher-skill student-need mismatching, which makes identifying the average effect vs maximal effect even noisier.

Finally, timely use of that information is difficult. Say that it takes 8 years to have enough data to really know if someone is a good teacher (3 years to hit their stride, 5 of collection). You'd have liked to be able to encourage or discourage that teacher a long time ago, but now that they're almost a decade in are you going to fire them?
posted by a robot made out of meat at 11:47 AM on March 7, 2011 [5 favorites]

Two things:

One: I've taken Spanish, German, and Japanese. Four semesters of german and one semester of spanish and Japanese. The Japanese class was designed by a woman from Japan and it was spectacular. The book was written by people from Japan and it was designed to maximize learning of the material. There was a focus on memorizing vocabulary before the class-- via listening to a CD that have every vocabulary word on audio such that you could practice listening to and repeating each word. Then the grammar sections were designed to demonstrate the appropriate sentence stucture with replacements of various nouns and verbs also on audio so that you could practice listening to and repeating out loud each sentence understanding it's meaning and feeling comfortable with the sentence structure.

Memorization activities were actually incorporated into class time such that we repeated and used the words in class. It was done magnificently and no other language class I've taken was designed so well.

Critical thinking and creative thinking are important but I wonder if in classes where a lot of memorization is really the focus of the class we could focus on learning from countries that do that well and apply that in our classes. We could still have classes designed to help students explore their critical thinking/creative thinking skills along with that.

Two: I have to throw in there that we have really high rates of learning disabilities and mental health issues that specifically affect learning. The reading I've been doing points to brain inflammation as a major issue in mental health issues. Further more, food additives , artificial flavorings, trauma, chronic stress, maternal infant separation, childhood abuse, and lack of plant phytochemicals in diet, all seem to have correlative and probable causal links with this inflammation. Treating mental health issues as strictly neurological without identifying causal life factors that are causing inflammation does not solve the actual biological problem.

We actually know a lot about the factors that are causing learning disabilities and until we realize that we need to address poverty, stress, and adversity in families we aren't going to really fix this throwing more pills at kids or screaming at our teachers for not knowing what to do with a bunch of emotionally disgruntled hyperactive/depressed stressed out kids who don't care about learning because their lives are fucked up.
posted by xarnop at 11:49 AM on March 7, 2011

There's another unintended consequence when you use standardized tests to rate teacher performance: good teachers stop teaching in at-risk schools. The job is already mind-numbingly difficult, and no single teacher can guarantee that his or her school will meet it's benchmarks. If you've got the chops and the seniority to pick your assignment, why put yourself through that meatgrinder AND risk being fired if the school doesn't make the grade? Anecdotally, I know two very good teachers -- one with over twenty years' experience in inner-city schools, and one with ten years teaching ESL to at-risk kids -- who have landed cushy gigs in the suburbs in the last few years; both made the move largely in response to NCLB testing.

I agree that there are a lot of poor teachers out there, and that we need some way to encourage them to seek other employment; but I can't see any way in which the current system is working.
posted by steambadger at 11:50 AM on March 7, 2011 [3 favorites]

Exactly what a robot made out of meat said is what I have in mind. There's no solid consensus about how to reliably measure teacher performance in the first place, and yet, the only kinds of policy proposals that even seem to be on the table anymore are those that take the easy availability of reliable and standardized performance metrics without any additional investment in the administrative side of education as a given. Nobody but nobody is out there saying "What we need to do is spend more education money on administrative functions outside of the classroom," and yet, that's really what everyone's demanding in these pushes to collect longitudinal metrics and evaluate teacher performance more precisely.
posted by saulgoodman at 12:03 PM on March 7, 2011 [1 favorite]

l33tpolicywonk: "There's a barrage of studies that suggest teachers are the number one most important factor driving student outcomes and that teacher quality is low particularly in poor communities of color.

Is teacher quality the most important factor in student achievement full stop or the most important factor in student achievement inside of school? (I've only done casual reading on the subject, but I thought it was the latter.)
posted by queseyo at 12:09 PM on March 7, 2011 [1 favorite]

Huh. Interesting. Like jessamyn once did, I score for ETS. I've actually found it quite okay, but I'm also scoring essays at a graduate level, which changes things, you know, when you know these students have minimally been through higher-level education. I've had a lot of conversations like the one quoted in the OP:

He hesitated the way he had been specifically trained not to. Then he hit, "3."

It didn't take long before a supervisor was in his face. He leaned down with a printout of the King essay.

"This really isn't a 3-style paper," the supervisor said.

Puthoff pointed out the smart use of examples and the exceptional prose. The supervisor just shook his head and pointed out how short the paragraphs were.

"You know, it's more of a 2," the supervisor repeated. "Not enough elaboration."

We're actually encouraged to have conversations like these, and my supervisors have always seemed quite happy to talk about the intricacies of the essays, but I don't know how much this varies from test to test and company to company. My biggest frustration has been with how the rubric mirrors traits that correlate with good writing, but don't necessarily reflect it, but the vast majority of the time, they do. I was worried I'd feel like a soulless cog, but overall, I actually kind of like my job.

(And that's all I should probably say in light of the aforementioned scary NDA and all).
posted by PhoBWanKenobi at 12:14 PM on March 7, 2011 [1 favorite]

When I teach Statistics, I bring up "regression to the mean". The classic example of it is the fact that very tall parents have kids that are unusually tall, but not as tall as their parents.

It seems to me that standardized tests and school grades suffer from the same problem. Suppose you are a school that scored above average this year. We shouldn't expect you to score above average next year because of regression to the mean. But that is exactly what you are dinged for; unless you improve each year, year after year, your school grade goes down, as I understand it. How could you possibly be successful?
posted by wittgenstein at 12:18 PM on March 7, 2011 [3 favorites]

queseyo: "Is teacher quality the most important factor in student achievement full stop or the most important factor in student achievement inside of school?"

The research goes back and forth on this, but it's not a question that I think has all that much policy relevance. Let me put it this way: schools are the agent that we as a society trust to educate kids. That's why we send them there 6.5 hours a day for 180 days a year for 13 years by law (frankly, I think that's too short, but that's another conversation). Yes, all children would do significantly better if they were well-fed, lived in safe neighborhoods and had loving parents who read to them every night. And, like the almost-socialist I am, I support programs that strive to do those things in my voting habits and political activity and financial decisions - and you should too.

But the bottom line is, despite all of those challenges, the best teachers can close the achievement gap in three years and produce 1.5 years of learning for every year in the classroom. Further, a country with zero childhood poverty still hasn't taught children literacy and numeracy - doing so is a sophisticated craft that requires professional specialists. Everybody in that system - teachers, principals (especially principals), superintendents and politicians - have to be held accountable for teaching kids those skills, and the only way to make sure the best-off kids are learning as much as the worst-off kids is to give both kids the same test and measure the difference.
posted by l33tpolicywonk at 12:21 PM on March 7, 2011 [1 favorite]

It is the job of principals to supervise teachers, not of some soulless corporation that is raking in money hand over fist.

I saw the same kind of numbers gaming with the trouble ticket system when I worked at EDS. You were measured on how quickly you got stuff out of your queue. So, if something you didn't do ended up in your queue, rather than spend time find the right team to send it to, you sent it to any other team so you could make your metrics. If you were waiting for a part to fix a printer, you did the same thing, hoping that the part would come in by the time the ticket had spent 3-6 weeks being bounced from queue to queue.

Our metrics always looked great, but customer service and satisfaction took a nosedive.

Supervisors need to be able to supervise. In theory, I'm all in favour of measuring things. It keeps the playing field even and everyone can see where the problems are. Except that when you worship numbers and only look at numbers, only the numbers are happy.
posted by QIbHom at 12:28 PM on March 7, 2011 [1 favorite]

How could you possibly be successful?

Cheat!

No really, that is what we are doing and have been doing for decades. A self-fulfilling bullshit prophecy. The golden example I have is an Honors Psychology class I took in college, where the average grade obtained by students was an A-. No one did worse than a B. The university arbitrarily declared in the various handbooks that C is a grade meaning "average." The justification of the inflated grade was that this was an honors class, and anyhow, the regular Psych 101 will balance it out.

This is the same kind of thinking that goes into the standardized tests. Arbitrary numbers will correct any "statistics" we need.
posted by Mister Fabulous at 12:29 PM on March 7, 2011 [3 favorites]

I don't mean to be contrarian but I think the people who say teacher evaluation is too difficult to attempt are missing the point. I had a 4th grade teacher who showed us horror movies 2-3 days a week instead of teaching, and a 9th grade biology teacher whose entire course consisted of memorizing perhaps 100 biology terms. (No labs, no homeworks, one multiple-choice test.) It should be straightforward to demonstrate that these kinds of teachers are below average. Of course there are many teachers whose test scores would be slightly below average, but within the margin of error when considering the student population and conditions, etc. ... those ones should be kept of course!
posted by miyabo at 12:30 PM on March 7, 2011 [1 favorite]

It should be straightforward to demonstrate that these kinds of teachers are below average.

I am a person who works with data for a living. Often education data, in fact. You assume too much of data. Data is very slippery and potentially misleading taken too literally.

It should be nontrivial for school officials to identify the kinds of teachers you describe, but sometimes those teachers are politically connected to the school administration. Sometimes they do manage to go under the radar. No amount of indirectly observing a student's performance at a given point in time is going to reliably predict whether or not a given teacher is showing horror movies in class all day. There's no necessary connection between poor performance on standardized tests to this kind of thing. Looking for this kind of problem teacher is the principal's job, really.
posted by saulgoodman at 12:55 PM on March 7, 2011 [2 favorites]

ach. "~~non~~trivial for school officials to identify..."
posted by saulgoodman at 12:58 PM on March 7, 2011

"The legitimacy of testing is being taken for granted," he says. "It's a farce."

But, but, this one goes to 11.
posted by Mental Wimp at 1:06 PM on March 7, 2011

(I haven't read every comment)

East Manitoba, "teaching to the test" means they cover material that will be on the test and they cover test-taking strategies. On the ACT, for example, 90% of the grammar questions are gleaned from a merely 10 or 15 grammar rules, short and easy to memorize. The point is not to learn grammar; the point is to memorize those 10 or 15 rules, which will get you through 90% of the questions. The rest can be successfully ignored.

One thing that is wrong with the U.S. education system is that so much of the CONTENT has been removed in favor of "skills," such as (just) reading and all these things like "identify the thesis" and so forth, so that in theory you use those same "skills" on any reading that you do. The trouble is, if you give a low-literacy fourth-grade boy who loves baseball a passage on baseball to read and a passage on iguanas to read, his CONTENT knowledge in baseball will give him a VAST advantage and he will score far better on the baseball passage than the iguanas passage even though he's "just" applying these allegedly mechanical, transferable skills.

Content is absolutely crucial to education, but "what skills should we teach?" is a much easier argument to have than "what CONTENT should we teach?" And try going into a failing school where kids can't read and announcing, "Instead of working on literacy skills 3 hours a day, we're going to spend 90 minutes of that time learning about iguanas. And planets. And Mark Twain. And Delaware. And stuff." Parents and community leaders FLIP THE HELL OUT. "If they can't read, why are you wasting time on Delaware?????!!!!!????" And even if you convince them that content matters, then its WHAT content, WHOSE content, etc.

And really, you go read a lot of primer-level reading material today and it's all so dry and boring and devoid of interesting facts or characters or ethical questions or dilemmas or plots ... it's just there for you to practice your reading skills. Well, why would you want to? What's the point of reading that crap?

(And indeed, a number of low-skills readers for whom it suddenly clicked suddenly discovered the joy of video game walkthroughs on the internet, or Sports Illustrated, or Beanie Baby Market Monthly, or whatever it is that feeds their personal interest. Motivated by interest in the content, their "skills" suddenly dramatically improve.)
posted by Eyebrows McGee at 1:10 PM on March 7, 2011 [8 favorites]

Suppose you are a school that scored above average this year. We shouldn't expect you to score above average next year because of regression to the mean.

Well, that's close, but not quite right. It's regression to the mean, not regression to the other side of the mean. [Slight derail: it's actually regression to higher density, but that's a different discussion; here, we're talking about unimodal distributions, so they both mean the same thing]. In this context, regression to the means leads us to expect next year's test scores be closer to the mean than this year's.
posted by Mental Wimp at 1:12 PM on March 7, 2011

miyabo: you're right that you'd think it should, in principle, work. And it does! It just turns out to not work very well in practice. Your ability to selectively weed out useless teachers is less than you might think. An administrator can combine tests with a variety of non-standardized test metrics, like randomly watching classroom performance. Just asking students is surprisingly effective. I think these measures work better at the administrator level, and that admins should be the ones with skin in the game. I'm told that they already act like insane tyrants; hopefully putting their money on the line would get them to be better.
posted by a robot made out of meat at 1:32 PM on March 7, 2011

And I thought it was bad when I was taking Algebra for Dummies and the teacher gave us a practice test all week long... and then the final test was the same practice test for all but the front page. When I asked him about this, he said, "Shhh." And since this was Algebra for Dummies, he told me that my grade would have been an A, but the rest of the curve was so low that I had to get an A-.

Yeah.

But seriously now, every time I read one of These Schools Today articles, I seriously wonder if I would have ever been able to graduate. And other than the dumminess in math, I was in the freaking gifted program.
posted by jenfullmoon at 1:34 PM on March 7, 2011

Mental Wimp: I get your point, but as I understand it, your score going down from the previous year is enough to get you dinged, even if you are still above the mean. You are only compared to the previous year, I believe.
posted by wittgenstein at 1:42 PM on March 7, 2011

l33t, "There's a barrage of studies that suggest teachers are the number one most important factor driving student outcomes and that teacher quality is low particularly in poor communities of color."

Studies suggest that teachers are the most important factor WITHIN THE CONTROL OF SCHOOLS, and this has ENORMOUS policy relevance, because of high achievement in schools depends on a comprehensive anti-poverty program that addresses nutrition, housing stability, parental involvement, etc., and tries to remediate HUUUUUUUGE differentiators such as parental education level and income level, it's absolutely meaningless to JUST talk about "education reform" and expect the schools to solve the problem.

I mean, I live in a district in the county with (one of) the highest rates of lead poisoning in the nation. Unsurprisingly, we also have SKY-HIGH special ed rates, far above what's expected for our demographics. The teacher is CLEARLY not the most important factor in student achievement in this scenario: UNREMEDIATED LEAD IS. And that's not something the school has the power to do anything about. You can put the best teachers in the nation in those classrooms, but they're still going to be teaching lead-poisoned children who have almost zero access to health care. However, the local debate on "what's wrong with our schools" is about bad teachers and lazy students and absent parents and bloated administration, not poor housing stock with unremediated lead and total lack of access to health care for many families. Because that's not an EDUCATIONAL problem. Until it is.

It's an extreme example, but there are plenty of other extreme examples out there.
posted by Eyebrows McGee at 1:46 PM on March 7, 2011 [4 favorites]

"because IF high achievement"

Apparently I did not learn my 10 ACT grammar rules, sheesh. I am a typo machine today!
posted by Eyebrows McGee at 1:49 PM on March 7, 2011

I'm not sure if people here are familiar with Paul Lockhart (a mathematician), but I feel like this thread is as good as any to link to A Mathematician's Lament [pdf]. It's a little treatise on the state of mathematics education, but everything in it applies to education as a whole in my mind.

An excerpt:

"Sadly, our present system of mathematics education is precisely this kind of nightmare. In fact, if I had to design a mechanism for the express purpose of destroying a child’s natural curiosity and love of pattern-making, I couldn’t possibly do as good a job as is currently being done— I simply wouldn’t have the imagination to come up with the kind of senseless, soulcrushing ideas that constitute contemporary mathematics education."

Take out the specific mathematics references, and it's what we have here. He doesn't really have any solutions to offer per se, so only read if you want to curl up into a ball and die.
posted by King Bee at 1:53 PM on March 7, 2011 [1 favorite]

It's an extreme example, but there are plenty of other extreme examples out there.

Exactly--there are far more special cases out there than people want to imagine or that our instinctive intellectual tendencies toward generalization can handle.

For another example of how tricky data can be from my own experience, while matching some data we once came across a record that looked like a duplicate. The demographic info for both kids was virtually identical--same name, same gender, same social security number, everything. The only notable difference between the two records was that one record had a different birth date than the other--same month and year, mind you, but a different day. All along, we assumed it was a data cleansing issue that originated with the vendor who provided the data; we'd had issues with duplicate records slipping through their cleansing processes in the past. But nope. That wasn't it at all. Turns out, as a little in-person follow-up determined, there were two, actual flesh and blood kids at the school with the same names and all the other relevant details--they just hadn't been issued social security numbers yet (presumably, they were documented immigrants), and both kids had been entered into the system under their Uncle's social security number (which was an optional data element anyway).

That's to say nothing of all the weird stuff people do to data when they're implementing on-the-fly "work-arounds."
posted by saulgoodman at 2:02 PM on March 7, 2011

God, I wish I could find the law of economics and politics that says, basically, where there's an incentive attached to a measurement, the measurement becomes meaningless as a metric for the underlying behavior, because the incentive will always artificially influence the underlying behavior that you seek to measure.
posted by klangklangston at 2:18 PM on March 7, 2011 [4 favorites]

Eyebrows McGee: "It's an extreme example, but there are plenty of other extreme examples out there."

Yes, there are. But bemoaning the myriad of problems facing the poor in this country gets us nowhere closer to fixing any of them.

Do I think water supplies shouldn't have lead in them? Yes. Do I tend to vote for politicians who give a damn about issues like pollution in general and the pollution that harms poor people specifically? Absolutely. Does that prevent poor kids from being taught effectively and lifted out of poverty? I don't think so, and I don't think so because there are individual schools in many communities which are doing amazing work getting these kids to college - many of them despite funding challenges, relying on inexperienced teachers or TFA teachers, etc. All of this leads me to believe that money + will + good ideas can institutionalize best practices.

This really isn't personal. I don't want to send bad teachers to jail, or (unlike some) vilify them on TV, or care only about the science and the inference about what kinds of schools lead to kids getting to college and what kinds don't. Because, at the end of the day, there's lead in the water AND school administrations are often bloated AND kids don't have adequate health care AND the best teachers aren't rewarded enough AND it takes two years to fire an ineffective teacher AND housing stock is poor. We can even disagree about the most efficient route to get there (I think that, because poverty is generational, the best approach to closing it is to produce a generation of poor kids equipped to take 21st century jobs and thereby bring wealth and political power back to their communities) without losing sight of the fact that all of those problems exist, all at once, and as a collective political body we should be interested in solving all of them.

In fact, policies which get higher quality teachers and apportion them better among poor schools can and should be a bigger part of a generalized anti-poverty policy. The most recent paper which tries to argue this point, and which uses giant data sets from California and Florida, demonstrates that, while teachers on the high end of the quality spectrum perform about the same in rich and poor schools, teachers on the low end are substantially worse in poor schools than rich schools. Poor kids, just like rich kids, can do extraordinarily well in school - provided they get a good teacher. The problem is, they're far less likely to draw a good teacher out of the hat than their rich peers are.

So, yeah, I specialized in one because you can't be everywhere all the time, and so I decided this was the place I could best leverage change. You picked a different one, or different ones, to specialize in. That's good. That's how it's supposed to work. But denying the power of schools to create change (or, like Saul, saying there are so many special cases it's impossible to generalize solutions) is like denying the power of the EPA to get the damn lead out of the water because there are so many companies to regulate all at once: both are a form of policy nihilism that, frankly, isn't backed up very well by history. More to the OP's discussion, all of this is more reason (not less) why measuring the scholastic performance of poor kids is important, and why standardized measures should be used to do it.
posted by l33tpolicywonk at 2:32 PM on March 7, 2011

your score going down from the previous year is enough to get you dinged

If that's the case, then they definitely are added victims of regression to the mean. Another drop of sadness in the sad, sad sea that is NCLB.
posted by Mental Wimp at 2:58 PM on March 7, 2011

teachers on the high end of the quality spectrum perform about the same in rich and poor schools, teachers on the low end are substantially worse in poor schools than rich schools.

l33tpolicywonk, I haven't read the paper, but are you saying that poor teachers do fine in rich schools? That given a better environment, even bad teachers do okay? Because that would be awesome to hear.

Awesome, but also sad because instead of working toward funding parity, current efforts at education reform are aimed almost entirely at finding a way to fire the teachers who struggle to perform superhuman feats in the face of overwhelming difficulty. That doesn't seem fair to me. Children should not be punished because they were born into a hostile environment, and teachers should not be punished for being hired into one.
posted by jsturgill at 3:18 PM on March 7, 2011

This is on par with the Republican dismantling of public resources.

The Republican Education Playbook, in a nutshell:

1) Run on a campaign that government (and government services) are bad. Say that, for example, teachers are overpaid and underqualified.
2) Win the election
3) Reduce funding or eliminate programs altogether, in an effort to "balance the budget". Meanwhile, reduce taxes on corporations and the wealthy.
4) Introduce standardized testing that completely hijacks the curriculum of schools' lesson plans. Put teachers between a rock and a hard place.
5) Watch as base competency level of students drop precipitously. Blame the teachers.
6) Cry that we need to give public money towards private schools. Insist that public schools are a sinking ship, but never admit that you (the Republican politician) are a big reason that happened.
7) ?????
8) Profit!!! (No, really, profit. Those who own private schools will be that much better off if we squeeze the public schools to death.)

In the end, Republicans are simple creatures. They are motivated by only one of three things: either restricting women's sexual freedom, establishing a Christian theocracy, or making money.
posted by zardoz at 3:49 PM on March 7, 2011 [4 favorites]

At least Chinese, Japanese, Korean, Vietnamese, and Indonesian schools produce students who can add and read and speak foreign languages.

Ah. Ahah. Haha. Seriously, no. As someone who has worked in Japanese education for roughly 11 years, this is flat out wrong. My job is to teach students to speak English. However, the job of the Japanese teachers of English is to get the students to pass the overly difficult tests set forth by the national BOE, which believes that English fossilized sometime around the 1940s. The equivalent of 9th graders in Japan get broad-sided with relative clauses, spending serious amounts of time on who vs. whom. These same students, usually, get freaked out and don't know what to do past "I'm fine, thank you. And you?" in a conversation. Their teachers, by and large, teach entirely in Japanese* rarely speaking English. The students, in turn, see that the Japanese teachers (the people who should actually be the best speaker of English they know) essentially refuse to use the language they teach, so they learn from early on that speaking is the least essential part of their English education. They won't be tested on it, see, much like the writing section on American standardized testing mentioned above. The thing is, by teaching to the test obsessively, the Japanese students somehow manage to get decent scores in English without being able to speak at all. Most educational theory I've read about language acquistion says that using the language actively, speaking it regularly, is the fastest, surest way to increase fluency, and my professional experience (as well as personal experience as a failed French student and moderate speaker of Japanese) shows me the same.

So, yeah, please don't buy into the myth that Japanese education works. It's a collection of people doing the same things they're own teachers did 30 years ago, simply because that's how it was done before, and how it will always be done in the future.

^{*Of course, from either this year, or next year, the JBOE has mandated that all English courses be taught only in English. Typically, the JBOE has not created any sort of plan to implement this massive change. As many, many Japanese teachers of English can barely speak the language, it will be a disaster, scores will plummet, and within a couple years, things will revert back to the way they were before.

/ranty derail about comparing American education to Japanese education.}
posted by Ghidorah at 4:04 PM on March 7, 2011 [3 favorites]

You know, when I first encountered the world of Ye ar Zero, I was convinced that yes, it maybe could happen, but probably not because there are sort of systemic safeguards against that kind of thing. But now, a few years later, I'm watching a full-frontal assault on a lot of those safeguards. Some of the strategies are short-term (dismantle unions and restrict voting by the young and the poor), others are long term (ruin our educational system so people just do what their televisions yell at them to do).

It's getting increasingly creepy, and I hate that I feel generally helpless to do anything about it.
posted by hippybear at 4:05 PM on March 7, 2011

More to the OP's discussion, all of this is more reason (not less) why measuring the scholastic performance of poor kids is important, and why standardized measures should be used to do it.

l33tpolicywonk, I think you make some good points here. But one of my biggest problems with the standardized testing booms is this: In forcing high-stakes standardized tests every year to measure teaching quality, teachers are forced to teach less well.

When you have students who weren't raised to value education a whole lot? Students who don't see why they should care about learning stupid school stuff because they have bigger problems to deal with? They need teachers to inspire them to love learning. They need teachers who inspire. Teachers who inspire are teachers who innovate, who are free to teach creatively and to adapt how and what they teach to the needs of their students.

Teachers who are teaching to a test cannot do any of these things. Having to teach certain things is not the problem; the problem is having to teach certain things in certain ways, and to focus on certain things to the exclusion of all else. I know teachers who now have no choice in how they teach their math lessons, down to the very words that come out of their mouths. They have the scripts that their school district makes them read and a day-by-day schedule of lessons from which they cannot deviate. How is a teacher dispassionately reading a script supposed to improve learning? How is a breakneck pace of lessons designed to get to the test going to help a child (or a classroom of children) when they can't keep up?

And how are children supposed to learn to love learning, when instead of being inspired by learning about the rainforest and planets and art and fun historical anecdotes, all they get is an endless drill of grammar lessons and math worksheets?

Honestly, from what I have observed, standardized testing makes school suck more for kids, and sucks the joy and fun out of teaching. The focus on standardized testing may result in improvement on how students do on the tests, but it can hurt their actual comprehension of the material (as has been discussed by many people in this thread), it certainly reduces the chance of them finding joy in learning and school, and it results in them being much less educated about everything that is not on these tests. (I am talking specifically about elementary school here, where, IMHO as someone who knows many California elementary school teachers, the effects have been the worst. )

Standardized testing, now, is a joke, a business, a game. It turns schools from places designed to produced educated students to places designed to produce high test scores.

Of course there should be some way to hold teachers accountable and produce better teachers. But somehow, I think that policy changes that push creative, inspiring educators out by removing all of their ability to be creative in the classroom aren't actually going to improve things.
posted by mandanza at 4:13 PM on March 7, 2011 [2 favorites]

So, yeah, please don't buy into the myth that Japanese education works.

Fair enough. We will replace them forthwith with "Finnish."
posted by sonic meat machine at 4:23 PM on March 7, 2011

"l33tpolicywonk, I haven't read the paper, but are you saying that poor teachers do fine in rich schools? That given a better environment, even bad teachers do okay? Because that would be awesome to hear. "

One of the key findings of the LA Times review of student testing records was that the concept of "value added" should be included in teacher reviews, in part because poor or mediocre teachers at richer schools (or just "better" schools) coasted on the higher baseline of performance of their students.

Without that metric, teachers whose students go from a hypothetical 20 percent proficiency to 50 percent proficiency are seen as worse than teachers whose students go from an 80 percent proficiency to a 70 percent proficiency (to use made up numbers).

Finally, one constant that's been reaffirmed in study after study is that decreasing class sizes betters educational outcomes. But that's relatively expensive.
posted by klangklangston at 4:23 PM on March 7, 2011 [1 favorite]

jsturgill: "l33tpolicywonk, I haven't read the paper, but are you saying that poor teachers do fine in rich schools? That given a better environment, even bad teachers do okay? Because that would be awesome to hear. "

The general narrative explaining this phenomenon is that, given no other incentives, the teachers who can go to better schools. This paper would (rather interestingly) suggest that many great teachers stay at poor schools, many great teachers stay at / move to rich schools, and of the good-to-mediocre teachers, the better ones go to rich schools and leave the remainder behind. Of course, lots leave the profession, too. Making this incentive structure better through both monetary and non-monetary means is an interest I'm going into PhD work to study, so I'm happy to chat more via e-mail if you're curious. Fair warning: there be social science and econ dragons.

mandanza: "Honestly, from what I have observed, standardized testing makes school suck more for kids, and sucks the joy and fun out of teaching."

Thanks for bringing this up. This conversation is always a really difficult one to have, because everybody has experiences with education, those experiences can be really emotional and are often negative overall.

Still, I think we're making a false comparison here between the kind of great teachers that, by being rigorous, encourage positive test performance as an ancillary effect, and the kind of poor teachers who view test prep as their exclusive end. If you flip through a book like Teach Like a Champion (which comes with lots of great videos, btw) or explore the methodology of high-performing charters, I think it's evident that inspiring children to learn is much more about who you the teacher are than what you teach - it's about how you manage a classroom, the energy you bring to the craft, the attention and care you give individual children, etc. Those qualities aren't anathema to data - in fact, they benefit from a strong understanding of the correlation between your technique and student performance.

NCLB put priority on reading and math instruction because, frankly, they're prerequisites to everything else, and they're the content areas where poor kids most often get short-changed. In the absence of a national curriculum, they're also the easiest subjects to make tests for. Lots of folks from all stripes in the eduworld are getting on the national curriculum bandwagon, because they recognize how important it is that everyone understand what's essential for kids to know.

I agree with you that some schools spend too much time on test prep, and use too many prescriptive materials to do it. I think the best way to fix that problem is to make the tests harder, and reveal less about their content. If you reveal what the tests measure, but leave the content of the tests secret, only students with breadth & depth will do well on them. That creates the right incentives in classrooms.

klangklangston: "Finally, one constant that's been reaffirmed in study after study is that decreasing class sizes betters educational outcomes. But that's relatively expensive."

I'd urge some caution here. Class size studies generally assume a) that class size is being reduced by a lot and b) that students being put in a smaller class have the same quality teacher. On a large scale, class size reduction not only means lots of money, it means hiring more people from a pool whose quality has not been raised to compensate. More on this from Andy Rotherham.
posted by l33tpolicywonk at 4:47 PM on March 7, 2011

There's a city not far from Portland, Maine called Lewiston. Due to the slings and arrows of outrageous fortune, Lewiston attracted a huge number of African refugees, Sudan and Somalia primarily. Do you think an old mill-town like Lewiston—already hurting from the closing of the mills and in a kind-of permanent economic depression—do you think this city can just go hire a couple new teachers and call it a day? Hell no. This isn't like Texas, where 99% of the time the only other "non-English" language that your school system will have to support is Spanish. Sudan and Somalia have dozens of languages, some specific to just certain villages. You think a town already hurting can afford a couple dozen ESL teachers in all the various subjects and grades?

So what happens is, the kids do poorly on their exams, and thanks to No Child Left Behind, the school system is penalized for underperforming. So, because they can't afford to hire all the teachers they need, their budgets shall be penalized, forcing them to hire less teachers!

Brilliant!
posted by Civil_Disobedient at 4:48 PM on March 7, 2011 [3 favorites]

We have another weird thing that happens in rural VT. A lot of the NCLB standards are based on percentages. So if X percentage of your kids do Y badly then the whole school gets in some sort of trouble. So in a typical classroom with a typical [20-30] students, a percentage like this is at least, you know, more than one or two kids. In smaller rural schools, which may have excellent teachers, if they have one struggling kid in a five or six kid classroom, that kid can tip the percentages in the "you fail" direction. This is lousy for the kid and lousy for the school which may be otherwise doing okay. The system as-is sort of rewards getting rid of kids who have real trouble whether it's because of low English ability or whatever else. People get stressed about the tests and can't focus on the other education and social assistance these kids need.
posted by jessamyn at 4:53 PM on March 7, 2011

Civil_Disobedient, I was under the impression that, under No Child, ESL had largely been eliminated, and that ESL students had pretty much been put into mainstream classrooms. I heard this about a friend of a friend who'd gotten a MATESOL, and went back to the States after working in Japan. Since she was an ESL teacher, she was essentially given all of the ESL students, but then told to follow the regular curriculum (study for the test). Not surprisingly, her classes performed the worst, and she was singled out for her student's low test scores.

Maybe I misheard, but this story was another brick in the wall of why I don't really want to go back home and become a teacher, which kinda sorta used to be my dream in life.
posted by Ghidorah at 4:54 PM on March 7, 2011

My mother works at an inner-city school. One day the state testing people came by to brief everyone on protocol. They told them this: "If there is a fire in the school, make sure to collect the test booklets before exiting the building".
posted by GilloD at 4:59 PM on March 7, 2011 [1 favorite]

Also, I used to work for one of these bastards whose name shall not go left unmentioned. They were called Measured Progress and they were based out of Dover, New Hampshire. At the time I believe they were the principal test writers and scorers for the Maine and Massachusetts State Exams (as well as a few other municipalities across the U.S.).

If truth is an affirmative defense against slander let me just say that these fuckers were fucking evil. Evil. They had, like, 10 permanent employees and a staff of something like 200 that were in a constant state of temp-dom (i.e., no-benefit-dom). The tests would come in throughout the year but there were a couple of big seasons where they'd hire a couple hundred temps just for grading. The grading process was relentless. There was a lot of pressure put on hires to churn through as many test booklets as possible. You'd have teams of about 10 with one overseer that made sure everyone was grading "correctly." They gave all the new hires a crash-course in how to score tests but so much depended on the personality of the team leader. For some it was just, "Did he say this phrase? Yes. Did he use this word? Yes. Check, check, check, tally the points." Doesn't matter if the writing was in Iambic Hexameter, if you didn't have X, Y and Z in your answer, WRONG.

And if you didn't make your quota, you didn't get the call for the next round of scoring. There are a lot of English substitute teachers and Philosophy majors in the NH/ME region that could tell you some horror stories about Measured Progress.
posted by Civil_Disobedient at 5:07 PM on March 7, 2011 [1 favorite]

I worked for a different test corrections outfit about 6-7 years ago at a temporary Midwest facility. It was interesting in sobering and depressing ways. They spent the bare minimum to set it up. We were lucky to have AC. Cheapest possible furniture rented. Pretty much get in, get out.

There were an interesting cross section of people who took the work - some hard on their luck, some biding their time until graduate school started. Many had pretty strong work backgrounds, but they were unlucky enough to end up in the wrong place in their careers. All of us were temps, except possibly the security guard that came with the rental.

As for scoring the essays, they were soul crushing no matter what the perspective. There was intense pressure to maintain the pace and to score within the specified rubric. Deviations were found and pounded out. Heck, a few people washed out of training, which must have been soul crushing.
posted by ZeusHumms at 5:13 PM on March 7, 2011

I've been making this argument for years. It's perfectly true that any absolute performance standard you set is going to be arbitrary and students who start out with a learning deficit cannot have any realistic hope of meeting the prescribed threshold of competence.

The way to deal with that sort of problem is to measure relative improvement. If 80% of children attending a particular school graduate with a C (so to speak), something is clearly amiss in the overall system of school and community. But it could be a sign of enormous progress; graduating with a C is much better than graduating with a D or and F, and an 80% graduation rate is much better than having 40% of students drop out. If we can show a trend of improvement, then it's easier to make the case for further investment of financial, political, or human capital when we're trying to deal with policy.

I don't like the right-wing idea that public education should be dismantled or that schools should teach intelligent design or (insert your least favorite RW trope here). But while the right has (mis)used the whole idea of standardized testing to undercut public education, rejecting all testing isn't any better. There has to be some way of objectively measuring a student's progress as they go through the k-12 system, and refusing to countenance any kind of testing just leads people to suspect something is being hidden or ignored. And while it's harder to explain relative improvement than meeting or exceeding some absolute standard...that too is a teaching challenge. Surely teachers of economics, social science, and math could come up with ways to, well, educate the general public about how to interpret the results, or how to spot the flaw in an anti-public-education argument. If the public doesn't respond, rethink the presentation of the lesson.
posted by anigbrowl at 5:19 PM on March 7, 2011 [2 favorites]

Absolutely, C_D!

The same is true here in San Diego where huge numbers of our students are "new Americans", refugees and immigrants from across the globe. How is it that those kids can be included in standardized testing?

What I've witnessed that is sad beyond words is how much our teachers struggle and make personal sacrifices to help these families and at the same time be under the gun by uncle sam for poor performance based on very poorly devised testing. It's just wrong. I'm talking about helping with food, lodging, transportation, medicine on TOP of struggling to teach basic skills AND prep for standardized tests. (Of course it's not their "job" to do those things; they are done because it's vital to the education of the kids.)

I may being going too far saying that NCLB is racist or classist, but it's certainly true that it and standardized testing totally undermine and negate the efforts of educators across the country.
posted by snsranch at 5:31 PM on March 7, 2011

There has to be some way of objectively measuring a student's progress as they go through the k-12 system...

Unless there isn't. I don't know how much tests have improved in the thirty years since I left the public education system; but back then, standardized tests mostly measured the ability to take standardized tests (which I was very good at, by the way). Throw in the sort of implementation horror tales we're seeing all through this thread, and you have a recipe for making education reformers feel good without accomplishing much of anything.

Even if the tests were perfect, there would still be factors they didn't measure. If a learning-disabled kid with an abusive father and a drug-addicted mother has a safe haven at school, she's likely to learn more and do better in the world, even if she never masters the art of testing. On the flip side, a gifted child who rips each test a new one without effort is likely to be bored in an environment where testing is the primary focus, and might not learn nearly as much as she could.

I'm not saying tests can't be useful. They might even be essential. But emphasizing them at the expense of everything else involved in education is a bad idea; and relying on them even when the system is demonstrably broken is just nuts.
posted by steambadger at 6:27 PM on March 7, 2011

A cynic would say that these tests aren't about improving the system, but more about providing excuses to gut the system, avoid any kind of systemic improvements, and freeing money to subsidize private education.
posted by ZeusHumms at 6:48 PM on March 7, 2011

l33tpolicywonk: "NCLB put priority on reading and math instruction because, frankly, they're prerequisites to everything else, and they're the content areas where poor kids most often get short-changed. "

No no no. Reading and math, as tested, are not taught as content areas, or tested as content areas, but as skills, and they are treated as skills that are, as you say, pre-requisites for everything else. This is patently not true. Reading is not a skill

My seven-year-old twins have learned a ton of science without knowing much math at all.

We will continue to short-change the poor kids, if we drill them earlier and earlier, and include more and more test prep, treating reading and math as skill, while rich kids get to enjoy reading diverse content.

Another quote:
"I agree with you that some schools spend too much time on test prep, and use too many prescriptive materials to do it. I think the best way to fix that problem is to make the tests harder, and reveal less about their content. If you reveal what the tests measure, but leave the content of the tests secret, only students with breadth & depth will do well on them. That creates the right incentives in classrooms."

What does "harder" mean? The more incentive you put on the test, the more pressure that will be felt down the line to improve performance on the test, by any means necessary. My take on that is that, yes, you will get cheating, but also, you will get teachers (and principals) guessing what will be on the test, or using what little they know about what will be on the test, and drilling that. They will make up skills that don't exist, just so that they can feel like they are doing their best job to teach to the test. This is pretty close to what currently happens with reading comprehension skills.

I am on board with not throwing up our hands at the big impact of poverty. But this is a straw man of the education activists on the other side. No one is saying that poor kids can't learn. What people are saying is that poverty matters, and that we should take that into account with our policies. You responded to the poster who brought up lead poisoning with :
"Do I think water supplies shouldn't have lead in them? Yes. Do I tend to vote for politicians who give a damn about issues like pollution in general and the pollution that harms poor people specifically? Absolutely. Does that prevent poor kids from being taught effectively and lifted out of poverty? I don't think so, and I don't think so because there are individual schools in many communities which are doing amazing work getting these kids to college - many of them despite funding challenges, relying on inexperienced teachers or TFA teachers, etc. All of this leads me to believe that money + will + good ideas can institutionalize best practices."

I just can't believe that there are plenty of examples of schools who do amazing work despite having the same funding, same students and same expectations as public schools. I have read fairly widely (and I appreciate your links) and most of the cases I see of KIPP, Green Dot, SEED, etc, either have far more resources or selective attrition. I have been convinced by some of the lottery studies (Hoxby et al) that NYC charters may be doing better (on even the good tests like the NAEP) than relatively more of their public counterparts, but it is not clear why (my money is on the fact that they spend way more time in school).
But when I see the pressure to institutionalize best practices, mostly what I see is shortcuts and policymakers who argue in bad faith, and try to identify "scalable" transformations where none exist.
I wish you luck in your Ph.D. on using monetary (and other) incentives to shape teacher behavior. I hope that you read some psychologists (Dweck: Mindset, Nisbett: Intelligence and How to Get It, Stanovich: What Intelligence Tests Miss, Willingham: Why Don't Students Like School?) and not just economists like Hanushek. The social science and econ dragons are often there because they are actually really complicated issues with massively intercorrelated data, long-term interactions, and no good general answers, not because they are waiting for the right economist with the golden lance.
posted by cogpsychprof at 6:53 PM on March 7, 2011 [1 favorite]

But denying the power of schools to create change (or, like Saul, saying there are so many special cases it's impossible to generalize solutions) is like denying the power of the EPA to get the damn lead out of the water because there are so many companies to regulate all at once: both are a form of policy nihilism that, frankly, isn't backed up very well by history.

For my part, I meant specifically, there are often too many special cases in the data to make analysis of the data straightforward. I did not mean to generalize to the claim that there are so many special cases of problems in education that we can't solve them through intentional effort, although I can see how the context could give that impression. That's the opposite of what I believe, actually. I just think we have to combat the tendency to be lazy about how we go about doing it. Address each special case in turn, don't generalize. Don't panic, and don't just use analysis to confirm whatever our current preferred prejudices are (and then use that confirmation as the basis for even more ham-fisted policy reforms that disrupt more than they do good). I know it's so much easier to say than do, but that's what we need to do IMO.

Finally, one constant that's been reaffirmed in study after study is that decreasing class sizes betters educational outcomes. But that's relatively expensive.

Here in Florida, we passed a constitutional amendment to lower class sizes several years back. Ever since, our elected officials have completely stonewalled and used every trick in the book to avoid implementing it, and Rick Scott's latest budget proposals now effectively makes achieving compliance with the constitution impossible anytime soon. If our state constitution is effectively the contract between the state government and the people, then Floridians I think have a very good case for viewing our state in breach of contract.
posted by saulgoodman at 8:41 PM on March 7, 2011 [1 favorite]

So, because they can't afford to hire all the teachers they need, their budgets shall be penalized, forcing them to hire less teachers!

It's "fewer teachers."

But then, you probably got a crappy American education, so it's not really your fault. My wife would kick me if I let this slide, though.

In smaller rural schools, which may have excellent teachers, if they have one struggling kid in a five or six kid classroom, that kid can tip the percentages in the "you fail" direction.

This issue gets overlooked with the VPK programs I work with, too. I've groused about it endlessly. I'm not even a statistician and I know you've got to weight the kids' scores differently depending on the total number of kids in a class. If a provider with four kids splits 50/50 into passing and failing groups, that's not nearly as solid an indicator of their performance as a 50/50 split in a class of 20 kids (especially if the two kids in the first case happened to be twins whose parents died in a car crash a few days before the assessments were administered or whatever might have happened in the real world to cast those feeble little shadows of reality that remain in the data). The two rates in raw percentage terms are the same, but they are not equally meaningful metrics.

Legislative bodies are apparently oblivious to such fine-grained distinctions. And unfortunately, it's apparently politically impossible in Florida to leave such technical details up to technocratic rule-making bodies anymore either considering the new governor declared a freeze on all rule-making in the state subject to his personal review and approval of every new rule to make sure it's business friendly enough.
posted by saulgoodman at 9:11 PM on March 7, 2011

(from Reading is not a skill)

Can’t you teach kids how to reason about texts, and thereby wring the meaning out of it even if they don’t have the right prior knowledge?

To some extent, but it doesn’t seem to help as much as you might expect. For one thing, this sort of reasoning is difficult mental work. For another, it’s slow, and so it breaks up the flow of the story you’re reading, and the fun of the story is lost. Hoping that students without relevant prior knowledge will reason their way through a story is a recipe for creating a student who doesn’t like reading.

That's why we have dictionaries, and why they are such an empowering tool for a curious mind. I can't believe the Washington Post is providing a platform for this asinine drivel.
posted by anigbrowl at 12:43 AM on March 8, 2011 [1 favorite]

I can speak about the testing w/r/t math. I taught in Washington and Oregon, in suburban and rural schools. [edited to take out lost of unnecessary backstory. Upshot: I've left teaching, this year.] Things may have changed, but I doubt by much.

In Washington, the tests ARE graded by teachers. Teachers can volunteer to go away for a week in the summer (and get paid, a decent hourly rate last I checked) and grade the state's tests. In Oregon, I have no idea how they are graded anymore, it's been years since I taught there, but when I did, half of the test was on a computer and half was a series of monthly open-ended problems. Monthly? Yes, monthly.

So what's the real impact of these tests? Kids don't take them seriously unless they have a cost so: now they need it to graduate. Aside from the time testing takes from actual learning (my Washington school took the test seriously: for two weeks in March and a week in April, we essentially shut down, just for the testing--oh, and students who weren't testing didn't have any class at all because all of the teachers were proctoring. Yes, ALL.), you have to take time out of the REST of your curriculum to make sure you cover other odds and ends that are on the test but not in your curriculum. Why? This is high-stakes testing at its finest, y'all. Those kids were worried, especially when my sophomores who, for whatever reason, were repeating freshman algebra (moved halfway through the year... or they don't get food at home, and who can learn like that?... or maybe their brains were just not biologically mature enough to get variables the previous year) were going to take a test that was geared to sophomores who were already 2/3 of the way through Geometry. So, because they were an immature freshman, they are being shoved into a test that they have to take anyway but they were guaranteed to fail? Huh. And they need this to graduate? Way to take kids who already were beginning to check out of the educational process and push them right out the door. For scale: 1/3 of my "freshman" classes were typically sophomores.

But then there's also the writers of the test. I can't speak to other states, of course, but the content of the Washington test is a mile wide, an inch deep. There is no way you can cover everything on the test by the time the kids have to take it unless they had come into high school ALREADY ABOVE grade level. And certainly no way to cover it in depth, giving them time to puzzle over the whys and wherefores, to see problems (and by problems I mean situations, not just problems 30-34) in a different light, from a different angle.

I doubt most of the state representatives mandating the test could have passed it.

Sophomores needing to know trig functions meant we never actually got to teach proofs. Look, I can hear non-mathy groans now, but proofs are one area where you learn to THINK, to PROCESS, not just to churn and press buttons on your calculators, and it cannot be rushed--but of course, they can't really be tested on a standardized test so... In fact, any opportunity we had for that kind of learning--to just sit and discuss and ask WHY--was pushed out. Not by us, the teachers--but by the parents ("What are you doing to prepare my kid for the state tests?") and the administrators ("What kind of practice testing are you doing?" "How does this align with the state tests?") and the district office ("you need to take a day out of your class in the next 10 days to do this district wide assessment." "You need to get a sub from your classroom so you can come grade this district wide assessment.")

So what was the test testing? Really, I mean. Test-taking, sure--the ability to avoid getting rattled when something came up that you didn't know so that you could go on to a problem you did know. Question patterning, probably. Expectation management, definitely.

Was it testing what I can do as a teacher? HELL no. Because I had less and less choice as a teacher.

We were one of the better schools in the state, and had a less-than-3/4 passing rate.

Contrast this with my AP Statistics classroom. I was given a clear, focused set of guidelines. I was able to take time on discussion, experimentation, on why and how and when. Yeah, I taught to the test--but always with the idea that if they don't understand the foundational concepts, all the teaching to the test I wanted to do wouldn't matter. I had a pass rate consistently above 80%. The national average is much, much lower. These weren't typical "AP kids"--more than half of my Stats students didn't take a single other AP class. Of course, that test is written by stats teachers (both HS and University), graded by stats teachers, half multiple choice, half free response, and (here's the kicker) really expensive. And though Stats students are, in general, students who would never make it in Calculus, they tend to be older, they tend to get enough to eat, they tend to have support at home, they tend to have a goal in mind, and they tend to be well trained in How To Succeed in School.

HARDER tests are not the answer. If you're going to do testing, and I can only speak to the high school level and in math, but here's what you need:
--an even playing field for kids coming into high school. Seriously, there needs to be pull-out math remediation in grade- and middle-school like there is for reading. No freshman should be counting on their fingers still. And yet.
--an even playing field for kids coming to SCHOOL. I was in a relatively wealthy suburban district, but in my freshman classes, I had kids not getting enough to eat--if it weren't for school lunches, they wouldn't have eaten much at all. I had kids floating in and out of not just my classroom but my school and my school district with the vagaries of migrant work, foster system, custody disputes, jobs, and other family issues. Anywhere from 10-20% of the class. How can I test them on what they learned in California, Massachusetts, Texas, Oregon, the next school district over? And yet.
--a better test that doesn't take four days, that doesn't reduce math to skills, that is aligned to where students are (are they learning what we think they are?) than where some mythical legislator thinks they should be.

Sigh. Longer than I meant it to be.
posted by e to the pi i at 1:30 AM on March 8, 2011 [1 favorite]

That was most interesting. Monthly state testing is absurd, I agree.
posted by anigbrowl at 2:17 AM on March 8, 2011

It's "fewer teachers."

Do small people have fewer height? Do anorexics have fewer weight? Do amputees have fewer limb?
posted by Civil_Disobedient at 4:54 AM on March 8, 2011 [1 favorite]

Fewer vs. Less.
posted by hippybear at 5:03 AM on March 8, 2011

Kidding vs. Serious
posted by Civil_Disobedient at 5:06 AM on March 8, 2011 [1 favorite]

To be clear, I'm not nihilistic, as apparently accused, about the need to make changes IN the educational system, and I have repeatedly supported, in my capacity as a board member, onerous processes to remove bad teachers, get rid of bad practices, etc.

HOWEVER. If you look at a school where 1/3 of the students are special ed (with some portion due to lead paint), and 99% of them are receiving free breakfast and free lunch and a significant number of those students don't EAT on weekends, and when we had to close for four snow days in a row we fretted ourselves sick over whether it was better to TRY to have school and force students to slog through dangerous snow or whether to leave them without food because they wouldn't get any at home, and any number of those students have parents who were teenagers themselves, who have no education, who are illiterate, who suffer from housing instability, parental partner abuse, drugs, violence, gangs, and the many other scourges of the inner-city impoverished, and you look at this school where 1/3 of the students are special ed and all of them are impoverished and you say, "The reason they are not performing up to the level of their regular-division wealthy peers on the other side of town must be because the teachers are bad," and you thereupon fire 50% of the teaching staff, or close the neighborhood school, or take one of NCLB's other solutions, but in any case penalize the school for having a student body with a lot of problems that are OUTSIDE the school, you are not just rearranging the deck chairs on the Titanic, you are shooting the rescue workers on the theory that it's somehow their fault the ship is sinking.

YES, those schools need our BEST teachers, and yes, good teachers can make a real difference in how much impoverished students learn in a year. But to pretend that the beginning and end of fixing education in the U.S. is through education reform is either appallingly ignorant or it's deliberately misleading. An educational system that truly leaves no child behind is going to require broad-based and strong anti-poverty programs, period.

You can, and we certainly try to, create an education system that does the best we can with the constraints we have. But it's foolish to pretend that a better teacher can ameliorate the effects of lead poisoning. (And that's not from the water, l33t, that's from the poor housing stock that SHOULD be remediated under state programs to fix exactly this problem but the state can't be arsed to pay for it. Although the water's not helping, we have old pipes.)

The current system looks at these failing schools with impoverished students and says, "You have tons of problems! We will provide you no resources to cope with them, and we will repeatedly penalize you for failing to perform as well as wealthy districts!" And the remediation money that we do get is earmarked for specific things -- we may not be able to afford a reading coach, or smaller classes, or longer school days, but we can afford computers for every student! I mean, we get so many Title I technology grants we're running out of stuff to buy. But there's not the funding for more teachers, or teacher materials that the teachers want (crayons! books!), and on the rare occasions funding can be applied to staff, it's typically for a single year (rarely for 3 and always with strings) and then the funding stream is gone and that position has to disappear too. The incentives are entirely misaligned, and the proposed solutions were certainly not created by educators.
posted by Eyebrows McGee at 6:09 AM on March 8, 2011 [1 favorite]

Eyebrows McGee: "YES, those schools need our BEST teachers, and yes, good teachers can make a real difference in how much impoverished students learn in a year. But to pretend that the beginning and end of fixing education in the U.S. is through education reform is either appallingly ignorant or it's deliberately misleading. An educational system that truly leaves no child behind is going to require broad-based and strong anti-poverty programs, period."

I appreciate the efforts you're making, Eyebrows, but the graph above leads pretty inevitably to this question: if schools can't be the primary agent of changing child poverty, then why invest in this debate at all? Put more directly: if the lead in the pipes is the first and tallest barrier to student achievement, why are we send poor kids to school at all?

You can argue the incentives in education suck (and I agree), you can argue that politicians and school leaders deserve a greater share of accountability here (and I agree), you can argue that NCLB didn't design either of the two entirely well (and I agree), but what I find difficult to stomach where ever I see it is the argument that poverty keeps poor kids from learning. It always strikes me as comparable to a hospital arguing they can't be held responsible for their morbidity rate because they have to treat so many sick people.

In the aggregate, while poverty is an important element of student performance, and while poverty is generally correlated with kids getting a crappy education, the data is showing time and time again that poverty does not prevent kids from getting a quality education. Look at DC, where scores went up from 07 to 09 while poverty did too. Look at how well KIPP does at closing the achievement gap, working exclusively with the kinds of students in the kinds of neighborhoods you describe. Most importantly, look at all the evidence I linked upthread about how different the impact of teachers can be within the same school.

The present system doesn't work. Inter-district choice or busing doesn't ultimately work at scale because there are too many poor kids, and not enough quality schools to put them in. The only way, therefore, to make the education problem better is to produce more quality schools with more quality teachers. Education is, in short, a necessary but not sufficient part of the long-term solution to poverty. Not the only solution, certainly not the easiest solution, but a big part of what we have to fix.
posted by l33tpolicywonk at 7:07 AM on March 8, 2011 [1 favorite]

Diane Ravitch (yes, that one), in 1999, on tests and cheating: To say that tests create cheating is wrong. What creates cheating is people who cheat. If we spent as much time teaching kids as showing them the answer, they might have learned to read.
posted by l33tpolicywonk at 7:13 AM on March 8, 2011

l33t, it's intensely frustrating that you're not reading what I'm writing. I said that yes, ABSOLUTELY, teachers make a difference and that I have been committed, as a board member, to making those changes, but that it is unrealistic to PUNISH schools for not solving problems outside the purview of schools -- in other words, that education is a necessary but not sufficient part of the long-term solution to poverty. My beef is that insisting education fix problems that education cannot fix, and then PENALIZING SCHOOLS FOR FAILING TO DO SO, in ways that make those schools worse, is both stupid and counterproductive. And pretending that "better teachers" will fix ALL our problems is disingenuous, and likely to lead to more of the current situation: Magical solution fails to fix all problems! Completely bail on magical solution! Run madly in a different direction that will also fail to fix all problems! Bail on that solution! Repeat, while fixing nothing but creating vast upheaval and wasting taxpayer money!

And you come back and berate me AGAIN for not wanting to fix schools because as long as there's lead IN THE PAINT, NOT THE PIPES (to clarify again, because you're not reading what I'm writing), education does no good, well, I'm not really sure why I'm still in this discussion, since whatever you're reading, it's not what I'm writing.
posted by Eyebrows McGee at 7:30 AM on March 8, 2011 [1 favorite]

"but what I find difficult to stomach where ever I see it is the argument that poverty keeps poor kids from learning"

PS -- I didn't actually say "poverty" keeps kids from learning. I said HUNGER keeps kids from learning, which is true. Hungry children can't concentrate. I said lead poisoning puts kids in special ed, which is true (and lead poisoning is more common in impoverished areas with older, unremediated housing stock). I said housing instability impacts student learning, which is true. Parental abuse impacts student learning, which is true. It's not "poverty" per se; it's all of these social problems that are often associated with poverty, many of which can be solved or reduced. Pretending that a student who comes to school on Monday without having eaten all weekend is in the same situation as a student with adequate nutrition is foolish, blind, and frankly cruel. THAT KID NEEDS FOOD. That kid needs education, but that kid needs food. To, as a society, say, "All that kid's problems would be solved if she just had a better teacher" is bottomlessly cruel. THAT KID NEEDS FOOD. Poor nutrition impacts not only concentration, but brain development. She could be suffering permanent, irreversible damage. And if she has food? She will perform better in the classroom. She can have the greatest teacher in America but if she's starving, she will not perform up to her potential. Period.

And, honest to God, ASK teachers in impoverished schools what would truly make their lives easier and they will tell you: Kids who aren't hungry. Kids who have access to health care. Kids who don't move every three months. You can't ask an 8-year-old to come into a classroom and pretend he's not hungry. For Christ's sake, if education is about the actual children in the schools, we need to help those actual children, not some hypothetical statistical child. And the actual children in my schools need FOOD. I'm unclear on what kind of monster looks at a hungry child and says, "Hunger isn't a problem here."

Now, if education is about making numbers happy, then, no, hungry, abused children are not a problem as long as they make their numbers. But if you are in an actual school with actual children? You CANNOT pretend that their problems begin and end at the schoolhouse door. And that's true in wealthy communities as well, where parental abuse, substance abuse, sexual assault, and other major impacts on student learning are problems as well.
posted by Eyebrows McGee at 7:40 AM on March 8, 2011 [2 favorites]

You can't expect us to feed the poor, Eyebrows McGee. They're like stray animals.
posted by sonic meat machine at 8:08 AM on March 8, 2011

l33tpolicywonk: See, here's the thing--and I'm speaking from an engineer's perspective when I say this: As a general rule, when you go designing a system, it's pretty stupid to design that system to be dependent for its routine functioning on the at-will performance of miracles. And I get the sense, that's basically what you're arguing for: Redesign the entire education system to depend solely on those rare, miracle-worker teachers we've all heard touching stories about who seem to have that magic touch for changing kids' lives.

If this were a perfect world, where such teaching geniuses were and forever would be commonplace (rather than one where they are exceedingly rare), you might just be able to do this, but in the real world, you have to accept that not only won't you ever achieve an entire labor force of perfect geniuses, you will even find you have to accept that the vast majority of your workforce, through no deliberate fault of their own, will tend to look mediocre when compared to their highest achieving colleagues or when compared to some mythical ideal teacher.

You can't design systems that ignore basic realities like this and not expect them to fail.
posted by saulgoodman at 9:41 AM on March 8, 2011 [1 favorite]

What Saulgoodman said. It seems a bit silly, to put it mildly, to predicate success on having exceptional teachers across the board, in every class in a failing school system.

It also seems odd to attempt to resolve problems in the classroom by devoting extra resources to administration rather than providing greater funding the actual field workers who get things done. I mainly view administration as a reactionary, self-serving hinderence to education, not a facilitator of it.

Having data to guide the decision making process is good, but I feel pretty hesitant to say that this sort of testing provides quality data. And if the data is no good, it might very well be worse than no data.

I wonder if making schools into schools + community centers in struggling districts, open late and on weekends, charging no fees for use, and providing quality extra curricular activities and free food, might not be a much better use of taxpayer money than subsidizing companies who make and grade standardized tests.
posted by jsturgill at 10:37 AM on March 8, 2011

l33tpolicywonk:
"In the aggregate, while poverty is an important element of student performance, and while poverty is generally correlated with kids getting a crappy education, the data is showing time and time again that poverty does not prevent kids from getting a quality education. Look at DC, where scores went up from 07 to 09 while poverty did too"

No one argues that poverty prevents kids from learning. What people do argue is that it has a large impact, and that this impact has been ignored when planning large scale education reform. The straw man of "blaming poverty" is pretty well trashed now. There is a difference between "blaming poverty" and acknowledging it, and compensating for it.

Also, your citation of DC scores going up from 07 to 09 (and the citation which gives Rhee credit for this rise: "In other words, Michelle Rhee said poverty was no excuse for her own performance, and then delivered.") while poverty also did is disingenuous. Understanding the NAEP scores is tricky stuff. There are a lot of demographic changes, and long term changes that have nothing to do with Rhee. If you drill down into the DC scores you can also see that while the city population as a whole may be experiencing a rise in poverty, once you get into the particular demographics of the schools, this isn't always as straightforward.
I am sure you agree that NAEP scores are tricky, but then why do you cite this simplistic account of DC scores going up while poverty does too? Oh, right, because the straw man that you are shadow boxing says that poverty is the only thing that matters.
posted by cogpsychprof at 10:44 AM on March 8, 2011

I'm gonna bow out after this, lest I completely take over the thread, but let me try to sum up: we seem to all agree that schools for the urban poor aren't educating kids well. In fact, we seem to all agree that some teachers are better than others: in fact, that some teachers are dramatically better than others. What's left isn't an argument between one of several alternatives, it's an argument about whether it is worthwhile to undertake systematic efforts to improve teacher quality by assessing the performance of teachers using, in part, value added data which mathematically compensates for socioeconomic differences between students.

We have, in the US, a teaching system which has a wildly low percentage of top academic performers compared to other countries, uses little to no objective data to assess quality and makes hiring and firing decisions based almost exclusively on seniority rather than quality. Is it more likely the quality of that corps can never, ever increase or more likely there's something we can do about it?
posted by l33tpolicywonk at 10:48 AM on March 8, 2011

Oh, and I am happy to beat up on Diane Ravitch's mistaken conclusions from twenty years ago. But if you simply plan to dismiss her points as wrong because of who she is, rather than the substance of the argument, then I can't get on board. She is not a god, and most of us doubters of the current wave of education reform are not blind followers. I don't like ad hominem attacks of Gates/Broad or Walton either, I am much more persuaded by specific consideration of the specific effects of their policies.
posted by cogpsychprof at 10:48 AM on March 8, 2011

You are assuming that good teaching as a stable trait explains more of the variation in educational quality than good teaching practices and resources for training and support.

This is an extreme view of the person vs. environment debate. The data does not support this extreme view, but the money has.

I have always found extreme arguments of this type, talking about "lower quality people." (what else is "quality of the corps" supposed to mean?) distasteful.

What alternative do I propose? Aim to improve teaching quality, not teacher quality. Provide support, broad and interesting curricula, resources, smaller class sizes, and social and community help for parents to get involved.

You know, the stuff that people who have the money to opt out of our system have been choosing for their own children for decades.
posted by cogpsychprof at 11:01 AM on March 8, 2011 [1 favorite]

« Older "A flamingo falls on them from above" | Meet the Geminoid DK. Newer »

This thread has been archived and is closed to new comments

MetaFilter

Inside the multimillion-dollar essay-scoring business
March 7, 2011 8:56 AM Subscribe

Tags

Share

Inside the multimillion-dollar essay-scoring business March 7, 2011 8:56 AM Subscribe

Tags

Share

Inside the multimillion-dollar essay-scoring business
March 7, 2011 8:56 AM Subscribe