The Worst 8th Grade Math Teacher In New York City
May 19, 2012 6:57 AM Subscribe
Would you want Carolyn Abbott, the worst 8th grade math teacher in New York City, to teach your kids? Probably.
According to the Teacher Data Report, Carolyn Abbott, who teaches at the gifted and talented Anderson School, is the worst math teacher in the city. Her accelerated students score in the 98th percentile on the state's math test in the 7th grade, but drop their scores to the 89th percentile in the 8th grade. (More than one-third of Abbott's little slackers do score a perfect 100 on their Regents exam, however - a metric that is not included when Abbott's own score is calculated.)
According to the Teacher Data Report, Carolyn Abbott, who teaches at the gifted and talented Anderson School, is the worst math teacher in the city. Her accelerated students score in the 98th percentile on the state's math test in the 7th grade, but drop their scores to the 89th percentile in the 8th grade. (More than one-third of Abbott's little slackers do score a perfect 100 on their Regents exam, however - a metric that is not included when Abbott's own score is calculated.)
She has decided to leave the classroom, and is entering the Ph.D. program in mathematics at the University of Wisconsin-Madison this fall.
That'll show...them?
posted by Gator at 7:07 AM on May 19, 2012 [2 favorites]
That'll show...them?
posted by Gator at 7:07 AM on May 19, 2012 [2 favorites]
Spoiler: she has since left teaching in NYC.
But yeah, our system is screwy.
posted by roomthreeseventeen at 7:09 AM on May 19, 2012
But yeah, our system is screwy.
posted by roomthreeseventeen at 7:09 AM on May 19, 2012
That's one of the weirdest metrics I've seen for a long while. Makes no sense at all.
posted by arcticseal at 7:10 AM on May 19, 2012
posted by arcticseal at 7:10 AM on May 19, 2012
So she teaches the hell out of them in 7th grade, they score at the 98th percentile, then go to 8th grade, where the teacher is pretty good and teaches them enough to score at the 89th percentile, and she's declared the worst teacher in New York City. That's a hell of a system.
Solution: Don't teach your kids anything, let them score somewhere above the 50th percentile (it's a GATE school, after all), then in 8th grade they score at the 89th again and YOU'RE THE BEST TEACHER IN NEW YORK CITY!
posted by Huck500 at 7:12 AM on May 19, 2012 [3 favorites]
Solution: Don't teach your kids anything, let them score somewhere above the 50th percentile (it's a GATE school, after all), then in 8th grade they score at the 89th again and YOU'RE THE BEST TEACHER IN NEW YORK CITY!
posted by Huck500 at 7:12 AM on May 19, 2012 [3 favorites]
Or I guess she teaches both grades, but THE PLAN WILL STILL WORK! It'll be easier, actually.
posted by Huck500 at 7:13 AM on May 19, 2012 [1 favorite]
posted by Huck500 at 7:13 AM on May 19, 2012 [1 favorite]
So she teaches the hell out of them in 7th grade, they score at the 98th percentile, then go to 8th grade, where the teacher is pretty good and teaches them enough to score at the 89th percentile, and she's declared the worst teacher in New York City. That's a hell of a system.
My understanding is that she teaches them for both 7th and 8th grade. The difference is that by the time they sit down to take the exam again in the 8th grade, they have all already been admitted to the ultra-competitive highschools each one will attend, based in part on their 7th grade scores. They know the exam has no consequence for them, so they slack. On the Regents, which I assume they do care about for whatever reason, they excel.
posted by DarlingBri at 7:17 AM on May 19, 2012 [14 favorites]
My understanding is that she teaches them for both 7th and 8th grade. The difference is that by the time they sit down to take the exam again in the 8th grade, they have all already been admitted to the ultra-competitive highschools each one will attend, based in part on their 7th grade scores. They know the exam has no consequence for them, so they slack. On the Regents, which I assume they do care about for whatever reason, they excel.
posted by DarlingBri at 7:17 AM on May 19, 2012 [14 favorites]
scoring teachers based on the achievements of their students is much like your salary being determined by the performance of the guy down the line that you pass the widgets along to.
posted by HuronBob at 7:44 AM on May 19, 2012 [7 favorites]
posted by HuronBob at 7:44 AM on May 19, 2012 [7 favorites]
> “The purpose of these reports is not to look at any individual score in isolation ever,” said the Education Department’s chief academic officer, Shael Polakow-Suransky. “No principal would ever make a decision on this score alone and we would never invite anyone — parents, reporters, principals, teachers — to draw a conclusion based on this score alone.”
"Which is why we're releasing the results, which name individual teachers as well as their schools, to the public. We trust that no parent anywhere, ever, will look up their child's teacher and freak out over a low rating...which incidentally may contain a margin of error ranging from 35 to 53 percent."
What a load of horseshit.
posted by The Card Cheat at 7:51 AM on May 19, 2012 [11 favorites]
"Which is why we're releasing the results, which name individual teachers as well as their schools, to the public. We trust that no parent anywhere, ever, will look up their child's teacher and freak out over a low rating...which incidentally may contain a margin of error ranging from 35 to 53 percent."
What a load of horseshit.
posted by The Card Cheat at 7:51 AM on May 19, 2012 [11 favorites]
So this metric is susceptible to ceiling effects, does not take into account students' actual grade level performance or recent curriculum, nor their motivation to perform if they're already more than capable of passing.
I have an idea - let's test all the idiots at the NY Dept of Education to see if they remember even the slightest thing from the stats class they took that one time for their masters.
posted by slow graffiti at 7:51 AM on May 19, 2012 [14 favorites]
I have an idea - let's test all the idiots at the NY Dept of Education to see if they remember even the slightest thing from the stats class they took that one time for their masters.
posted by slow graffiti at 7:51 AM on May 19, 2012 [14 favorites]
The single most damning analysis of value-added measures of teacher performance I've seen was on this issue of same-year scores within a subject. Gary Rubenstein analyzes the scores of '665 teachers in NYC who taught two different grade levels of the same subject in 2010,' finding 'the average difference between the two scores was nearly 30 points...the correlation coefficient was a miniscule .24.'
posted by persona at 7:54 AM on May 19, 2012 [7 favorites]
posted by persona at 7:54 AM on May 19, 2012 [7 favorites]
I suppose I shouldn't be surprised that people are trying to come up with a (relatively simple) numeric formula to indicate how "good" or "bad" a particular teacher is — after all, education is becoming measured more and more by numbers only — but in my defense, it's such a ridiculous idea that I feel justified in being surprised.
posted by hypotheticole at 8:07 AM on May 19, 2012 [1 favorite]
posted by hypotheticole at 8:07 AM on May 19, 2012 [1 favorite]
I'm reminded of the Space Shuttle Columbia accident. There, the managers decided the models and data showed there wasn't a problem and vetoed requests to actually look and see if the spaceship had a hole in it.
I'm also reminded of the recent Zooey Deschanel commercial for Apple. Rather than looking out the window, she instead asks her phone "Is that rain?" When the phone confirms that is raining, she goes to the window to look at the rain and says "Oh".
Technology and modeling and stats are useful, but lets not forget the basics, ya'll.
posted by Brandon Blatcher at 8:07 AM on May 19, 2012 [17 favorites]
I'm also reminded of the recent Zooey Deschanel commercial for Apple. Rather than looking out the window, she instead asks her phone "Is that rain?" When the phone confirms that is raining, she goes to the window to look at the rain and says "Oh".
Technology and modeling and stats are useful, but lets not forget the basics, ya'll.
posted by Brandon Blatcher at 8:07 AM on May 19, 2012 [17 favorites]
Which is why we're releasing the results, which name individual teachers as well as their schools, to the public.
Actually, the city official is probably being here. They know perfectly well it's a terrible idea to look at the scores in isolation, which is why they don't use isolated scores to make firing decisions. But it wasn't their choice to release the results; they're state records, and once the New York Times and other newspapers filed Freedom of Information Act requests, the matter was out of their hands.
On the other hand, this New York Times story makes an assertion I've never heard before, which is that officials inside the Education Department spurred the reporters to make the FOIA requests, presumably against the will of their superiors. That's nuts, if it's true.
posted by escabeche at 8:12 AM on May 19, 2012 [1 favorite]
Actually, the city official is probably being here. They know perfectly well it's a terrible idea to look at the scores in isolation, which is why they don't use isolated scores to make firing decisions. But it wasn't their choice to release the results; they're state records, and once the New York Times and other newspapers filed Freedom of Information Act requests, the matter was out of their hands.
On the other hand, this New York Times story makes an assertion I've never heard before, which is that officials inside the Education Department spurred the reporters to make the FOIA requests, presumably against the will of their superiors. That's nuts, if it's true.
posted by escabeche at 8:12 AM on May 19, 2012 [1 favorite]
Having lived with people in the public education field all my life and grown up seeing it from the inside I'm not yet convinced you wouldn't be better off letting your kid live in the woods and learn the ways of the wolf pack rather than put them in most schools.
posted by The Whelk at 8:14 AM on May 19, 2012 [6 favorites]
posted by The Whelk at 8:14 AM on May 19, 2012 [6 favorites]
scoring teachers based on the achievements of their students is much like your salary being determined by the performance of the guy down the line that you pass the widgets along to.
the point of the teacher rankings isn't to bring modern statistical methods of quality control to the industrial model of education, but to put pressure on the teacher's union and add to the atmosphere of crisis in education in order to push through radical action.
Actually, the city official is probably being here. They know perfectly well it's a terrible idea to look at the scores in isolation, which is why they don't use isolated scores to make firing decisions. But it wasn't their choice to release the results; they're state records, and once the New York Times and other newspapers filed Freedom of Information Act requests, the matter was out of their hands.
but the NY school officials want to have their cake and eat it too: they are saying on the one hand, these numbers are garbage and don't take them too seriously, but also, look at those 521 sucky teachers...
posted by ennui.bz at 8:18 AM on May 19, 2012 [6 favorites]
the point of the teacher rankings isn't to bring modern statistical methods of quality control to the industrial model of education, but to put pressure on the teacher's union and add to the atmosphere of crisis in education in order to push through radical action.
Actually, the city official is probably being here. They know perfectly well it's a terrible idea to look at the scores in isolation, which is why they don't use isolated scores to make firing decisions. But it wasn't their choice to release the results; they're state records, and once the New York Times and other newspapers filed Freedom of Information Act requests, the matter was out of their hands.
but the NY school officials want to have their cake and eat it too: they are saying on the one hand, these numbers are garbage and don't take them too seriously, but also, look at those 521 sucky teachers...
At a briefing on Friday morning, an Education Department official said that over the five years, 521 teachers were rated in the bottom 5 percent for two or more years, and 696 were repeatedly in the top 5 percent.basically, the numbers are garbage and they don't want anyone looking at them except when they decide to publicize them or make decisions based upon them. This is exactly what FOIA requests are for...
But citing both the wide margin of error — on average, a teacher’s math score could be 35 percentage points off, or 53 points on the English exam — as well as the limited sample size — some teachers are being judged on as few as 10 students — city education officials said their confidence in the data varied widely from case to case.
posted by ennui.bz at 8:18 AM on May 19, 2012 [6 favorites]
Since I'm inclined to be skeptical of VAM scores, and so are most people in this thread, it's worth mentioning one high-profile study that I think makes the best possible case for their use (though even the authors of this study would not, I imagine, recommend placing very much weight on a single-year individual score. They find small but measurable income gains associated with large differences in single-year VAM scores.
posted by escabeche at 8:19 AM on May 19, 2012 [1 favorite]
posted by escabeche at 8:19 AM on May 19, 2012 [1 favorite]
And of course, I have a somewhat personal stake in this story -- I'm a math professor at Wisconsin and Carolyn will be leading discussion sections for our undergraduates next year. I can't say her scores give me any pause, and I will encourage her to lead off her first section by announcing that she's the worst math teacher in New York.
posted by escabeche at 8:22 AM on May 19, 2012 [21 favorites]
posted by escabeche at 8:22 AM on May 19, 2012 [21 favorites]
There are certainly some problems with the value-added metric, but finding a quirky outlier (someone who is teaching an entirely different curriculum from that being taught to the vast majority of NYC eighth graders) really isn't a useful way of highlighting those problems. The question of whether or not value added metrics are useful for assessing typical public-school teachers teaching a shared curriculum is simply not relevantly addressed by a demonstration that it does not usefully assess a wildly non-typical teacher teaching a different curriculum.
The Gary Rubenstein piece linked above is similarly flawed: first it makes the extremely dubious assumption that a teacher who teaches math and English must, a priori, be similarly effective in both disciplines. Secondly, it looks at a ludicrously short run of results in order to marvel at the fact that there are fluctuations year to year (within the same subject) or within the year (from subject to subject). But this is like concluding that batting average tells us nothing at all about the value of a baseball player because it varies from game to game or from week to week. Nobody, nobody at all, is suggesting that teacher salaries should vary from year to year according to the results of value-added metrics. Even the most fervid believers in these metrics accept that you need to look at how a teacher performs over a number of years to draw any meaningful conclusions.
scoring teachers based on the achievements of their students is much like your salary being determined by the performance of the guy down the line that you pass the widgets along to.
Really? So no part of a teacher's effectiveness can be measured by whether or not their students learn anything? That seems an odd notion.
posted by yoink at 8:24 AM on May 19, 2012 [4 favorites]
The Gary Rubenstein piece linked above is similarly flawed: first it makes the extremely dubious assumption that a teacher who teaches math and English must, a priori, be similarly effective in both disciplines. Secondly, it looks at a ludicrously short run of results in order to marvel at the fact that there are fluctuations year to year (within the same subject) or within the year (from subject to subject). But this is like concluding that batting average tells us nothing at all about the value of a baseball player because it varies from game to game or from week to week. Nobody, nobody at all, is suggesting that teacher salaries should vary from year to year according to the results of value-added metrics. Even the most fervid believers in these metrics accept that you need to look at how a teacher performs over a number of years to draw any meaningful conclusions.
scoring teachers based on the achievements of their students is much like your salary being determined by the performance of the guy down the line that you pass the widgets along to.
Really? So no part of a teacher's effectiveness can be measured by whether or not their students learn anything? That seems an odd notion.
posted by yoink at 8:24 AM on May 19, 2012 [4 favorites]
Jeez teaching assessment is a mess.
I don't understand how anyone is so stupid as to use this value-added measure as a decisive measure, though. It's obvious that there will be cases like this in which it's a terrible gauge of quality.
And when you hear the horror stories about teachers who are retained in NY despite their demonstrable awfulness!
Teaching assessment continues to be a mess up to the university level. We have two measures: student evaluations and "peer evaluations." There is some evidence that student evaluation scores are not only valueless, but *inversely correlated* with teacher quality. In peer reviews, your colleagues tell you weeks ahead of time that they're going to sit in on one class. No one is so incompetent that they can't teach one competent class with weeks to prepare.
posted by Fists O'Fury at 8:35 AM on May 19, 2012 [1 favorite]
I don't understand how anyone is so stupid as to use this value-added measure as a decisive measure, though. It's obvious that there will be cases like this in which it's a terrible gauge of quality.
And when you hear the horror stories about teachers who are retained in NY despite their demonstrable awfulness!
Teaching assessment continues to be a mess up to the university level. We have two measures: student evaluations and "peer evaluations." There is some evidence that student evaluation scores are not only valueless, but *inversely correlated* with teacher quality. In peer reviews, your colleagues tell you weeks ahead of time that they're going to sit in on one class. No one is so incompetent that they can't teach one competent class with weeks to prepare.
posted by Fists O'Fury at 8:35 AM on May 19, 2012 [1 favorite]
yoink correct on count 1 and at mostly correct on count 2.
1. The fact that Carolyn Abbott is actually a good math teacher is not in itself a signal that VAM is a bad idea -- you can always find crazy outlying examples, like the bus driver who made $135,000 in overtime, but the outliers are by definition not typical of the system.
2. Batting average is a good comparison; it has roughly the same year-to-year correlation as VAM does. Batting average is not a meaningless statistic; it tells you something about a hitter's ability. On the other hand, it would be nuts to make hiring decisions based on batting average. It would be almost equally nuts to make hiring decisions based on batting average over a 5-year span. We know enough about baseball to know that batting average is a) heavily influenced by factors outside a hitter's control, and b) insensitive to many of the characteristics of hitters that determine their actual value. We keep track of batting average, and we use it as one part of a big package of measures that help us make decisions about baseball players.
The psychometricians and education researchers who developed VAM (some of them my colleagues here at Wisconsin) think that VAM should be used in the same way. So do school administrators. That's because psychometricians and school administrators are not fools. yoink is right that you can't draw meaningful conclusions from a single year's numbers, and maybe not even from multiple years' numbers. The problem is that people are going to read these numbers in the newspapers and draw conclusions from them whether they're meaningful or not. And I don't see an easy fix for that. State records are state records, and I don't think the school system has any legal means of collecting this numbers without publicizing them.
posted by escabeche at 8:39 AM on May 19, 2012 [2 favorites]
1. The fact that Carolyn Abbott is actually a good math teacher is not in itself a signal that VAM is a bad idea -- you can always find crazy outlying examples, like the bus driver who made $135,000 in overtime, but the outliers are by definition not typical of the system.
2. Batting average is a good comparison; it has roughly the same year-to-year correlation as VAM does. Batting average is not a meaningless statistic; it tells you something about a hitter's ability. On the other hand, it would be nuts to make hiring decisions based on batting average. It would be almost equally nuts to make hiring decisions based on batting average over a 5-year span. We know enough about baseball to know that batting average is a) heavily influenced by factors outside a hitter's control, and b) insensitive to many of the characteristics of hitters that determine their actual value. We keep track of batting average, and we use it as one part of a big package of measures that help us make decisions about baseball players.
The psychometricians and education researchers who developed VAM (some of them my colleagues here at Wisconsin) think that VAM should be used in the same way. So do school administrators. That's because psychometricians and school administrators are not fools. yoink is right that you can't draw meaningful conclusions from a single year's numbers, and maybe not even from multiple years' numbers. The problem is that people are going to read these numbers in the newspapers and draw conclusions from them whether they're meaningful or not. And I don't see an easy fix for that. State records are state records, and I don't think the school system has any legal means of collecting this numbers without publicizing them.
posted by escabeche at 8:39 AM on May 19, 2012 [2 favorites]
Looking at the person singled out (and pushed out of the system) by being reported as the worst teacher in all of NYC is unfair? She's not an outlier, she's first blood. I think looking at the effects these scores have on teachers is valuable.
posted by Garm at 8:46 AM on May 19, 2012 [1 favorite]
posted by Garm at 8:46 AM on May 19, 2012 [1 favorite]
New York may have problems, but it least it has the Anderson School and Hunter and Lower Lab and Stuyvesant and Science: perfect models for what big public systems can deliver to motivated parents.
posted by MattD at 8:46 AM on May 19, 2012 [2 favorites]
posted by MattD at 8:46 AM on May 19, 2012 [2 favorites]
I'm an enthusiastic booster of metrics for evaluating teacher performance---if you have no metrics, you can't improve, and that's a fact. But this makes clear that having metrics is not enough; your metrics actually have to be *good*. Some people will use this as a cudgel to attack any kind of teacher evaluation, which is a pity, but hopefully it will also inspire some serious thought in the Ed department along the lines of "How can we actually evaluate that which we're supposed to evaluate."
posted by ThatFuzzyBastard at 9:08 AM on May 19, 2012 [1 favorite]
posted by ThatFuzzyBastard at 9:08 AM on May 19, 2012 [1 favorite]
the point of the teacher rankings isn't to bring modern statistical methods of quality control to the industrial model of education, but to put pressure on the teacher's union and add to the atmosphere of crisis in education in order to push through radical action.
This is exactly how it's being used here in Indiana, with theGreekRepublican chorus chanting "charter schools, charter schools..." in the background, egging the process on.
posted by Thorzdad at 9:09 AM on May 19, 2012 [1 favorite]
This is exactly how it's being used here in Indiana, with the
posted by Thorzdad at 9:09 AM on May 19, 2012 [1 favorite]
I find the value-added assessment of teachers really perplexing. The LA Times finagled its way into the numbers out here recently (to the grouchiness of many people.) What I noticed when looking at the numbers for various schools was that schools with 10/10 Greatschools ratings (which I'm sure are also a flawed metric, but just go with it for a second) had teachers with really low value-added numbers. Whereas the (poorly performing) local elementary school had teachers with quite high value-added numbers.
If you look at the individual schools, it turns out that in the poorly-performing school, incoming first-graders are almost all English learners. So just for that reason maybe your low initial test scores that jump each year as the kids become more confident using English? And when you look at the numbers, the test scores do jump, but they're jumping from "the worst" to "less bad". But those teachers wind up with high value-added numbers. (I don't know, maybe these teachers are amazing and their school performs poorly for other reasons. I just don't totally understand this as a method of teacher ranking.)
At one of the 10/10 schools, they have a locally-famous teacher who is very high-performing, a hit with kids and parents, etc. His value-added scores were thoroughly mediocre. But the school is in the home school of UCLA graduate student housing, and reviews of it mention the very high number of kids of visiting scholars, etc. So is it possible that in this school you are seeing kids who come to school very well-prepared, and who do home enrichment with geeky parents? The incoming test scores at this school are high and stay high. So even though this school does very well every year by every metric you care to apply, the individual teachers get lousy value-added scores.
So I feel like there's something fundamental about this idea of ranking teachers by the value they add that I just don't grok.
posted by thehmsbeagle at 9:15 AM on May 19, 2012
If you look at the individual schools, it turns out that in the poorly-performing school, incoming first-graders are almost all English learners. So just for that reason maybe your low initial test scores that jump each year as the kids become more confident using English? And when you look at the numbers, the test scores do jump, but they're jumping from "the worst" to "less bad". But those teachers wind up with high value-added numbers. (I don't know, maybe these teachers are amazing and their school performs poorly for other reasons. I just don't totally understand this as a method of teacher ranking.)
At one of the 10/10 schools, they have a locally-famous teacher who is very high-performing, a hit with kids and parents, etc. His value-added scores were thoroughly mediocre. But the school is in the home school of UCLA graduate student housing, and reviews of it mention the very high number of kids of visiting scholars, etc. So is it possible that in this school you are seeing kids who come to school very well-prepared, and who do home enrichment with geeky parents? The incoming test scores at this school are high and stay high. So even though this school does very well every year by every metric you care to apply, the individual teachers get lousy value-added scores.
So I feel like there's something fundamental about this idea of ranking teachers by the value they add that I just don't grok.
posted by thehmsbeagle at 9:15 AM on May 19, 2012
MattD: Good luck getting a child in to one of those schools, when even pre-K students in the system are having to take and score at 99%+ on entry exams to get into the pipeline of schools that feed into NYC's elite publics. [And yes, this means that these 'motivated parents' you describe often start their children with private tutors and exam cramming when they are 3.]
That those schools exist alongside some of the ruinously terrible high schools in Brooklyn and the South Bronx is a testament to just how dire things are in the city's education system.
posted by yellowcandy at 9:15 AM on May 19, 2012 [2 favorites]
That those schools exist alongside some of the ruinously terrible high schools in Brooklyn and the South Bronx is a testament to just how dire things are in the city's education system.
posted by yellowcandy at 9:15 AM on May 19, 2012 [2 favorites]
OK. So why do we have teachers and school administrators at all? If the metrics are so all-knowing as to what constitutes good and poor teaching, why don't we just write a program based on those concepts, and have computers teach our kids?
Teaching ability = some semi-arbitrary number is just ridiculous.
Teaching ability = some semi-arbitrary number = funding is even crazier.
posted by Benny Andajetz at 9:26 AM on May 19, 2012 [1 favorite]
Teaching ability = some semi-arbitrary number is just ridiculous.
Teaching ability = some semi-arbitrary number = funding is even crazier.
posted by Benny Andajetz at 9:26 AM on May 19, 2012 [1 favorite]
The psychometricians and education researchers who developed VAM (some of them my colleagues here at Wisconsin) think that VAM should be used in the same way. So do school administrators. That's because psychometricians and school administrators are not fools. yoink is right that you can't draw meaningful conclusions from a single year's numbers, and maybe not even from multiple years' numbers. The problem is that people are going to read these numbers in the newspapers and draw conclusions from them whether they're meaningful or not. And I don't see an easy fix for that. State records are state records, and I don't think the school system has any legal means of collecting this numbers without publicizing them.
if you make a tool which is only used incorrectly, which harms people in the process, children even, maybe you should stop making that kind of tool. it's disingenuous to suggest that the problem is that VAM isn't being used correctly. performance metrics for teachers has a very specific political agenda and your colleagues are parties to it whether they play "i'm just a neutral scientist" or not.
I'm an enthusiastic booster of metrics for evaluating teacher performance---if you have no metrics, you can't improve, and that's a fact.
that's simply not true, most of the universe of human activities aren't being assigned performance metrics and people improve all the time. and it's not just a question of whether you have a valid metric, it's whether the process, education in this case, has any *useful* metrics. that is metrics you can reasonably measure given the range of uncontrolled variables, the uncontrolled testing environment, the feedback loops between testing and error through cheating and/or manipulation of testing, etc. how do you assign performance metrics to human relationships? because that's what teaching and learning is mainly about...
but i'll repeat, this isn't above implementing industrial scale quality control in education, it's about advancing a political agenda, mainly, to privatize the public schools.
you could design a data centric public school system, where useful data could be used to improve education. but the idea that you can improve education by creating rewards and punishments based on "performance" is just completely wrong-headed. luckily, no one is actually interested in doing that. it's just SMASH, DISRUPT, and SELL-OFF.
posted by ennui.bz at 9:47 AM on May 19, 2012 [5 favorites]
if you make a tool which is only used incorrectly, which harms people in the process, children even, maybe you should stop making that kind of tool. it's disingenuous to suggest that the problem is that VAM isn't being used correctly. performance metrics for teachers has a very specific political agenda and your colleagues are parties to it whether they play "i'm just a neutral scientist" or not.
I'm an enthusiastic booster of metrics for evaluating teacher performance---if you have no metrics, you can't improve, and that's a fact.
that's simply not true, most of the universe of human activities aren't being assigned performance metrics and people improve all the time. and it's not just a question of whether you have a valid metric, it's whether the process, education in this case, has any *useful* metrics. that is metrics you can reasonably measure given the range of uncontrolled variables, the uncontrolled testing environment, the feedback loops between testing and error through cheating and/or manipulation of testing, etc. how do you assign performance metrics to human relationships? because that's what teaching and learning is mainly about...
but i'll repeat, this isn't above implementing industrial scale quality control in education, it's about advancing a political agenda, mainly, to privatize the public schools.
you could design a data centric public school system, where useful data could be used to improve education. but the idea that you can improve education by creating rewards and punishments based on "performance" is just completely wrong-headed. luckily, no one is actually interested in doing that. it's just SMASH, DISRUPT, and SELL-OFF.
posted by ennui.bz at 9:47 AM on May 19, 2012 [5 favorites]
but i'll repeat, this isn't above implementing industrial scale quality control in education, it's about advancing a political agenda, mainly, to privatize the public schools.
You don't think there is a political agenda in avoiding any kind of objective measurement of teacher performance?
posted by rr at 10:33 AM on May 19, 2012 [2 favorites]
You don't think there is a political agenda in avoiding any kind of objective measurement of teacher performance?
posted by rr at 10:33 AM on May 19, 2012 [2 favorites]
OK. So why do we have teachers and school administrators at all? If the metrics are so all-knowing as to what constitutes good and poor teaching, why don't we just write a program based on those concepts, and have computers teach our kids?
Teaching ability = some semi-arbitrary number is just ridiculous.
Teaching ability = some semi-arbitrary number = funding is even crazier.
The intangibles humans put into teaching other humans are reflected in the measured results but that doesn't mean the input can be duplicated by machines. For example, one measure of a good piano teacher is his or her students' ability to sight read a piece on the first try, keep impeccable time, play fine grades of volume, etc. A computer could measure a lot of that but could not explain how to do it to a student, let alone actually make it happen over several years.
However, that may change in the future. Computer-generated lesson plans can't be worse than the terrible lesson plans to which some students are subjected. Education software/website quality seems to be advancing faster than education degree quality and school administration quality, so perhaps your prediction will come true in 20-30 years for the average student. There will be upsides and downsides if so.
posted by michaelh at 10:44 AM on May 19, 2012
Teaching ability = some semi-arbitrary number is just ridiculous.
Teaching ability = some semi-arbitrary number = funding is even crazier.
The intangibles humans put into teaching other humans are reflected in the measured results but that doesn't mean the input can be duplicated by machines. For example, one measure of a good piano teacher is his or her students' ability to sight read a piece on the first try, keep impeccable time, play fine grades of volume, etc. A computer could measure a lot of that but could not explain how to do it to a student, let alone actually make it happen over several years.
However, that may change in the future. Computer-generated lesson plans can't be worse than the terrible lesson plans to which some students are subjected. Education software/website quality seems to be advancing faster than education degree quality and school administration quality, so perhaps your prediction will come true in 20-30 years for the average student. There will be upsides and downsides if so.
posted by michaelh at 10:44 AM on May 19, 2012
Yellowcandy, no doubt it's tough to get in to those schools.
A more fully-argued point is that New York has a great model for a lucky few motivated parents that should be built on. People who get their kids into Anderson and its few peers are very happy but there is absolutely nothing special or hard to replicate about them. (Anderson, for example, is in an ordinary New York City school building on an ordinary block, completely indistinguishable from any other public school sites on the Upper West Side.) The school board could soak up all the G&T demand easily converting a couple dozen more school sites to test admission and appropriate faculty.
posted by MattD at 10:47 AM on May 19, 2012
A more fully-argued point is that New York has a great model for a lucky few motivated parents that should be built on. People who get their kids into Anderson and its few peers are very happy but there is absolutely nothing special or hard to replicate about them. (Anderson, for example, is in an ordinary New York City school building on an ordinary block, completely indistinguishable from any other public school sites on the Upper West Side.) The school board could soak up all the G&T demand easily converting a couple dozen more school sites to test admission and appropriate faculty.
posted by MattD at 10:47 AM on May 19, 2012
scoring teachers based on the achievements of their students is much like your salary being determined by the performance of the guy down the line that you pass the widgets along to.
More like a manger's salary or evaluation being determined by the performance of his direct reports. Which makes perfect sense to me.
Teaching ability = some semi-arbitrary number is just ridiculous.
Well yes, some semi-arbitrary number would be ridiculous. But do we throw out the whole attempt just because our current metrics are bad?
I understand the current push for teacher evaluation comes from Republican attempts to dismantle the public education system, which are horrible. But this backlash that teaching ability somehow can't be quantified is also crazy.
I suppose I shouldn't be surprised that people are trying to come up with a (relatively simple) numeric formula to indicate how "good" or "bad" a particular teacher is — after all, education is becoming measured more and more by numbers only — but in my defense, it's such a ridiculous idea that I feel justified in being surprised.
Again, why? In a way, all of this backlash is ironically anti-science. What's the alternative? "There's no possible way we can measure some real-world phenomenon! These teachers abilities are magical! Teachers are magical!"
I understand the pushback against the Republican agenda against education, and strategically attacking teacher assessment may be a valid approach to that. But again I think well-meaning liberals are on the wrong side.
I'm also not arguing teachers and students are just cogs in a machine that just computes their utility function at the end of the day. We can throw in some measures on creativity, increased passion for learning and exploring by students, etc. The metrics don't have to be dumb, which many of them currently are.
posted by formless at 11:32 AM on May 19, 2012 [2 favorites]
More like a manger's salary or evaluation being determined by the performance of his direct reports. Which makes perfect sense to me.
Teaching ability = some semi-arbitrary number is just ridiculous.
Well yes, some semi-arbitrary number would be ridiculous. But do we throw out the whole attempt just because our current metrics are bad?
I understand the current push for teacher evaluation comes from Republican attempts to dismantle the public education system, which are horrible. But this backlash that teaching ability somehow can't be quantified is also crazy.
I suppose I shouldn't be surprised that people are trying to come up with a (relatively simple) numeric formula to indicate how "good" or "bad" a particular teacher is — after all, education is becoming measured more and more by numbers only — but in my defense, it's such a ridiculous idea that I feel justified in being surprised.
Again, why? In a way, all of this backlash is ironically anti-science. What's the alternative? "There's no possible way we can measure some real-world phenomenon! These teachers abilities are magical! Teachers are magical!"
I understand the pushback against the Republican agenda against education, and strategically attacking teacher assessment may be a valid approach to that. But again I think well-meaning liberals are on the wrong side.
I'm also not arguing teachers and students are just cogs in a machine that just computes their utility function at the end of the day. We can throw in some measures on creativity, increased passion for learning and exploring by students, etc. The metrics don't have to be dumb, which many of them currently are.
posted by formless at 11:32 AM on May 19, 2012 [2 favorites]
I'm an enthusiastic booster of metrics for evaluating teacher performance---if you have no metrics, you can't improve, and that's a fact.
that's simply not true, most of the universe of human activities aren't being assigned performance metrics and people improve all the time.
Perhaps I'm not being imaginative enough, but I can't think of any such circumstance. If I want to cook better Chicken Florentine, I cook some, eat it, see if it tastes good, and if not, do something different next time. If I'm a sales clerk, my boss looks at my sale numbers, and evaluates me based on that. If I'm running a multi-branch company, and want to know how Franchise Manager X is doing, I compare his numbers to other franchises, adjust for location, and evaluate. What operation (in particular, what large-scale operation like public education) has ever been improved without metrics for improvement?
but i'll repeat, this isn't above implementing industrial scale quality control in education, it's about advancing a political agenda, mainly, to privatize the public schools.
This is not only a false assertion, it's a dangerous one. People---especially parents--- want data that helps them decide if their school is getting better or worse. If you keep insisting that only right-wingers would provide such data, they will naturally conclude that right-wingers must be the only ones who care about their children's education. There is nothing progressive about deliberate ignorance.
If you think existing metrics aren't useful, you're right! They suck! But insisting that metrics themselves aren't useful is suicide.
posted by ThatFuzzyBastard at 11:34 AM on May 19, 2012 [3 favorites]
that's simply not true, most of the universe of human activities aren't being assigned performance metrics and people improve all the time.
Perhaps I'm not being imaginative enough, but I can't think of any such circumstance. If I want to cook better Chicken Florentine, I cook some, eat it, see if it tastes good, and if not, do something different next time. If I'm a sales clerk, my boss looks at my sale numbers, and evaluates me based on that. If I'm running a multi-branch company, and want to know how Franchise Manager X is doing, I compare his numbers to other franchises, adjust for location, and evaluate. What operation (in particular, what large-scale operation like public education) has ever been improved without metrics for improvement?
but i'll repeat, this isn't above implementing industrial scale quality control in education, it's about advancing a political agenda, mainly, to privatize the public schools.
This is not only a false assertion, it's a dangerous one. People---especially parents--- want data that helps them decide if their school is getting better or worse. If you keep insisting that only right-wingers would provide such data, they will naturally conclude that right-wingers must be the only ones who care about their children's education. There is nothing progressive about deliberate ignorance.
If you think existing metrics aren't useful, you're right! They suck! But insisting that metrics themselves aren't useful is suicide.
posted by ThatFuzzyBastard at 11:34 AM on May 19, 2012 [3 favorites]
So tell me, what Chicken Florentine performance metrics are you going to use for your Chicken Florentine Improvement Initiative? Tastes good is arbitrary. You need a hard and fast objective number. And even if it does taste good, are you sure it couldn't taste better? And when you get to a point that you can't make the Chicken Florentine taste any better - you make Chicken Florentine that makes famous chefs fall down and weep with joy upon tasting it - would you agree that you are a terrible cook because you're not improving at the same rate as the guy who last week served up a dead rat, but this week managed to produce a slightly burned grilled cheese sandwich with very little fecal contamination?
The backlash only seems anti-scientific if you can't tell the difference between the scientific method and building a bamboo radar dish so that the sky gods will bless you with bountiful cargo. If you scroll to the end of at the link that escabeche included, the one that seems to support the value added score, you see lots of graphs with REALLY ZOOMED Y axes and really low correlation coefficients. That isn't exactly a hallmark of solid data analysis. (One of the stats they graphed was women with teenaged births. Was there previously some conjecture that good math teachers prevented teenaged pregnancy, or did they just graph a shit ton of statistics and pick the ones that game them a positive slope? I'm not sure, but I know which way I'd bet.)
Broken metrics are just a way to be wrong with precision, no matter how many sigmas, thetas and xis you garnish them with.
posted by Kid Charlemagne at 12:50 PM on May 19, 2012 [12 favorites]
The backlash only seems anti-scientific if you can't tell the difference between the scientific method and building a bamboo radar dish so that the sky gods will bless you with bountiful cargo. If you scroll to the end of at the link that escabeche included, the one that seems to support the value added score, you see lots of graphs with REALLY ZOOMED Y axes and really low correlation coefficients. That isn't exactly a hallmark of solid data analysis. (One of the stats they graphed was women with teenaged births. Was there previously some conjecture that good math teachers prevented teenaged pregnancy, or did they just graph a shit ton of statistics and pick the ones that game them a positive slope? I'm not sure, but I know which way I'd bet.)
Broken metrics are just a way to be wrong with precision, no matter how many sigmas, thetas and xis you garnish them with.
posted by Kid Charlemagne at 12:50 PM on May 19, 2012 [12 favorites]
ThatFuzzyBastard If I want to cook better Chicken Florentine, I cook some, eat it, see if it tastes good, and if not, do something different next time.
This is moving the goal posts. This is qualitative observation, not a metric. Sure you could keep track scientifically of everything that you do, rank different chicken florentines according to how good they taste, and assign numerical values to different recipes: thus creating a metric for chicken florentine, but most people don't use spreadsheets to evaluate their cooking, and many of them manage to make a decent chicken florentine anyway. The best food I've ever eaten is my mothers (cliché, but true). She's been cooking for large families since she was 12 and large numbers of guests since her 20s. She has extensive knowledge of the cuisines of many cultures, and never makes the same recipe twice because she has no idea what she does, doesn't keep track, and doesn't evaluate. What she does have is an excellent experiential understanding of how flavors combine. She invents things on the spot, crosses cultures with impunity, and never makes a bad meal. One might say that she has a scientific theory of underlying mechanism by which cooking works. She could probably even explain it to you. But it's not based on a metric.
ThatFuzzyBastard What operation (in particular, what large-scale operation like public education) has ever been improved without metrics for improvement?
The United States political system continues to improve. Over the past 200 years, people have higher standards of living, more wealth, better health, better education, and more civil rights. Sure there are dips (like the past decade) but over all, people are better off than they were 50 years ago. This is the result of a political system that is not based on metrics. Sure there are measures for the system as a whole, but the components we actually change -- the lawmakers -- are not evaluated by measurement. Lawmakers are not publicly ranked according to "how good a lawmaker" they are. Politicians are evaluated by voters' gut instinct, and qualitative issues like "did this guy mail a photo of his penis?"
Sure there are metrics available such as unemployment numbers and violent crime statistics, and they are even sometimes used to evaluate politicians. But these numbers reflect a tiny part of what lawmakers actually do, and the ability of a lawmaker to influence these numbers is limited. Similarly, evaluating teachers based on test scores is evaluating teachers on only a small part of what they do. Teachers build content level understandings that are not measured on tests. They exceed standards in ways that are not measured on tests. They help build understandings of social norms particular to a profession that are not measures on tests. They offer guidance with life issues and careers that is not measured on tests. They teach students what their role in a classroom is (which is tremendously important for academic success and can be measured on tests, just not on these tests). The list goes on but I'm tired of typing.
Politicians aren't evaluated by metrics for the simple reason that people want different things from their politicians. People have different values, they would each evaluate a politician differently. A southern evangelical Christian would demand very different rankings of politicians than a environmental activist. Ranking politicians on their "general effectiveness" doesn't make sense. The best metric is voting record, a count of how many times a politician agrees with your values, and realistically, very few people make use of that. Somehow, though, the system works.
Education has a similar problem. Parents want very different things from their children's teachers. "Conceptual understanding" vs. 'back to basics" is a classic example, but there are others. Parents, lawmakers, administrators, and the students themselves all have different ideas about what a student should learn and what their life path should be. They want different things emphasized in the curriculum. Metrics that rank teachers based on test scores do not take these differences in values and goals into account. Is the better algebra teacher one who emphasizes to students the math they will need to learn calculus or the math they need to pay their taxes?
ThatFuzzyBastard If you think existing metrics aren't useful, you're right! They suck! But insisting that metrics themselves aren't useful is suicide.
On this we agree. Metrics are incredibly useful. It would be nice if we had good metrics to evaluate teachers. It would be fantastic if we had good metrics to evaluate politicians. The problem is that we don't have a good understanding of what these metrics need to be, but (in education at least) we insist on having metrics anyway. This results in metrics being more popular than they are useful, which is damaging. It's lying with statistics.
formless Again, why? In a way, all of this backlash is ironically anti-science. What's the alternative? "There's no possible way we can measure some real-world phenomenon! These teachers abilities are magical! Teachers are magical!"
Metrics have the trappings of science, but they are not inherently scientific. Science is the process of understanding how things come about. Building theories of how the components work together. People don't just guess at the laws of physics and then test them. They do a lot of reasoning beforehand to build a hypothesis based on a theory. An understanding of cooking results in better meals than guessing at recipe variations and ranking them.
We are not currently at a stage of understanding how teaching works that we can produce decent metrics. All we can do right now is guess at what results a good teacher would generate and then check if that happened. That's not science. We don't understand enough about what a good teacher is yet to identify one. We are very much in the data gathering theory building stage. Evaluation and testing still has to be done, and is valuable in building and testing theory, but it's not at the stage where it can be used to test and evaluate teachers yet. Without and understanding of what we want and how to achieve it, testing is "magical thinking" with numbers.
So yes, some of the backlash against metrics is anti-science, but a lot of it is pro-science. Teachers are not magical, they are complex. We need to understand how they work first.
tl;dr: What Kid Charlemagne said, but nicer.
posted by yeolcoatl at 12:57 PM on May 19, 2012 [8 favorites]
This is moving the goal posts. This is qualitative observation, not a metric. Sure you could keep track scientifically of everything that you do, rank different chicken florentines according to how good they taste, and assign numerical values to different recipes: thus creating a metric for chicken florentine, but most people don't use spreadsheets to evaluate their cooking, and many of them manage to make a decent chicken florentine anyway. The best food I've ever eaten is my mothers (cliché, but true). She's been cooking for large families since she was 12 and large numbers of guests since her 20s. She has extensive knowledge of the cuisines of many cultures, and never makes the same recipe twice because she has no idea what she does, doesn't keep track, and doesn't evaluate. What she does have is an excellent experiential understanding of how flavors combine. She invents things on the spot, crosses cultures with impunity, and never makes a bad meal. One might say that she has a scientific theory of underlying mechanism by which cooking works. She could probably even explain it to you. But it's not based on a metric.
ThatFuzzyBastard What operation (in particular, what large-scale operation like public education) has ever been improved without metrics for improvement?
The United States political system continues to improve. Over the past 200 years, people have higher standards of living, more wealth, better health, better education, and more civil rights. Sure there are dips (like the past decade) but over all, people are better off than they were 50 years ago. This is the result of a political system that is not based on metrics. Sure there are measures for the system as a whole, but the components we actually change -- the lawmakers -- are not evaluated by measurement. Lawmakers are not publicly ranked according to "how good a lawmaker" they are. Politicians are evaluated by voters' gut instinct, and qualitative issues like "did this guy mail a photo of his penis?"
Sure there are metrics available such as unemployment numbers and violent crime statistics, and they are even sometimes used to evaluate politicians. But these numbers reflect a tiny part of what lawmakers actually do, and the ability of a lawmaker to influence these numbers is limited. Similarly, evaluating teachers based on test scores is evaluating teachers on only a small part of what they do. Teachers build content level understandings that are not measured on tests. They exceed standards in ways that are not measured on tests. They help build understandings of social norms particular to a profession that are not measures on tests. They offer guidance with life issues and careers that is not measured on tests. They teach students what their role in a classroom is (which is tremendously important for academic success and can be measured on tests, just not on these tests). The list goes on but I'm tired of typing.
Politicians aren't evaluated by metrics for the simple reason that people want different things from their politicians. People have different values, they would each evaluate a politician differently. A southern evangelical Christian would demand very different rankings of politicians than a environmental activist. Ranking politicians on their "general effectiveness" doesn't make sense. The best metric is voting record, a count of how many times a politician agrees with your values, and realistically, very few people make use of that. Somehow, though, the system works.
Education has a similar problem. Parents want very different things from their children's teachers. "Conceptual understanding" vs. 'back to basics" is a classic example, but there are others. Parents, lawmakers, administrators, and the students themselves all have different ideas about what a student should learn and what their life path should be. They want different things emphasized in the curriculum. Metrics that rank teachers based on test scores do not take these differences in values and goals into account. Is the better algebra teacher one who emphasizes to students the math they will need to learn calculus or the math they need to pay their taxes?
ThatFuzzyBastard If you think existing metrics aren't useful, you're right! They suck! But insisting that metrics themselves aren't useful is suicide.
On this we agree. Metrics are incredibly useful. It would be nice if we had good metrics to evaluate teachers. It would be fantastic if we had good metrics to evaluate politicians. The problem is that we don't have a good understanding of what these metrics need to be, but (in education at least) we insist on having metrics anyway. This results in metrics being more popular than they are useful, which is damaging. It's lying with statistics.
formless Again, why? In a way, all of this backlash is ironically anti-science. What's the alternative? "There's no possible way we can measure some real-world phenomenon! These teachers abilities are magical! Teachers are magical!"
Metrics have the trappings of science, but they are not inherently scientific. Science is the process of understanding how things come about. Building theories of how the components work together. People don't just guess at the laws of physics and then test them. They do a lot of reasoning beforehand to build a hypothesis based on a theory. An understanding of cooking results in better meals than guessing at recipe variations and ranking them.
We are not currently at a stage of understanding how teaching works that we can produce decent metrics. All we can do right now is guess at what results a good teacher would generate and then check if that happened. That's not science. We don't understand enough about what a good teacher is yet to identify one. We are very much in the data gathering theory building stage. Evaluation and testing still has to be done, and is valuable in building and testing theory, but it's not at the stage where it can be used to test and evaluate teachers yet. Without and understanding of what we want and how to achieve it, testing is "magical thinking" with numbers.
So yes, some of the backlash against metrics is anti-science, but a lot of it is pro-science. Teachers are not magical, they are complex. We need to understand how they work first.
tl;dr: What Kid Charlemagne said, but nicer.
posted by yeolcoatl at 12:57 PM on May 19, 2012 [8 favorites]
Nothing in this discussion leads me to regret my child deciding not to be a teacher. I do find it interesting that there is limited info on how these metrics affect decisions to become or stay a teacher. I'm beginning to think the public school system is getting the teachers it deserves. Who in their right mind would subject themselves to this kind of public scrutiny and second hand guessing for teacher scale pay?
posted by beaning at 2:05 PM on May 19, 2012 [2 favorites]
posted by beaning at 2:05 PM on May 19, 2012 [2 favorites]
It seems to me that if you want to demolish the public school system, a metric that ranks an otherwise excellent teacher at the bottom is exactly what you want.
posted by dirigibleman at 2:17 PM on May 19, 2012
posted by dirigibleman at 2:17 PM on May 19, 2012
People who get their kids into Anderson and its few peers are very happy but there is absolutely nothing special or hard to replicate about them.
I am a fan of aspirational public education, and God knows we can do better than metal detectors and gradeschool jails. But, what Anderson offers is special and is hard to replicate, which is why it just isn't that common. First of all, your child has to be (or has to be able to pass as) gifted and talented. Second of all, your family has to place a very high premium on education (or on saving 30k per year in private school tuition). Third, Anderson attracts not only exceptional students but exceptional teachers; it's as competitive for jobs as it is for student places. Finally, you have never seen a PTA like this PTA, presidency of which is literally "take a sabbatical, because this is now your full time job." In addition, that PTA leads a fundraising machine that rivals Obama's and has, at least in part, the parental wealth to support it. I went to swanky private schools in NYC pre-K through 12 and I nothing at any of those schools was really very much like what goes on at Anderson and the handful of schools like it. I completely leave room for the fact that other people in other districts may have different experiences, but it seems pretty unique on the public school landscape to me.
posted by DarlingBri at 2:24 PM on May 19, 2012
I am a fan of aspirational public education, and God knows we can do better than metal detectors and gradeschool jails. But, what Anderson offers is special and is hard to replicate, which is why it just isn't that common. First of all, your child has to be (or has to be able to pass as) gifted and talented. Second of all, your family has to place a very high premium on education (or on saving 30k per year in private school tuition). Third, Anderson attracts not only exceptional students but exceptional teachers; it's as competitive for jobs as it is for student places. Finally, you have never seen a PTA like this PTA, presidency of which is literally "take a sabbatical, because this is now your full time job." In addition, that PTA leads a fundraising machine that rivals Obama's and has, at least in part, the parental wealth to support it. I went to swanky private schools in NYC pre-K through 12 and I nothing at any of those schools was really very much like what goes on at Anderson and the handful of schools like it. I completely leave room for the fact that other people in other districts may have different experiences, but it seems pretty unique on the public school landscape to me.
posted by DarlingBri at 2:24 PM on May 19, 2012
Darling Bri, I appreciate what they've done at Anderson. People love it and for good reason. The Board of Ed could designate 8 more buildings on the UWS and UES (and couple downtown and in Brooklyn for all you hipsters) as "Andersons #2-#11" and achieve virtually the same outcome. The test bar would go from 99th percentile to 95th percentile with no appreciable dip in the quality of student or parent or dedication of PTA. Faculty would be no problem -- the Board of Ed pays a lot better than private school, and a lot of younger teachers of high quality would love to be out of the suburbs. The number of parents who want to stay in the city and want not to pay private school tuition is HUGE. Heck, you make the selective / exclusive system sufficiently big and a lot of people could start moving back in from the suburbs.
posted by MattD at 2:54 PM on May 19, 2012
posted by MattD at 2:54 PM on May 19, 2012
Poor kids don't do as well in school as rich kids. Instead of trying to reduce the number of poor kids, government puts in place a system that discourages teachers from teaching in poor areas. Nothing is solved.
I teach in one of the richest cities in the 5th richest county in the US. Half of my first graders come into the year already reading, because their house is full of books and their parents or nannies read to them.
It's so frustrating to see all this discussion of teacher quality when the real issue is poverty, it always has been. When I taught in Garden Grove, CA, one of the poorest cities here, guess what? My kids didn't come in reading. There were no books in their homes. Their parents often didn't speak English. And they didn't do as well on state tests.
My students now score in the 90th percentile when they take those tests. Because of me? Or because their parents can afford to spend time reading to them, helping with homework, and by just talking to them convey the education they themselves have?
Bottom line, I'd never move to a poorly-performing school, which means a poor school, even though I have great memories of those kids. Which is too bad, because now that I have a decade of experience teaching, I have a lot of ideas that I think would help those kids, and I loved teaching them. It's just too risky in today's climate.
I've talked to a lot of teachers about evaluation, and the only way we've come up with that makes sense to us is an idea that Bill Gates has endorsed... train experienced teachers to visit classrooms and watch teachers teach. Just so you know, all the teachers I know are fine with getting rid of crappy teachers. If you're going to use test scores, though, you might as well just fire all the teachers who work at poor schools, and then who's going to teach there?
posted by Huck500 at 4:19 PM on May 19, 2012 [7 favorites]
I teach in one of the richest cities in the 5th richest county in the US. Half of my first graders come into the year already reading, because their house is full of books and their parents or nannies read to them.
It's so frustrating to see all this discussion of teacher quality when the real issue is poverty, it always has been. When I taught in Garden Grove, CA, one of the poorest cities here, guess what? My kids didn't come in reading. There were no books in their homes. Their parents often didn't speak English. And they didn't do as well on state tests.
My students now score in the 90th percentile when they take those tests. Because of me? Or because their parents can afford to spend time reading to them, helping with homework, and by just talking to them convey the education they themselves have?
Bottom line, I'd never move to a poorly-performing school, which means a poor school, even though I have great memories of those kids. Which is too bad, because now that I have a decade of experience teaching, I have a lot of ideas that I think would help those kids, and I loved teaching them. It's just too risky in today's climate.
I've talked to a lot of teachers about evaluation, and the only way we've come up with that makes sense to us is an idea that Bill Gates has endorsed... train experienced teachers to visit classrooms and watch teachers teach. Just so you know, all the teachers I know are fine with getting rid of crappy teachers. If you're going to use test scores, though, you might as well just fire all the teachers who work at poor schools, and then who's going to teach there?
posted by Huck500 at 4:19 PM on May 19, 2012 [7 favorites]
Poor kids don't do as well in school as rich kids. Instead of trying to reduce the number of poor kids, government puts in place a system that discourages teachers from teaching in poor areas. Nothing is solved.
I am pretty sure it's way more complex and harder to solve than that. Poor kids in impoverished schools lag significantly behind their counterparts. But if you take inner-city kids and provide them with music education, studies tell us they will do better in math through gradeschool. Anderson only has music education because the PTA fundraises for it, to the tune of $1600 per child per year. And they can only do that successfully because, etc.
The way education is funded in the US is fundamentally broken; it's institutionally racist and classiest and not likely to change.
posted by DarlingBri at 5:10 PM on May 19, 2012 [1 favorite]
I am pretty sure it's way more complex and harder to solve than that. Poor kids in impoverished schools lag significantly behind their counterparts. But if you take inner-city kids and provide them with music education, studies tell us they will do better in math through gradeschool. Anderson only has music education because the PTA fundraises for it, to the tune of $1600 per child per year. And they can only do that successfully because, etc.
The way education is funded in the US is fundamentally broken; it's institutionally racist and classiest and not likely to change.
posted by DarlingBri at 5:10 PM on May 19, 2012 [1 favorite]
scoring teachers based on the achievements of their students is much like your salary being determined by the performance of the guy down the line that you pass the widgets along to.
A. That's ridiculous. I hope no actual teachers think this is how their jobs work.
B. What better way would you rate teachers?
posted by gjc at 7:38 PM on May 19, 2012
A. That's ridiculous. I hope no actual teachers think this is how their jobs work.
B. What better way would you rate teachers?
posted by gjc at 7:38 PM on May 19, 2012
If you're going to use test scores, though, you might as well just fire all the teachers who work at poor schools, and then who's going to teach there?
All you have to do is look at the year to year change in test scores. Or test at the beginning and then at the end. If the students don't improve, the teacher is ineffective.
posted by gjc at 7:40 PM on May 19, 2012
All you have to do is look at the year to year change in test scores. Or test at the beginning and then at the end. If the students don't improve, the teacher is ineffective.
posted by gjc at 7:40 PM on May 19, 2012
gjc (7:38 PM) What better way would you rate teachers?
Huck500 (4:19 PM) I've talked to a lot of teachers about evaluation, and the only way we've come up with that makes sense to us is an idea that Bill Gates has endorsed... train experienced teachers to visit classrooms and watch teachers teach.
There you go.
posted by yeolcoatl at 8:27 PM on May 19, 2012
Huck500 (4:19 PM) I've talked to a lot of teachers about evaluation, and the only way we've come up with that makes sense to us is an idea that Bill Gates has endorsed... train experienced teachers to visit classrooms and watch teachers teach.
There you go.
posted by yeolcoatl at 8:27 PM on May 19, 2012
All you have to do is look at the year to year change in test scores. Or test at the beginning and then at the end. If the students don't improve, the teacher is ineffective.
I cannot emphasize too strongly that this is not all you have to do. VAM is a lot more sophisticated than that, and as the results show, it's not nearly sophisticated enough.
posted by escabeche at 8:45 PM on May 19, 2012
I cannot emphasize too strongly that this is not all you have to do. VAM is a lot more sophisticated than that, and as the results show, it's not nearly sophisticated enough.
posted by escabeche at 8:45 PM on May 19, 2012
Most thinking people in the world are moving away from the idea that standardized tests are the best way of gauging student learning, which explains why politicians are moving towards that idea.
The business of assessing individual student learning (as others here have pointed out) is largely contextual. A student who starts at zero and advances to 50 has made an enormous leap. A student who starts at 95 and moves to 97 doesn't appear to have made as enormous a leap, but may well have. Heck, a student that starts at 95 and moves down to 80 is still doing better than the kid at 50, right?
My numbers here are, of course, meaningless and say nothing about any actual teachers or students because every student is an individual and their improvement is going to be specific to them.
If you actually want to assess student learning, you need to roll up your sleeves and involve yourself in the context of specific communities, specific schools, specific classrooms and ultimately specific students. This is a lot of work and very expensive. So much easier to treat students like they are meat of different grades (which, of course, is partially where the A-F grading system came from when it was adopted in the 19th century).
Of course, as others have wisely pointed out, what we actually want our schools to accomplish varies greatly from person to person, community to community. Some people gauge success on whether their kids get into top tier colleges. Some people gauge it on how well they are prepared to have a mid-level corporate job. Some people gauge it on whether the kid becomes a well informed citizen. Does anyone really think "what I want for my student is for them to be a fabulous standardized test taker?"
By my own personal metric, if a student of mine aces the SAT but is unable to think critically, work effectively with a group, communicate clearly and work creatively, my school has failed. I don't think a student who knows a bunch of facts about history, for example, is really better than a student who can do history. You know - research, draw reasoned conclusions, know which sources to trust and which to ignore, etc.
Anyhow, metrics (and standardized tests) are useful because they make it easier for lazy administrators and agenda grinding politicians to make uninformed but smart sounding decisions. They make it easier to justify keeping the lower classes under their heels and punish teachers who try to help them. They make it easier for school counselors to pad their resume and trade up to jobs with higher salaries. Plus, they allow them to save time and money by not doing the real work of going into classrooms and getting to know their students and faculty.
What they do not do is effectively gauge universal student learning or teacher performance.
Education is not the same as working on the assembly line, grinding out code or painting a god damned picture. Listening to people compare it to these things always reminds me of the parable of "The Blind Men and The Elephant". I think because most everyone went to school they think they know how it works. Hey, I have a spleen but I'll be damned if I'm going to try and tell a doctor what the spleen does.
Bah, I am trying to keep my rants to a minimum these days, but the world is PISSING ME OFF.
posted by Joey Michaels at 12:39 AM on May 20, 2012 [4 favorites]
The business of assessing individual student learning (as others here have pointed out) is largely contextual. A student who starts at zero and advances to 50 has made an enormous leap. A student who starts at 95 and moves to 97 doesn't appear to have made as enormous a leap, but may well have. Heck, a student that starts at 95 and moves down to 80 is still doing better than the kid at 50, right?
My numbers here are, of course, meaningless and say nothing about any actual teachers or students because every student is an individual and their improvement is going to be specific to them.
If you actually want to assess student learning, you need to roll up your sleeves and involve yourself in the context of specific communities, specific schools, specific classrooms and ultimately specific students. This is a lot of work and very expensive. So much easier to treat students like they are meat of different grades (which, of course, is partially where the A-F grading system came from when it was adopted in the 19th century).
Of course, as others have wisely pointed out, what we actually want our schools to accomplish varies greatly from person to person, community to community. Some people gauge success on whether their kids get into top tier colleges. Some people gauge it on how well they are prepared to have a mid-level corporate job. Some people gauge it on whether the kid becomes a well informed citizen. Does anyone really think "what I want for my student is for them to be a fabulous standardized test taker?"
By my own personal metric, if a student of mine aces the SAT but is unable to think critically, work effectively with a group, communicate clearly and work creatively, my school has failed. I don't think a student who knows a bunch of facts about history, for example, is really better than a student who can do history. You know - research, draw reasoned conclusions, know which sources to trust and which to ignore, etc.
Anyhow, metrics (and standardized tests) are useful because they make it easier for lazy administrators and agenda grinding politicians to make uninformed but smart sounding decisions. They make it easier to justify keeping the lower classes under their heels and punish teachers who try to help them. They make it easier for school counselors to pad their resume and trade up to jobs with higher salaries. Plus, they allow them to save time and money by not doing the real work of going into classrooms and getting to know their students and faculty.
What they do not do is effectively gauge universal student learning or teacher performance.
Education is not the same as working on the assembly line, grinding out code or painting a god damned picture. Listening to people compare it to these things always reminds me of the parable of "The Blind Men and The Elephant". I think because most everyone went to school they think they know how it works. Hey, I have a spleen but I'll be damned if I'm going to try and tell a doctor what the spleen does.
Bah, I am trying to keep my rants to a minimum these days, but the world is PISSING ME OFF.
posted by Joey Michaels at 12:39 AM on May 20, 2012 [4 favorites]
All you have to do is look at the year to year change in test scores. Or test at the beginning and then at the end. If the students don't improve, the teacher is ineffective.
Or maybe they had an influx of new immigrants who struggled with the standardized tests, or maybe the major employer in town went broke and the place is slipping into poverty, or maybe they had ten special needs kids instead of three.
If you ignore context, the metric is useless. It was pretty useless to begin with, frankly.
posted by Joey Michaels at 12:43 AM on May 20, 2012 [2 favorites]
Or maybe they had an influx of new immigrants who struggled with the standardized tests, or maybe the major employer in town went broke and the place is slipping into poverty, or maybe they had ten special needs kids instead of three.
If you ignore context, the metric is useless. It was pretty useless to begin with, frankly.
posted by Joey Michaels at 12:43 AM on May 20, 2012 [2 favorites]
As a teacher in a prestigious independent school, I am very glad that my effectiveness is not measured only by achievement on standardized tests, though we do administer the tests. It's measured kid by kid, week by week, year by year. It's measured by extensive observation from my peers, by regular communication with the parents of my students, and by what I can get Ronald to do that he couldn't do last week when he came by during homeroom to work on prepositional phrases. It's measured--inadequately, of course--by the stories my former students tell about me, by the way that senior I taught in sixth grade brightens up when he sees me, by the invitation I just got to the Princeton graduation of the financial aid student I first taught in 5th grade. I'm grateful that my community is not in thrall to the idea that it is only possible to describe success by reducing it to a set of anonymous numbers. The numbers are a proxy for reality. They're a descriptor, with no more validity than any other set of adjectives or stories. They aren't reality.
A school year in a classroom is a tiny percentage of a student's life, but it is still a complex, rich world with a huge number of interlocking tasks. Some of those tasks are achieved. Some aren't. Some aren't achieved until three years later.
One of my kids said Friday while we were discussing sentence structure, "How come we forget we learned this?" and I said, "Because that's how learning goes. You learn it, and you forget it, over and over, and one day you will get to high school, the teacher will mention it, and you will think you always knew it."
As most people who become teachers learn, the reality of teaching isn't anything like the concept students (and former students) have of it, which is a shame because so many people theorizing about education today have never taught. We often hire interns who went to our school, and they are endlessly amazed to find that all those things they thought were a waste of time actually work.
posted by Peach at 5:01 AM on May 20, 2012 [2 favorites]
A school year in a classroom is a tiny percentage of a student's life, but it is still a complex, rich world with a huge number of interlocking tasks. Some of those tasks are achieved. Some aren't. Some aren't achieved until three years later.
One of my kids said Friday while we were discussing sentence structure, "How come we forget we learned this?" and I said, "Because that's how learning goes. You learn it, and you forget it, over and over, and one day you will get to high school, the teacher will mention it, and you will think you always knew it."
As most people who become teachers learn, the reality of teaching isn't anything like the concept students (and former students) have of it, which is a shame because so many people theorizing about education today have never taught. We often hire interns who went to our school, and they are endlessly amazed to find that all those things they thought were a waste of time actually work.
posted by Peach at 5:01 AM on May 20, 2012 [2 favorites]
I'm beginning to think the public school system is getting the teachers it deserves. Who in their right mind would subject themselves to this kind of public scrutiny and second hand guessing for teacher scale pay?
This is offensive to me. Firstly, the public school system may or may not be getting the teachers it deserves, but it's not the public school system being taught-it's individuals. So, do the present crop of kids in public schools deserve to have shitty teachers?
Secondly, who in their right mind would work there? Hmm, I don't know. Maybe people who have a sense of purpose or more practically, have a mortgage, kids, and/or little money or time for retraining? I would quit tomorrow but I am still waiting for my green card and my lotto winnings to take on a different job training course. (because god knows it's not really the right time to rack up 40k in student loan debt right now).
Thirdly, again, the economy is a bit fucked up right now, and the teacher pay scale, at least in NYC, if you've been in the system for awhile, is a HELL of a lot more than you would make at any entry level position, without the aforementioned re-training via student loans.
So, this comment actually really bothers me. The tone of "well then, that's that. 'They' deserve it." Just try to remember that the 'they' is actually real people. Some of them formerly idealistic teachers, the majority of those actual real people though are kids, who don't get to chose who's standing in front of them everyday if they are not from families with a certain amount of privilege or choice.
But you're right, your kid probably is better off not teaching, I will give you that.
posted by bquarters at 6:25 AM on May 20, 2012 [1 favorite]
This is offensive to me. Firstly, the public school system may or may not be getting the teachers it deserves, but it's not the public school system being taught-it's individuals. So, do the present crop of kids in public schools deserve to have shitty teachers?
Secondly, who in their right mind would work there? Hmm, I don't know. Maybe people who have a sense of purpose or more practically, have a mortgage, kids, and/or little money or time for retraining? I would quit tomorrow but I am still waiting for my green card and my lotto winnings to take on a different job training course. (because god knows it's not really the right time to rack up 40k in student loan debt right now).
Thirdly, again, the economy is a bit fucked up right now, and the teacher pay scale, at least in NYC, if you've been in the system for awhile, is a HELL of a lot more than you would make at any entry level position, without the aforementioned re-training via student loans.
So, this comment actually really bothers me. The tone of "well then, that's that. 'They' deserve it." Just try to remember that the 'they' is actually real people. Some of them formerly idealistic teachers, the majority of those actual real people though are kids, who don't get to chose who's standing in front of them everyday if they are not from families with a certain amount of privilege or choice.
But you're right, your kid probably is better off not teaching, I will give you that.
posted by bquarters at 6:25 AM on May 20, 2012 [1 favorite]
Dude, if your answer to "Who in their right mind would work there? is "People who feel trapped.", why on earth are you offended that people would ask the question in the first place?
The "they" that people think deserve the train wreck they're creating is the school administrators. You're absolutely right, though, that they're not the ones who are going to suffer the worst of it's effects though.
posted by Kid Charlemagne at 10:11 AM on May 20, 2012 [1 favorite]
The "they" that people think deserve the train wreck they're creating is the school administrators. You're absolutely right, though, that they're not the ones who are going to suffer the worst of it's effects though.
posted by Kid Charlemagne at 10:11 AM on May 20, 2012 [1 favorite]
if you have no metrics, you can't improve, and that's a fact
The fact that you have there is actually: if you have no metrics, you can't measure your improvement. That's not the same thing as saying there isn't any improvement.
The problem with improving according to metrics is that, more often than not, they people choose the ones that are cheap and easy to measure rather than the ones that measure areas where improvement is important. And the problem with making people's jobs depend on metrics is that they're not stupid, and they'll start doing things which make the metrics look better rather than doing things which make them better at their jobs.
This phenomenon is fairly well know in software engineering: Managers lacking any firm grasp of how software gets developed have imposed performance evaluations based on number of lines of source code created. Developers respond by padding out all their source files with random code copied and pasted from the internet. Software engineering books, articles and blog posts often feature some aphorisms which encapsulate this. Gilb's law:
posted by robertc at 4:10 PM on May 20, 2012 [2 favorites]
The fact that you have there is actually: if you have no metrics, you can't measure your improvement. That's not the same thing as saying there isn't any improvement.
The problem with improving according to metrics is that, more often than not, they people choose the ones that are cheap and easy to measure rather than the ones that measure areas where improvement is important. And the problem with making people's jobs depend on metrics is that they're not stupid, and they'll start doing things which make the metrics look better rather than doing things which make them better at their jobs.
This phenomenon is fairly well know in software engineering: Managers lacking any firm grasp of how software gets developed have imposed performance evaluations based on number of lines of source code created. Developers respond by padding out all their source files with random code copied and pasted from the internet. Software engineering books, articles and blog posts often feature some aphorisms which encapsulate this. Gilb's law:
Anything you need to quantify can be measured in some way that is superior to not measuring it at all.Yip’s Law:
Anything can be made measurable in a way that is inferior to not measuring it at allCorollory to Yip’s Law:
Given the choice between not measuring and using an ineffectual measurement, management will always chose the ineffectual measurement, as long as it is cheapThe problem here is not that there doesn't hypothetically exist some way of measuring teacher performance, the issue is that a way of measuring teacher performance that's actually meaningful would cost multiple times more than employing the teacher that's being measured. Is there any evidence the people doing the measuring would be willing to invest that amount of money in doing it?
posted by robertc at 4:10 PM on May 20, 2012 [2 favorites]
Is there any evidence the people doing the measuring would be willing to invest that amount of money in doing it?
Judging by the importance of "good school district" in real estate, I'm going to say yes. And taking the economic gains escabeche's linked paper measures, the cost of developing an effective measure (collecting parent's tax records) sounds miniscule compared to the upsides ("replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students’ lifetime income by more than $250,000 for the average classroom in our sample").
I will grant you that it is a "working paper" and as such has not been peer reviewed. Not that I have any particular faith in the peer review process, but they do tend to be more widely read and thus rebutted.
posted by pwnguin at 4:49 PM on May 20, 2012
Judging by the importance of "good school district" in real estate, I'm going to say yes. And taking the economic gains escabeche's linked paper measures, the cost of developing an effective measure (collecting parent's tax records) sounds miniscule compared to the upsides ("replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students’ lifetime income by more than $250,000 for the average classroom in our sample").
I will grant you that it is a "working paper" and as such has not been peer reviewed. Not that I have any particular faith in the peer review process, but they do tend to be more widely read and thus rebutted.
posted by pwnguin at 4:49 PM on May 20, 2012
Just as a thought, which I'm sure is crazy, we could maybe measure... fewer but better things?
posted by DarlingBri at 4:56 PM on May 20, 2012
posted by DarlingBri at 4:56 PM on May 20, 2012
The goal of standardized testing is a good and valid one. It can be very useful to try and figure out how effective teachers and school systems are at doing their job of educating children. That's a good thing to know and while you might never be able to get perfect information on how good millions of people are at their jobs you can probably get some idea if you're smart about it.
There exist tests that are sufficiently difficult to game that doing well is correlated with actually knowing the material. Tests like the subject specific GRE, many of the AP tests seem to do a good job reflecting knowledge of the material rather than knowledge of the test. I think a big part of how they are able to do this is by being legitimately challenging for most students.
From what I've been able to pick up from these tests the questions aren't particularly challenging, the material covered is very narrow, what's covered is very predictable and consistent from test to test, it also is a bit jargony for my tastes asking questions about things like 'counting numbers' which is what I gather they are currently calling natural numbers these days. I guess this makes sense; the implied goal: "No Child Left Behind" which seems to be aimed at insuring that everyone reaches a minimum standard of competency. I suppose that's a worthwhile goal but given the limited resources available for educating children its likely to have what I think is too high a price. There are many children for whom the goals of the test are either insultingly trivial or unrealistic and will get an inferior 'standardized' education to maximize their score on this standardized test.
posted by I Foody at 8:34 AM on May 21, 2012
There exist tests that are sufficiently difficult to game that doing well is correlated with actually knowing the material. Tests like the subject specific GRE, many of the AP tests seem to do a good job reflecting knowledge of the material rather than knowledge of the test. I think a big part of how they are able to do this is by being legitimately challenging for most students.
From what I've been able to pick up from these tests the questions aren't particularly challenging, the material covered is very narrow, what's covered is very predictable and consistent from test to test, it also is a bit jargony for my tastes asking questions about things like 'counting numbers' which is what I gather they are currently calling natural numbers these days. I guess this makes sense; the implied goal: "No Child Left Behind" which seems to be aimed at insuring that everyone reaches a minimum standard of competency. I suppose that's a worthwhile goal but given the limited resources available for educating children its likely to have what I think is too high a price. There are many children for whom the goals of the test are either insultingly trivial or unrealistic and will get an inferior 'standardized' education to maximize their score on this standardized test.
posted by I Foody at 8:34 AM on May 21, 2012
Judging by the importance of "good school district" in real estate, I'm going to say yes.
I wasn't aware that any of the people doing the measuring were in real estate, I thought they were in politics. What is the value add for them between 'pretending to have good schools' and 'having good schools'? Is it something that could get them re-elected after hiking taxes?
posted by robertc at 1:49 PM on May 21, 2012
I wasn't aware that any of the people doing the measuring were in real estate, I thought they were in politics. What is the value add for them between 'pretending to have good schools' and 'having good schools'? Is it something that could get them re-elected after hiking taxes?
posted by robertc at 1:49 PM on May 21, 2012
robertc: "Is it something that could get them re-elected after hiking taxes?"
In my experience growing up in suburbia, ballot measures raising sales tax to benefit local public school districts succeed overwhelmingly.
posted by pwnguin at 6:21 PM on May 22, 2012
In my experience growing up in suburbia, ballot measures raising sales tax to benefit local public school districts succeed overwhelmingly.
posted by pwnguin at 6:21 PM on May 22, 2012
« Older Turkey Dubstep | At least it was here Newer »
This thread has been archived and is closed to new comments
posted by michaelh at 6:59 AM on May 19, 2012