Tags:

Teachers boycott standardized testing
January 15, 2013 8:29 AM Subscribe

Teachers at two Seattle high schools have decided to boycott a district-required standardized test.

The boycott began at Garfield High School, and has since spread to Ballard High School. All teachers at both schools support the boycott. (Here are the thoughts of one teacher; here's a former student and current education reform advocate.)

The required Measures of Academic Progress (MAP) exams, which test reading and math, are given two to three times a year. But one teacher spokesman said the test “produces specious results, and wreaks havoc on limited school resources” during the weeks the test is administered.

posted by showbiz_liz (99 comments total) 23 users marked this as a favorite

It will be interesting to see how these struggles change with the shift to the common core standards.
posted by JimmyJames at 8:37 AM on January 15

Funny how people always want to blame the people who aren't intimidated by threats when the people who make the threats carry them out. The way I see it, if the state or the federal government cuts funding to these schools, the state or federal government is the party with agency and the party on whom the blame should fall.
posted by enn at 8:39 AM on January 15 [32 favorites]

They made a pretty hard but admirable choice. I would be uneasy as a teacher that my actions would negatively impact my students, but then there's the idea that the nationwide student body will benefit if these stupid, stupid tests are thrown in the garbage.
posted by angrycat at 8:46 AM on January 15 [4 favorites]

Whoa. I went to Garfield. I wonder if many of the teachers from when I was there are still there. In any case, I am super, super proud of my high school. which is not the sort of sentiment I normally express. Go Bulldogs!
posted by You Can't Tip a Buick at 8:47 AM on January 15 [8 favorites]

enn, that wasn't my point at all. My point is that perhaps they should be working with state and federal legislators.
posted by roomthreeseventeen at 8:47 AM on January 15

I don't know anything about the MAP exams other than what I've read in this post, but giving standardized tests "2-3 times a year" seems a bit excessive to me.

The students are "sitting there feeling really stupid, saying, ‘I don't know how to do this,'" McBride said.

I do wonder, though, if this is a failure of the test, or a failure of education.
posted by madajb at 8:50 AM on January 15 [1 favorite]

I wish teachers in Georgia would make a stand like that. Unfortunately, they apparently have decided it is easier to just cheat.
posted by TedW at 8:51 AM on January 15

I do wonder, though, if this is a failure of the test, or a failure of education.

In a word, "Yes."

Teachers have to teach. But they also have to have their kids get good test scores, which means that they're under pressure to teach to the test, and not really teach, generally. Because teaching kids to excel at a test and giving a kid a good education don't really have enough overlap. Teachers are stuck between a rock and a hard place with this, and when kids are stymied by the process, I think the system, not the teachers, is to blame.
posted by entropone at 9:03 AM on January 15 [17 favorites]

The boycott seems like a terrible idea to me if state or federal funding is based on test scores. However imperfect the tests may be, if you're going to have less money coming into the school, that's never good.

this is pretty much the same argument liberals make when people murmur about creating a 3rd party in American politics. Sacrifices need to be made, sometimes that sacrifice is an entire generation. Until we start making decisions as a country in the context of 100 years, progress will forever be retarded.

When you tell a teacher their job depends on test scores they can't possibly produce you put them in a position of making a decision between cheating or being unemployed. The best case scenario that can come from this is that the teachers abandon the curriculum and spend the entire year teaching for the test. This is what the "best" schools do and this serves no one, not the students, not the teachers and not society. For more damning evidence of this see last week's Frontline episode on Michelle Rhee.
posted by any major dude at 9:04 AM on January 15 [11 favorites]

mattbucher: "Where is the outrage over the SAT, ACT, GMAT, GRE, LSAT, MCAT? Where is the outrage over tests in college? You want kids to be college-ready but not have to take difficult tests? What if your potential employer asks you to take an assessment? Is that worthy of outrage? Tests are a part of life. They should probably be scaled back in K-12, but I don't think they should be boycotted or eliminated."

Point being, I'd rather have an intelligent son who can perform well on tests, rather than a son who is simply proficient at test-taking.
posted by boo_radley at 9:04 AM on January 15 [11 favorites]

My point is that perhaps they should be working with state and federal legislators.

What makes you think they haven't been trying?

Anyway, "working with" legislators inevitably involves using collective power to put pressure on them from time to time. No body that has power over you will ever really work "with" you except under duress.

You want kids to be college-ready but not have to take difficult tests?

All the tests I remember taking in university were written by the teachers.
posted by regicide is good for you at 9:05 AM on January 15 [13 favorites]

Reason #34281 I am proud to live in Seattle. This place is as close to utopia you will ever find within U.S. borders. There are a number of Mefites with ties to Garfield and Ballard and hopefully they can chime in, but these are not known as failing schools. Garfield is where our kids would go if we don't go private and among work colleagues and neighbors, Garfield seems like a very viable option when compared to private schools.

oh god please don't turn this into a private vs public thread
posted by Slarty Bartfast at 9:05 AM on January 15 [3 favorites]

Where is the outrage over the SAT, ACT, GMAT, GRE, LSAT, MCAT? Where is the outrage over tests in college? You want kids to be college-ready but not have to take difficult tests? What if your potential employer asks you to take an assessment? Is that worthy of outrage? Tests are a part of life. They should probably be scaled back in K-12, but I don't think they should be boycotted or eliminated.

The teachers actually specifically state that they will continue to administer the end-of-year standardized tests, which are apparently a completely different thing from the MAP tests. From the Reuters link:

The MAP test that has become a point of contention at Garfield is given at schools around the country but is not required by Washington state.

Unlike the tests required by the state, which are the High School Proficiency Exam and the End-of-Course exams, it has no bearing on students' grades or their ability to graduate.

Kris McBride, the school's testing administrator and a supporter of the 19 teachers, said the instructors do not object to all standardized testing.
posted by showbiz_liz at 9:07 AM on January 15 [3 favorites]

My school district uses tests. The test scores are so important that instruction on anything that isn't THE TEST shuts down for weeks before. The kids spend week after week doing multiple choice practices.

Measurement can be good. But the way we reward schools on the basis of this measurement gives them enormous incentive to put the skill of number-two pencil coloring inside of ovals ahead of all other subjects.
posted by zippy at 9:09 AM on January 15 [7 favorites]

Where is the outrage over the SAT, ACT, GMAT, GRE, LSAT, MCAT? Where is the outrage over tests in college?

You don't take the SAT three times a year. Tests in college, at least good colleges, require one to be proficient at something besides multiple choice.

As zippy mentioned, testing shuts everything down. Spending two or three months out of a nine month school year teaching testing strategies instead of actual content is a horrible strategy for the long term education of a student, but one that the schools are expected to go through in order to get funding.
posted by dinty_moore at 9:15 AM on January 15 [5 favorites]

I live in a neighborhood in Minneapolis with one the highest rated public schools in the entire district. I recently got an email from my 1st grade daughter's teacher asking if parents could help out with grading homework. I can't tell you how bizarre it feels to have a first grader who even has any homework, let alone a load so high the teacher can't stay on top of it.

Here's my proposed education reform:
- scale this endless standardized assessment b.s. way the hell back
- pay teachers more, and have more of them
- year-round school, with only a short summer break of 1 or 2 weeks

I'd be willing to bet that last point would do more to raise the bar of U.S. K-12 education than any of the crap we've thrown at it in the last decade.
posted by mcstayinskool at 9:17 AM on January 15 [10 favorites]

Well, it's not the state or the federal government mandating these benchmark tests - it's the local school district. I applaud school districts that have the courage to trust their kids are learning without having to constantly administer optional benchmark tests anyway.
posted by mattbucher at 9:25 AM on January 15

Seattle teacher here -- I sub, so I've gotten around to all the high schools, and I spend a LOT of time at Ballard in particular. I'm kinda thrilled at this.

As noted, this isn't a protest against ALL standardized testing. Teachers generally find those annoying and they're frequently a waste of time, effort and money, and they cause huge disruptions... but NONE of them are as wasteful and meaningless as the MAP test. Regardless of one's feelings about standardized testing in general, MAP is worthless and probably the most disruptive over the course of a year. And, as noted, students generally know it doesn't count for anything, so they don't take it seriously.

With regards to working with legislators: Again, this is about a particular test, and this was adopted by the DISTRICT, not the STATE.
posted by scaryblackdeath at 9:28 AM on January 15 [11 favorites]

You want kids to be college-ready but not have to take difficult tests?

All the tests I remember taking in university were written by the teachers.

I took one multiple choice test in the whole of university, and that was for a very easy elective. We did 2-3 hour essay based exams instead (humanities).
posted by jb at 9:28 AM on January 15 [2 favorites]

Those tests are not taken two-three times a year, every year. A typical American student going to college will take the SAT or ACT one or two times, maybe three if PSATs and things are included. Then they may, years later, take one of the GMAT/LSAT/MCAT/GRE. That is not the same as training for three tests a year.

Many Seattle high school students (I'm not sure if it's universal yet) take the PSAT for free as sophomores and juniors. I distinctly remember having to pay for the privilege when I was in high school in CA in the '90s.
posted by scaryblackdeath at 9:29 AM on January 15

As Slarty Bartfast notes: these two schools are NOT "failing" schools by any stretch. Ballard is... well, pretty much one of my favorite schools. It has high-achieving students, several great science programs, it's a safe place... I could blab about how awesome Ballard is all day long.

Garfield is a place full of big contrasts. You see a huge amount of high-achievers AND a lot of struggling students. There are lots of college-bound kids with schedules crammed with AP classes and a nationally-competitive music program and there are kids with gang ties and constant behavioral problems. I am not at all surprised that Garfield saw huge problems with this test, because they're in a position to see a really wide spectrum of results.
posted by scaryblackdeath at 9:37 AM on January 15 [5 favorites]

As a country, we must hope that the bravery these two show is contagious.
posted by stoneweaver at 9:42 AM on January 15 [1 favorite]

Didn't the Simpsons already do this?
posted by Renoroc at 9:42 AM on January 15 [1 favorite]

Testing coordinator at a charter school here. Although I work in California, our school has also adopted the MAP, but has rarely if ever been able to implement it well.

The MAP was designed by the Northwest Evaluation Association (NWEA). It's an adaptive test, meant to measure the upper limits of a student's knowledge. The more they score correct, the more advanced the questions become.

It's supposed to be given two, if not three times a year. In this way, it's supposed to be able to measure a student's growth over the course of the year. NWEA collects all the data from the participating schools, which they use to create cutlines and norms for each grade-level. Schools across the country use this test, so the norms are closer to a nation-wide comparative measure than the state tests each state develops.

If a school was really serious about data-driven instruction, I can see how this would be a valuable tool. But I understand and agree with many of the teacher criticisms in the final linked article. It's disruptive. It's seemingly unconnected to any of the specific curriculum in the classroom. There are no sample items for teachers to examine and potentially test to. As some who also teaches several sections in addition to my admin responsibilities, I know I wouldn't want NWEA MAP used to evaluate my effectiveness.

I coordinate and give all these standardized tests on campus because that's my job, but as a teacher, I'm not happy about it. Thankfully, with the new common core state standards and the SMARTER Balanced Assessment coming down the pike, it looks like education is moving away from mind-numbing multiple choice. Some schools are also looking at moving away from annually testing each grade level, and saving the assessment for say, junior year in high school.
posted by mdaugherty82 at 9:46 AM on January 15 [5 favorites]

Tests are a part of life. They should probably be scaled back in K-12, but I don't think they should be boycotted or eliminated.

I'll start by saying that this is less about Seattle specifically, and more about the problems with testing. Let's say that I am a researcher and I came up with a blood test that will determine if you get a disease (call it MetaFilter Disease, or MFD). That is all the test does. A doctor takes a sample, applies the test, and determines if the disease is present. That's it. Does patient X have MFD.

Now, let's say that someone in the government decides that doctors do not face sufficient accountability. This person want to institute some kind of measure to look at how well doctors do their jobs. No one, except maybe the doctor's themselves (suspicious, eh?), would be against this. Why would anyone be?

Interestingly, the same person seeking greater government oversight has two key attributes:
1. they want to do oversight very cheaply, and
2. they have received millions in campaign contributions from the company that bought the right to my MBD test.

You see where this is going. The State Health Office decides that doctor's who are more effective will obviously find more patients that have MFD. If they aren't finding MFD, then clearly they aren't treating enough people.

This is a long way of saying, the test is intended to determine something specific. Using the same test to describe something the test wasn't designed to test is a complete misuse of the test.

Ability tests test student abilities. They are not normed to test teacher effectiveness. The reason why States use these test as proxies is because its cheap and test publishers are politically well connected. But you wouldn't use an HIV test to test for Medicare fraud if you were interested in finding fraud.

Honestly testing for teacher effectiveness would probably be very expensive and would have to involve a whole lot of research. This is so far away from what the testing supporters want out of "accountability" that it makes me truly wary of their intentions.
posted by Hypnotic Chick at 9:46 AM on January 15 [29 favorites]

As a country, we must hope that the bravery these two show is contagious.

I would like to sound one note of caution against getting our hopes up: the MAP test kind of goes out of its way to be worthless and, again, was adopted under a seriously unethical situation by a superintendent who was shown the door in 2011 over other ethics issues.

America may be split on how far we want to go on gun control, but it's not too hard to get most people behind a ban on civilian ownership of grenade launchers, y'know?
posted by scaryblackdeath at 9:49 AM on January 15 [1 favorite]

An example of its worthlessness: if you get a question wrong, the next question the MAP test gives you is almost certainly going to be easier. Students can and have figured this out, and thus they deliberately throw some questions to get it easier. Not sure if that's in the links provided, but it's known by more than a few students in the district. And, again, they all know that this test counts for nothing with regards to grades or graduation or college eligibility.
posted by scaryblackdeath at 9:52 AM on January 15

The boycott seems like a terrible idea to me if state or federal funding is based on test scores. However imperfect the tests may be, if you're going to have less money coming into the school, that's never good.

It's a perverse situation where, the worse your school does, the less aid it gets. Now, before the market force people go all reward the winners ... the test here is a diagnostic of problems, and problems cost money to address. Budgets around here are mostly consumed by expenses you can't waive away (heat and a/c, building maintenance, salaries).

I would guess perhaps a quarter to a third of the instruction in the nearby public school is done by temps, part-timers, and volunteers. And much of the in-class supplies comes from parent donations. I'm talking basics: paper, pencils, white board markers, glue (elementary school).

And there's one manager (the principal) for ... maybe 50 staff members?

It is a lean organization. It is still a bureaucracy, but in terms of number of layers of management, there's not a lot.

So take this school as representative. They're facing the test, their funding is already so low they can't really keep up with building repairs, so low they have to send layoff warnings to experienced teachers every summer, so low they can't afford paper ... and if their students do poorly on the test ...

... the school gets less money for paper and teachers and heat and light.

So of course they go nuts over this test. Those kids who are marginal readers are damn well going to be able to muddle through a three sentence comprehension question after a few weeks of focus on three sentence stories.

Hooray, now the school has more money! Maybe next year they can focus on real reading skil...oh, wait, there's another test that year. And another the next. And another the next.

So the school transforms itself from a place of general education to a Kaplan test prep center most of the year. The winners of this are makers of pencils, Scan-tron machines, and multiple choice tests. The losers are: kids, teachers, parents, and society.
posted by zippy at 10:19 AM on January 15 [5 favorites]

A friend of mine has children in these schools, and is thrilled, THRILLED about the opt-out decision. Apparently, the superintendent who chose to implement it sat on the board of directors of the company who produced it, and it's very expensive to administer; it costs funds that could be spent on, you know, music and art and resources for students with special needs. The test isn't linked to the school district standards, so kids are tested on material that might not have been presented to them. And since it's computer-administrated, kids who aren't as computer literate do dramatically, disproportionately badly on it.

I like data-driven metrics as much as the next person, but this is not a great system.
posted by KathrynT at 10:22 AM on January 15 [5 favorites]

I went to Garfield!

I'm not a fan of standardized tests, but I also think that there are much much much bigger fish to fry in the Seattle school system, which is unbelievably screwed up. This is pretty much like an Enron executive resigning to protest health code violations in the company cafeteria. I wish the teachers were fighting over something that matters.
posted by miyabo at 10:41 AM on January 15

What if your potential employer asks you to take an assessment? Is that worthy of outrage? Tests are a part of life. They should probably be scaled back in K-12, but I don't think they should be boycotted or eliminated.

Productive working member of society here! The last test I took was in December of 2005. It was my final exam for my compiler design class. It went OK. It in no way resembled any standardized test I ever took in primary or secondary school, and it certainly didn't resemble any working day I have ever had in an office.
posted by Mayor West at 10:47 AM on January 15 [4 favorites]

No matter how much some of us (myself included) may want it to be, the current boycott is not about school testing. It is about this test. Purchased by a corrupt superintendent from a company that she sat on the board of, without disclosing that affiliation, without any review. She is currently being investigated by the state auditor. In the meantime, 30,000 students (approximately) are forced to take what amounts to a couple weeks out of their school year.

I went to Garfield. My brother still goes there. In his time in Seattle Public Schools, he's taken a MAP test 24 times. MAP testing shuts down the school library for 35% of the school year. For kids without computers at home (which is many, many kids at Garfield), that's 35% of the year when they can't access the internet. Check their email. Do a school project. Turn in homework that must be typed. That, in of itself, is enough reason to boycott the test. And there are many, many other reasons: students not taking the test seriously, because the results do not matter (case in point: my brother topped out the test when he took it in 7th grade. In 9th grade, he failed it. Because he wanted to go back to class.); my brother's failing grade likely being used to evaluate his teacher; test material not matching Seattle Public Schools curriculum, leaving students to feel like failures.

Here's why to oppose this test.

Seattle Public Schools has spent, by some estimates, $10 million on this test. When I was at Garfield, I had teachers who asked wealthier students to bring in reams of paper, so that they could make copies of their finals. Math class was BYO graph paper. Our globes still had the Soviet Union. I had an assigned seat in a windowsill. After my freshman year at Garfield, the district ran out of money, and started laying off teachers, based on seniority. Garfield lost every teacher under 35. By the time they came up with some money for them in the fall (in some cases, after school had already begun), most of them had made other plans.

This is not the opening salvo in a fight against school testing. This is another match in the fight against the bureaucracy and corporate interests in the American education system that prevents good teachers from doing their job, and keeps bad teachers in the classroom. Against school boards more beholden to their corporate boards than kids.

Tangent: Kris McBride and Kit McCormick, both quoted in a variety of press releases, were my teachers, and they are both badasses who would eat glass if it would help their students learn. Same goes for most of the other teachers involved in the boycott. Garfield is not an easy place to teach. They are what make it a place that people want to send their kids.

McBride was so passionate about AP Statistics that each year after the AP she would hold a contest for the best poster to promote the benefits of taking statistics to the incoming senior class. Then, she would advertise. That's right, an overworked, underpaid high school math teacher literally leafleted the school with an ad campaign to encourage students to enroll in a math class senior year, because it might be of benefit to them in the future, whether they went to college or not. She bribed my precalculus class with baked goods. She was available for tutoring before and after school, even when you weren't in her class. She offered special tutoring for the SAT.

TL;DR: This is not about school testing. This about an expensive, useless, misused test originally purchased under unethical circumstances. And about fabulous, heroic teachers doing their jobs -- standing up for their students.
posted by femmegrrr at 10:51 AM on January 15 [144 favorites]

What if your potential employer asks you to take an assessment? Is that worthy of outrage?

Hell yeah. Death to false objectivity.
posted by MartinWisse at 10:56 AM on January 15

Whoa. I went to Garfield. I wonder if many of the teachers from when I was there are still there. In any case, I am super, super proud of my high school. which is not the sort of sentiment I normally express. Go Bulldogs!

My daughter went to Ballard! Go Beavers!!!
posted by Windopaene at 11:09 AM on January 15 [1 favorite]

Speaking as someone who has scored standardized tests from almost every school in the State of Washington, including the fantastically wealthy Garfield and Ballard High Schools, I have a message for the boycotting teachers:

Fuck you. You have absolutely no justification to whine about your budget being stretched thin, and that you don't have enough classroom time to waste on standardized tests. If you don't like doing your job, I suggest you go work at some other school, like one of the remote, rural Native American Reservations. Then you can learn what it means to be a teacher in a school with an inadequate budget.

I still remember the first time I scored the WA tests. The students from wealthy urban Seattle schools wrote about topics like Ayn Rand and libertarianism. Then I'd get essays from Native American kids living on a remote Reservation, they wrote about their abusive parents and how sometimes they had to miss school so they could go out hunting because they didn't have enough food to eat.

The tests aren't given for the benefit of the affluent students who can pass them, and whose families can afford private tutoring if they can't. The tests are (in part) given so the entire educational system can understand the entire spectrum of students' abilities. The tests indirectly study the teachers, the schools, and the curriculum. Sometimes, sacrifices must be made, a proficient student must "waste" an hour or two taking tests he could easily pass, time that he might feel could be better spent reading another chapter of Atlas Shrugged. But if the wealthiest schools and students all refuse to take the test, the remaining students who do take the test will give a skewed result. It will make it seem that educational achievement is much harder than it really is, and that everyone's goals should be adjusted downwards.
posted by charlie don't surf at 11:11 AM on January 15 [3 favorites]

These tests are not reasonable assessments of anything and if the privileged schools are the only ones with the freedom to take the lead on opposing them, it is absolutely their responsibility to do so.
posted by You Can't Tip a Buick at 11:16 AM on January 15 [2 favorites]

charlie don't surf- have the results of those tests actually resulted in any changes, on any level of the education system? Because it seems like these sorts of tests generally serve to keep money AWAY from low-performing schools. I think we can all agree that the disparity of educational opportunity is bad, but what role do standardized tests have to play in fixing that?
posted by showbiz_liz at 11:17 AM on January 15 [9 favorites]

RE: tests in general, I found this link from this Ask answer earlier today. While the majority of it does not apply here, one section does:

The most meaningful evaluations in life are:

Completely unexpected.

Totally comprehensive. Absolutely everything you ever learned could be included.

Include material you never studied and maybe never even heard of.

Standardized tests are none of these, especially when teachers get pushed into a corner and are forced to "teach to the test" (as others have said).
posted by King Bee at 11:19 AM on January 15 [1 favorite]

Wow, charlie don't, that's an intense contrarien rant. Are you defending this specific MAP test, or testing in general? I didn't go to Garfield but it seemed more inner city than privileged, there seem to be a fair number of semi-urban schools that are very mixed and inner-city or privileged depending on who is looking for what. (Cambridge Ma is another example, kids of Harvard profs mixed in with the kids from the projects) Seems quite hard on the staff.
posted by sammyo at 11:21 AM on January 15 [2 favorites]

Charlie Don't Surf: You are looking for enemies where you have none. I guarantee you that the teachers at Garfield and Ballard would be HAPPY to do things that would help the poorer and less fortunate schools in the state like the Reservation schools.

Show them evidence that participating in the MAP test does that.

I've scored tests from around the country, too. I've worked in wealthy schools and poor schools. Turns out rich kids have problems and challenges, too, just like poor kids. Any reasonable person might prefer to confront the challenges the rich kids face, but school can suck for pretty much anyone. Nobody has a monopoly on caring about kids, so get off your high horse.

This particular test, again (as noted by many above and the teachers at Garfield and Ballard) is an expensive, colossal waste of time and money best spent elsewhere. It was a blatant boondoggle foisted upon the district by a superintendent who was caught committing OTHER blatant boondoggles and was fired for it. (She passed away just a couple months ago, btw, so no use trying to hold her accountable now.)
posted by scaryblackdeath at 11:25 AM on January 15 [13 favorites]

Well just to stir things up billg has this covered: Gates Foundation study: We’ve figured out what makes a good teacher
posted by sammyo at 11:26 AM on January 15

sammyo: Garfield, in particular, is both inner city poverty and wealthy/privileged. It all depends on which classroom you're in at what time of day, or where you hang out at lunch. Garfield kids have been shot in drive-bys and they've gone off to college on mom & dad's dime. You get both ends of the spectrum.
posted by scaryblackdeath at 11:27 AM on January 15 [2 favorites]

*groans* ooohhhhh god, don't get me started about the Gates foundation....
posted by scaryblackdeath at 11:28 AM on January 15 [2 favorites]

Charlie Don't Surf: Um... how does forcing teachers in places like the Reservation to teach to tests that don't actually measure anything useful and then penalizing them with budget cuts when they don't do well and so aren't able to improve help those schools, again?

Related to the rest of the thread: These fuck up specialized programs for special needs students too. I remember hearing stories about how the special ed teachers at my high school had to spend weeks coaching their students not on how to take the test, but on general self-esteem stuff, because kids with learning disabilities were given the tests that everyone else got. Some of them were impossible to take. Same deal with ESL students who didn't speak English well. There were, IIRC, numbers of students that had to show up for the test for funding reasons, even if they couldn't read the test because they didn't speak English or if their disabilities made it extremely difficult for them to do them. They counted that number for individual programs, so you had to have a certain number of special ed students show up, and since it was such a small program they had to dedicate class time to encouraging the kids to show up so that they got the number of butts in seats they needed. That's got to be even harder to do at schools that are more budget-stressed than the one I went to.
posted by NoraReed at 11:28 AM on January 15

have the results of those tests actually resulted in any changes, on any level of the education system?

Absolutely. I can't speak to the WA State school system, but educational test providers also create what they call "professional development." These are materials intended for teachers, to help them teach certain skills that the student test results showed they did poorly at. Test results are also usually considered when developing the curriculum, and when writing textbooks.

This is the whole point of testing. It's not just a way to subject students to needless pain. It's a way to assess the schools and the curriculum, as well as the students. It's a way to find the areas where the students need help, what teaching methods and materials to use to help them, and to determine if that help was effective.
posted by charlie don't surf at 11:30 AM on January 15

The original letter from the teachers focuses on students who don't have access to a lot of resources.

"In an appeal of the Board’s 2010 decision to renew the MAP contract, a parent group raised concerns about the negative impact of this test “on non-English speakers, Special Education students, and minority and low income children.” These concerns were never addressed nor were the claims refuted. Imagine a native Somali student with limited English skills, sitting in front of a computer taking an evaluative reading test that will no doubt be confusing and overwhelming to the student. The test is supposed to determine the student’s reading level, but without taking into account the student’s language challenge or the student’s limited time in the United States, which makes it almost impossible to understand the context of some passages. For these students and our students with IEPs, the test does actual harm. The students feel stupid yet are being forced to take a test that has NO benefit to them or their educational goals. We object to a test that may violate the rights of groups of students for whom schooling already constitutes an uphill battle."

"Ninth graders and students receiving extra support (ELL, SPED, and students in math support) are targets of the MAP test. These students are in desperate need of MORE instructional time. Instead, the MAP test subtracts many hours of class time from students’ schedules each year. If we were to participate this year, we would take 805 students out of class during 112 class periods. The amount of lost instructional time is astounding. On average students would EACH lose 320 minutes of instructional time. This is over 5 hours of CORE class time (language arts and math) that students are losing. We object to participating in stealing instructional time from the neediest students."

"In addition to students losing class time to take the test, our computer labs are clogged for weeks with test taking and cannot be used for other educational purposes. For example, students who have a research project no longer have access to the computers they need to further their exploration into their research topic. This especially hurts students without computers at home. We object to our educational resources being monopolized by a test we cannot support."

As for the idea that this test is a good way to asses student learning,

"Seattle Public School staff has notified us that the test is not a valid test at the high school level. For these students, the margin of error is greater than the expected gain. We object to spending time, money, and staffing on an assessment even SPS agrees is not valid."

"We are not allowed to see the contents of the test, but an analysis of the alignment between the Common Core and MAP shows little overlap. We object to our students being tested on content we are not expected to teach."

"Even the NWEA itself, the parent company to MAP, has advised districts to carefully restrict the use of the test and its results. NWEA also cautions to ensure 100% random selection of students enrolled in any course if the test is used for evaluation and to take into consideration statistical error in designing evaluation policies. NWEA says that problems become “particularly profound at the high school level.” None of these or other criteria urged by NWEA has been met. We object to being evaluated by a test whose author suggests extreme caution in its use and warns against valid legal action if the test is used in personnel decisions."
posted by femmegrrr at 11:34 AM on January 15 [5 favorites]

educational test providers also create what they call "professional development." These are materials intended for teachers, to help them teach certain skills...

Of course they do, but I'd suggest that's part of the problem. Have the test publishers done the research that would justify the secondary products they are selling? What's more, is it sufficient to realy on an assessment that is not normed for the purpose in which it is being applied?

In light of the Bill Gates link above, it would seem that the assessment of teacher effectiveness would require more inputs than what any student test could offer.
posted by Hypnotic Chick at 11:38 AM on January 15 [2 favorites]

This is the whole point of testing. It's not just a way to subject students to needless pain. It's a way to assess the schools and the curriculum, as well as the students. It's a way to find the areas where the students need help, what teaching methods and materials to use to help them, and to determine if that help was effective.

Good tests do that. Bad tests do nothing but cause anxiety and waste multiple testing days, which in turn wastes even more time spent preparing for a test instead of, you know, teaching. I'm pretty sure the teachers are standing up for their right to do their job, i.e., teach their students. Not waste millions of dollars on an inefficient, overused test. Do you somehow that the MAP test will cure their students of their love for Ayn Rand?
posted by jetlagaddict at 11:40 AM on January 15 [1 favorite]

People interested in this issue might like to read Tested by Linda Perlstein. Perlstein chronicles a year at a Maryland elementary school that is, by testing measures, succeeding. It's fascinating. I read it for a class in getting certified to teach K-6 in Washington state. She also has a pretty good blog on education issues.

The book left a very strong impression on me that was then reinforced by my time in our public schools here: teachers and principals (largely) have little problem with the idea, in general, of standardized testing. It's a good way to accumulate data. But in the last decade--since NCLB, basically--testing took on a disproportionate importance. It's this absurd level of importance that is so alarming, not the basic idea of the tests themselves. I remember taking standardized tests when I was in school a few decades ago, and it wasn't a big deal. Now it's a huge deal, and making it into a huge deal has probably been counterproductive.
posted by MoonOrb at 11:49 AM on January 15 [3 favorites]

Charlie Don't Surf: Um... how does forcing teachers in places like the Reservation to teach to tests that don't actually measure anything useful and then penalizing them with budget cuts when they don't do well and so aren't able to improve help those schools, again?

Logical fallacy: assumes what you are trying to prove. "If the sun didn't shine, wouldn't it be dark?" Loaded question, "If I murdered you, how would you like that?" Persuasive essay score: zero.

I remember hearing stories about how the special ed teachers at my high school..

And who did you hear these second-hand stories from?

These "modified assessments" are one particular type of test I've worked on a LOT. Special Needs students are NOT given the same test, at least, they may be given the same questions, but the criterion for judging the tests are modified. The tests measure how well these students perform compared to the entire student population, by measuring them separately. ESL students also need accurate measurements. I've seen smart ESL students give essays that would fail the regular tests, but get very high scores (aside from grammar and mechanics) on the Modified.

On preview, from Hypnotic Chick:

Have the test publishers done the research that would justify the secondary products they are selling?

Yes, that's the whole point. The test is part of that research. You know, tests don't just spring up out of thin air. People have to write the questions, test the proposed questions by giving experimental tests to sample student populations, and test the scorers to see if they can accurately and consistently assess the answers. New tests are almost constantly in development, they write a new test for next year while the current year's test is being given.
posted by charlie don't surf at 11:56 AM on January 15

Reason #34281 I am proud to live in Seattle. This place is as close to utopia you will ever find within U.S. borders.

Believe it or not, there are similar-sized cities with competently managed schools where every student can get a college-prep education, not just the few percent who get into a gifted program (Garfield) or whose parents can afford private school. There are worse places than Seattle where education is concerned, but there are also far, far better.
posted by miyabo at 11:56 AM on January 15 [4 favorites]

And, I mean, I see what you did there with the Ayn Rand stuff - I was at Garfield right in the middle of the first tech boom, and there were a number of morons there into libertarianism. But they were well outnumbered by intelligent, politically engaged, and ethical people. And, look, Garfield is an instance of what Jonathan Kozol has called a "zebra school", which is what happens when school districts attempt to fake integration by sending disproportionately white and asian "advanced programs" to disproportionately black and latin@ schools. Some of the white stripes on the zebra were stupid libertarians, but most of the people in the school (on both sets of zebra stripes) were smarter than that. The Seattle == rich == libertarian move is, well, more than a bit sloppy.
posted by You Can't Tip a Buick at 11:57 AM on January 15 [7 favorites]

Yes, that's the whole point. The test is part of that research. You know, tests don't just spring up out of thin air. People have to write the questions, test the proposed questions by giving experimental tests to sample student populations, and test the scorers to see if they can accurately and consistently assess the answers. New tests are almost constantly in development, they write a new test for next year while the current year's test is being given.

You do realize that not everything churned out by the professional development industry is worth the paper it's printed on, right? Nor are all tests of equal worth and effectiveness. Why is it difficult for you to believe that this particular test is bad?
posted by scaryblackdeath at 12:03 PM on January 15 [4 favorites]

Yes, that's the whole point. The test is part of that research. You know, tests don't just spring up out of thin air.

I have no doubt the tests are normed with students. I'm asking if they are normed with teachers.
posted by Hypnotic Chick at 12:03 PM on January 15

charlie:

Maybe you have stumbled into the wrong thread... This isn't a thread against testing in general, or against standardized tests in general... This thread is about a specific test that apparently eats up a lot of resources and apparently was adopted under some ethically dubious circumstances.

Honestly I find it troubling that you may have been involved with scoring tests (particularly essays) as your reading comprehension skills (in regards to this thread) seem a bit thin.

To be fair to Charlie, although I had a hard time reading in a non-biased way a post that starts out by saying "fuck you" to teachers, the post title is "Teachers boycott standardized testing", and nowhere in the post was impropriety regarding the adoption of the test mentioned or linked to. So regardless of information introduced by commenters, the thread was not framed in terms of corruption, as pertinent as it seems. I'm pretty sure the title is all Charlie read though, since he is referring specifically to tests that take place once a year, and not tests like MAP that are given once per term.

I just made a little note in my mental chronofile to the effect of "Testing industry employee says Fuck You to teachers". Consider your industry not well represented by your public comments.
posted by StrangerInAStrainedLand at 3:09 PM on January 15 [5 favorites]

Another school has joined the boycott.
posted by femmegrrr at 3:11 PM on January 15 [2 favorites]

charlie don't surf, do you have experience with or specific knowledge of the MAP test we are discussing here, or just generic testing?
posted by jacalata at 3:14 PM on January 15

"Fuck you. You have absolutely no justification to whine about your budget being stretched thin, and that you don't have enough classroom time to waste on standardized tests. If you don't like doing your job, I suggest you go work at some other school, like one of the remote, rural Native American Reservations. Then you can learn what it means to be a teacher in a school with an inadequate budget."

Oh, shut the fuck up with this bullshit until you've cured world hunger, you ass.
posted by klangklangston at 3:26 PM on January 15 [11 favorites]

Worth noting, just for the sake of detail: Ballard is already in the middle of running the MAP test and has decided to follow through with current plans rather than throw it all out the window & cause a further disruption for its students... but yeah, everything else still applies.

Also worth noting, and probably more relevant, is that Seattle's current superintendent hasn't been at the job long. Whether he sees this as a challenge to his authority or if he understands that this isn't personal but rather addresses a lame test remains to be seen.
posted by scaryblackdeath at 3:38 PM on January 15 [1 favorite]

Some various comments from various commenters:

I have no doubt the tests are normed with students. I'm asking if they are normed with teachers.

Most of the tests are written by teachers (former teachers at least), and often tested with teachers as scorers. Most of the tests I've been involved with, have a process of setting score levels by working with teams of teachers in the school systems where the test is deployed.

charlie don't surf, do you have experience with or specific knowledge of the MAP test we are discussing here, or just generic testing?

I have several years of experience with the WASL tests, the predecessor to the current tests, and some other misc WA tests that I can't recall specifically. If I had direct involvement in the MAP, I probably wouldn't be able to comment about it due to NDA. I have no current professional interest in the MAP.

Consider your industry not well represented by your public comments.

I don't speak on behalf of anyone but myself. BTW, did you watch The Simpsons on Sunday night? Apparently the industry's public image, as represented on the show, could hardly be lower.

your reading comprehension skills (in regards to this thread) seem a bit thin.

I did RTFA, did I have to analyze every comment in this thread before proceeding? The OP content I read strongly reminded me of the last thread on MeFi that talked about testing. It's the same old story. The teachers are too brilliant to waste their time on such petty tasks as testing. Their students can all pass the exam, so what is the point of even taking it? The teachers know more about what the students need to know than the State Board of Education that sets the standards that they use, but are useless so they disobey them anyway. The teachers should be in control of the School Board, not the other way around. It is unreasonable for parents to expect their children to perform to a specific academic standard, their precious little snowflakes would melt under the pressure. Blah blah blah.

But let me pose some questions that the teachers in the boycott aren't considering. They say the test does not accurately assess their students' educational achievements. How do you know that, until they take the test? Considering the test as a broader research tool, wouldn't it be a valid research result to demonstrate a specific testing method is invalid, or incapable of a proper assessment? At the very minimum, that would indicate that this specific test needs changes. Wouldn't that help improve educational testing overall? Or perhaps they are afraid that research indicates the test isn't perfect (what is?) but that it is adequate, and improving? It would represent a challenge to the teachers' autonomy. That seems to be the core of all they complaints by teachers.
posted by charlie don't surf at 4:37 PM on January 15

It really sounds like you truly can't imagine ANY situation in which a standardized test could be poorly designed or poorly implemented, or in which a teacher could object to such tests for any reason besides sheer laziness.
posted by showbiz_liz at 5:22 PM on January 15 [8 favorites]

I'm another Garfield High School alumnus, and a current parent of a kindergartener in the Seattle Public Schools.

My kid's teachers also hate the MAP test -- among other things, because even kindergarteners are forced to take it in the computer lab. Half of these kids have never used a mouse and keyboard before, or don't realize something is wrong if their headphones are hooked up to some other kid's computer. So the supposed reading and math tests actually reflect a host of other factors like computer literacy.
posted by mbrubeck at 6:24 PM on January 15 [9 favorites]

It is quite possible that your research project yields some amount of interesting data, while nevertheless also being a low-yield waste of time compared to actual instruction.

I'd be more comfortable with these research projects if the test subjects were from private schools rather than public ones. As it stands, private schools tend to do significantly less testing and release very little of the data to the public, correct? I have a hunch — an uninformed one, so feel free to correct me — that if the powers that be considered extensive standardized testing regimens actually valuable for the students actually subject to them, they'd use them on their own kids as much as they use them on public school students.
posted by You Can't Tip a Buick at 6:33 PM on January 15 [5 favorites]

mbrubeck: do you remember the at-the-time-new standardized test we had to take one year at Garfield, the one with the bafflingly extensive section on volumes measured in imperial units? I had remembered that as being part of the WASL pilot, but a quick look at wikipedia shows that the timeline is wrong for that...
posted by You Can't Tip a Buick at 6:36 PM on January 15

I don't remember that one, Buick... but I've lost or repressed quite a lot of my memories from high school.
posted by mbrubeck at 6:39 PM on January 15 [1 favorite]

As it stands, private schools tend to do significantly less testing and release very little of the data to the public, correct?

I actually know of at least one private school that's dropped MAP testing because it was such a waste of time and frustrating from a technical level, so, yes. In some respects, it's a different situation, because private schools tend to come with much more hands-on reporting anyway-- paragraphs written about your kid, more personalized portfolios, etc. There's far less kid turnover on a year-to-year basis, let alone month to month. So I think it's fair to say that some of the reasons why "standardized" testing are valuable in a larger environment aren't as useful in some private schools.

Oddly, you do have to take a lot of standardized tests to get into most private schools, even at the elementary levels...but on a year to year level, I think it tends to be far less prevalent.
posted by jetlagaddict at 7:05 PM on January 15 [1 favorite]

I took the WASL with the question whose answer was mayri klay lee turno (get it?). The state's record when it comes to standardized tests is not good.
posted by miyabo at 8:02 PM on January 15 [1 favorite]

Most of the tests are written by teachers (former teachers at least), and often tested with teachers as scorers. Most of the tests I've been involved with, have a process of setting score levels by working with teams of teachers in the school systems where the test is deployed.

This is not norming.

The teachers are too brilliant to waste their time on such petty tasks as testing. Their students can all pass the exam, so what is the point of even taking it?

You are not seeing the whole picture..."The test—teachers at Ballard said in a letter explaining their decision—has "been re-purposed by district administration to form part of a teacher’s evaluation, which is contrary to the purposes it was designed for, as stated by its purveyor, making it part of junk science." (link).

In other words, they are using this test to test the teachers. In this use such a test would be psychometrically unsound.
posted by Hypnotic Chick at 9:11 PM on January 15 [3 favorites]

using this test to test the teacher

Well, if a teacher's name is not notorious enough to appear as the answer to a multiple-choice question, I think that should count as a positive on their annual review.
posted by miyabo at 9:31 PM on January 15 [2 favorites]

This is not norming.

What more could you want? Every test I've worked on is normative. The teachers look at actual test results from students, to establish a broad profile of score points from the top to the bottom. Those papers are used by scorers as standards to compare other other student work against. As a scorer, I get a book full of sample papers, pre-evaluated by teachers and other experts. We study them to understand why this paper is a 3 and not a 2, for example. Then they test us to make sure we can judge that accurately. Students are rarely ever judged against totally arbitrary standards from a rubric except during the initial normalization. Students are judged against another student's performance. For example, we get a paper and it says it is an example of how 3 Level student gets his 3 Score point on the test. If your paper is this good or better, it gets a 3 too.

In other words, they are using this test to test the teachers. In this use such a test would be psychometrically unsound.

It is pretty standard to evaluate students to measure their teacher's effectiveness. It's not the only measure of course. Standardized testing indirectly tests everything, including the students, their teachers, the test writers, the test scorers, the test analysis, and the conclusions drawn from that analysis. For example, right now, the NY Regents started using student test scores as part of teacher evaluations. Many teachers are pissed because now they get graded, whereas before they could just bullshit their way through their jobs. And yes, some outstanding teachers get disproportionately low ratings. But it's a new program and that is the whole point, they're trying to find out how good or bad teachers affect students and their test scores. They are learning how to
posted by charlie don't surf at 9:38 PM on January 15

Charlie Don't Surf, I can't help but notice that you seem to respond to everyone here but me. Any particular reason for that?
posted by scaryblackdeath at 9:39 PM on January 15

Because they're asking me specific questions or responding directly to a statement I made. Did you have a specific question? You say you've worked as a scorer, presumably you already know something about this topic.
posted by charlie don't surf at 9:44 PM on January 15

The boycott seems like a terrible idea to me if state or federal funding is based on test scores.

Garfield HS can't take federal funds, anyway, since they ban military recruiters from the school.

Purchased by a corrupt superintendent from a company that she sat on the board of

The MAP is run by NWEA, which is a non-profit. One of the things that has always bothered me about the anti-MAP rhetoric is how EEVL the MAP is because it's an EEVL CORPORATION when it's a non-profit flowing out of Northwest school districts.

I could never get a read on Maria Goodloe-Johnson. Was she corrupt? Inept? Unwilling to pay homage to the strong PTAs that actually run the upper middle class white schools of the district (not principals or teachers)? I tend to think all three. The school closure debacle of 2009 was one of the most hare-brained things I've ever seen -- they didn't even consult a demographer to determine capacity needs, and did they ever pay for it just a year later.

Seattle Schools is a Yeats poem come to life -- the administration lacks all conviction, while the PTA rabble-rousers are full of passionate intensity. I'm not sure my daughter is getting a good education because of that nasty confluence.

An example from a couple years ago: The teachers proposed carving out space in the library for a walled off, dedicated computer classroom. This would mean they could finally teach technology classes and remove the computer lab and all its noise from the library. On the whole, a reasonable choice, especially given the MAP test being computerized, and given there were roughly two computers per CLASSROOM (vs one computer per STUDENT in the Eastside districts -- and mind you, two computers in a classroom was one more than many south end schools had).

The hue and cry from a section of the parents was loud, and frankly, Luddite. We should not let our young children with the plastic minds near those computers, said one parent of a kid who was the same age I was when I learned to program BASIC on our classroom computer in 1980. How dare they remove any library space and books! (Never mind that some of the books dated from the 1950s and probably hadn't been cracked since then.)

The kicker, of course, was the parents declaring this was the EEVL work of Maria Goodloe-Johnson. The teachers, mind you, proposed this. This didn't come from downtown, or from her handpicked principal the teachers had run out on a rail for being divisive and uncommunicative. This came from the teachers, who given a sum of money, decided to spend it on the thing they thought was most important. And the PTA parents essentially tried to block them from making their choices.

Luckily, common sense prevailed, and the computer lab is doing well, though it hasn't seemed to have affected MAP scores at all.

As for the MAP, I have mixed feelings. Twice a year the school shuts down for a week to do assessment. It's a little much. OTOH, the MAP is where my daughter absolutely kills it. In fact, the MAP is what we used to force her teacher to stop trying to set a ceiling on her reading skill -- the MAP clearly showed she was reading two grade levels ahead, not at grade level like her previous year's teacher said. As a diagnostic tool, it's great. But it is a little much, especially given the ACTUAL assessment for NCLB isn't the MAP, it's the MSL, which eats a whole OTHER week of school.

In general, though, I wish I could get my daughter educated without dealing with administrative know-nothings, PTA busybodies, and people screaming like Picard about the line needing to be drawn here every single time the smallest issue comes up. I'm tired of all this crap. I just want my daughter to get a good education. I don't want the political BS that comes with it.
posted by dw at 10:25 PM on January 15 [5 favorites]

I have several years of experience with the WASL tests, the predecessor to the current tests, and some other misc WA tests that I can't recall specifically. If I had direct involvement in the MAP, I probably wouldn't be able to comment about it due to NDA. I have no current professional interest in the MAP.

The WASL was not the predecessor to the MAP, it was replaced by the MSP and the HSPE and the MAP testing was introduced separately. I'll mark that as a 'no, not familiar'.

They say the test does not accurately assess their students' educational achievements. How do you know that, until they take the test?

They've been taking it since 2009, they may have had the chance to form opinions without testing this specific cohort of students.
posted by jacalata at 11:11 PM on January 15 [4 favorites]

Go Bulldogs! Cave Canem! Perhaps I have been graduated long enough for nostalgia to be kicking in.

I am really proud of Garfield, and the other Seattle schools. There are loads of problems - but you gotta start somewhere. I think we had 3 or 4 principles while I was there, the lunch time cliques were really segregated (as were AP classes and the commercial foods program), but lots of the teachers and students were great, hard working, and inspiring. I hope this will result in more money for education, not just testing.
posted by PistachioRoux at 12:09 AM on January 16 [1 favorite]

Is this thing some relation to the California Achivement Tests they used to give us back in the late 70s and early 80s, for seeing how we were at math and reading?

I'm just trying to figure out the relationship of these tests. I'm not a huge fan of a teach-to-the-test setup, but I think that's because of my high school being a proponent of the tallest-nail theory for academics (but if you had sports skills, you were a minor deity), and teach-to-the-test, as I've read about it, seems to be that same kind of thing, where there's no reason for a teacher to even try to let you, much less push you, to go farther.
posted by mephron at 4:25 AM on January 16 [1 favorite]

Go Bulldogs! Cave Canem!

Is Cave Canem really their motto? If so that is really cool!
posted by TedW at 4:57 AM on January 16 [1 favorite]

Charlie Don't Surf: Here, I'll put it in again:

Yes, that's the whole point. The test is part of that research. You know, tests don't just spring up out of thin air. People have to write the questions, test the proposed questions by giving experimental tests to sample student populations, and test the scorers to see if they can accurately and consistently assess the answers. New tests are almost constantly in development, they write a new test for next year while the current year's test is being given.

You do realize that not everything churned out by the professional development industry is worth the paper it's printed on, right? Nor are all tests of equal worth and effectiveness. Why is it difficult for you to believe that this particular test is bad?
posted by scaryblackdeath at 6:42 AM on January 16 [1 favorite]

I teach in a private academy in Korea. My kindergarten-7th grade students are tested monthly on what they've learned, and every three months to check their level against an objective standard. These scores are compared to other of our schools across the country. They are used to assess student/teacher/manager/campus performance.

It is a huge waste of time...but you get used to it, it becomes routine. You eventually stop being angry at the loss of class time, and you accept the stupid additional stress it causes your students as just part of the job.

I'm glad these teachers are not allowing themselves to accept it and move on.
posted by nile_red at 6:46 AM on January 16 [1 favorite]

It is pretty standard to evaluate students to measure their teacher's effectiveness. It's not the only measure of course. Standardized testing indirectly tests everything, including the students, their teachers, the test writers, the test scorers, the test analysis, and the conclusions drawn from that analysis.

I understand that this makes sense to you. In fact, looking at it, it seems to make sense to me. But what you are accepting as a matter of course is fraught with controversy. So let take a look at what is going on.

When it comes to assessment, the kind that will be published and sold to school districts, the first thing to realize is that writing the test is not the hard part. The hard part is:
1. determining if the test is valid and if it is reliable, and
2. finding a representative sample of the population that will provide insight on how to interpret the test results.

Fundamentally, both of these steps need to occur before the test is published for sale. Now, when I say determine validity and reliability I mean that some kind of statistical analysis will be run on the test and that information will be provided in the testing materials. The same with sample data and normative data. It happens ahead of time and is provided in the methodology.

Well, so what? The point is that no matter how "standard" it is for anyone to use student scores to evaluate teachers, unless the test publishers publish R/V analysis, sampling data, and norms (not to mention theoretical basis) about the test as applied to teachers it must, by definition, be understood as invalid. It literally hasn't been validated.

This is why I say above, you wouldn't use an HIV test to test for Medicare fraud if you were interested in finding fraud.

So, to this: Standardized testing indirectly tests everything, including the students, their teachers, the test writers, the test scorers, the test analysis, and the conclusions drawn from that analysis. That may be the case, but unless it is done under the scientific conditions, the information you get just isn't meaningful. The NWEA doesn't publish norms on teacher evaluations (like the do for students, PDF)

So if you are correct and such testing is a standard used for teacher evaluations, it would merit rejection...rightfully.

In the end, this whole phenomenon comes out of the misunderstanding the public has about what goes into assessment, specifically the legwork around getting an assessment to be considered psychometrically sound. It is a rigorous process. Getting meaningful results is hard. But definitely using the results of any given test to determine a phenomenon not addressed, not analyzed, not sampled, not normed is a form of malpractice...whether it occurs in a doctor's office or school district.
posted by Hypnotic Chick at 6:54 AM on January 16 [4 favorites]

OK scaryblackdeath, I thought that was a rhetorical question. Of course I don't believe every test is perfect. That's my job, working on improving tests and eliminating unworkable methods. Unfortunately due to NDA I can't discuss the specific details to the level that Hypnotic Chick would like. But I can say that it is possible to make assertions like hers, that assume that no testing methods can be properly validated. That is ridiculous. If you really want to improve the quality of test results and test methodology, one of the best ways would be to increase the sample size to 100% of all students. The teachers know this and are determined to prove their case that the tests are invalid, by sabotaging them.
posted by charlie don't surf at 5:01 PM on January 16

So, you can't talk about details, but her assertions are ridiculous because teachers sabotage testing, etc.

How long until you just declare "I know more about this than you can possibly imagine" and be done with it?

Because right now, you're acting like a petulant shill rather than an honest participant.
posted by klangklangston at 5:13 PM on January 16 [4 favorites]

But I can say that it is possible to make assertions like hers, that assume that no testing methods can be properly validated. That is ridiculous

I want to be completely clear. I am not saying that the MAP or any other ability test cannot be validated for the use of assessments of teachers. Instead, I am saying that no standardized test that I know of has been normed on a sample of that population.

Such norming on a sample of studied teachers, where it to exist, would be provided if it existed, as it is for the testing of students (as I linked above).

If you really want to improve the quality of test results and test methodology, one of the best ways would be to increase the sample size to 100% of all students. The teachers know this and are determined to prove their case that the tests are invalid, by sabotaging them.

As I noted above, there is a very basic misunderstanding at play here. I would suspect that the vast, vast majority of ability tests given to students are ironclad, methodologically speaking. I find it hard to believe that the MAP would be invalid or unreliable as a test of student ability. That isn't the issue. The issue is whether you should use a test meant to describe phenomenon X (student reading skill) to describe phenomenon Y (teacher effectiveness). This isn't about the test being a poor instrument. Its about using the (quite possibly high-quality) instrument in the wrong way.
posted by Hypnotic Chick at 7:04 PM on January 16

Instead, I am saying that no standardized test that I know of has been normed on a sample of that population.

The test is normalized on the entire sample population, every year. There is no point where the process is totally validated as a final product and set in stone forever. This test uses a new methodology and even a well established test system is a continually developing process that uses each year's scoring to validate ongoing results. This is why the teachers are trying to boycott and remove a large data set from the test, it will make it more difficult to validate the test for any use whatsoever. As far as I can tell from the MAP docs, this is the most detailed study of intra-year educational development, and it hasn't been around long enough to track those students through college performance, which would give long term validity to the methods, or show where it could be improved. The teachers want to stop that from happening, and good luck with that when your teaching contract is up for renewal.

The issue is whether you should use a test meant to describe phenomenon X (student reading skill) to describe phenomenon Y (teacher effectiveness). This isn't about the test being a poor instrument. Its about using the (quite possibly high-quality) instrument in the wrong way.

I haven't seen anything that indicates student test performance is invalid as an indicator of teacher effectiveness. The School Boards across the country are pretty set on the idea that an effective teacher is one who can improve his students' test scores. I hear a lot of gripes from teachers that don't want to be evaluated at all. Whenever I see a group that says they are a special case, unaccountable to anyone but themselves, I really want to see some precise metrics, because they're hiding something.
posted by charlie don't surf at 7:27 PM on January 16

I haven't seen anything that indicates student test performance is invalid as an indicator of teacher effectiveness.

Well, but... if you haven't seen anything that indicates that it IS valid, then you can't just ASSUME that it is.
posted by showbiz_liz at 7:59 PM on January 16 [2 favorites]

Yeah, I've seen stats that convinced me it IS a pretty good index, although it's not the only index. Here's a simplified version, an article from the WSJ that describes the NY City school system opening up its teacher ratings to public inspection. THIS is what the teachers don't want. It's worth noting the little graphic, that shows teachers are partly ranked by how their student test scores deviate from predictions.
posted by charlie don't surf at 9:14 PM on January 16

"I haven't seen anything that indicates student test performance is invalid as an indicator of teacher effectiveness."

Then you're either willfully ignorant or disingenuous. It's an axiom that incentives change behavior, and that by incentivizing a metric, it's measurement value decreases. You can measure how well kids are performing, but using that as a direct statistical inference to discern teacher performance is incredibly problematic.

"The School Boards across the country are pretty set on the idea that an effective teacher is one who can improve his students' test scores."

The Bible is true because God says it's true, and he's in the Bible!

"I hear a lot of gripes from teachers that don't want to be evaluated at all."

And you also hear about teachers who don't object to testing broadly, however, they do object to one test and the impact it has on their ability to teach. You heard about that most recently in the article at the top of this post.

"Whenever I see a group that says they are a special case, unaccountable to anyone but themselves, I really want to see some precise metrics, because they're hiding something."

This is like deciding that the best art is the art that sells for the most, then complaining that artists who disagree just don't want to be accountable to the market.

This isn't about people "hiding something," this is about realizing that testing is not the point of teaching, and that if you're going to take up time, you shouldn't be wasting it.
posted by klangklangston at 9:15 PM on January 16 [4 favorites]

The test is normalized on the entire sample population, every year.

This just isn't correct. Ok, let's break this down. You have a sample and you have a population. They are not the same thing. As a researcher, you are trying to find a representative segment of the whole which will limit the data you need to look at. To the best of my knowledge, there is no assessment norms its data based on a whole population. It just isn't done. So let's look at the MAP documentation (at the link I provided above):

The 2011 NWEA RIT Scale Norms Study provides growth and status norms... The study’s results are based on grade level (K-11) samples of at least 20,000 students per grade. These samples were randomly drawn from a test records pool of 5.1 million students, from over 13,000 schools in more than 2,700 school districts in 50 states. Rigorous post-stratification procedures were then used to maximize the degree to which both status and growth norms are representative of the U.S. school-age population.

So, no, this test is not normed to the whole population. Note - legitimate ability assessments will provide sample and norming data. Now, you are right in that this data is not set in stone. The 2012 NWEA norms will look at up to 240,000 new kids, as will the 2013 norms...and so on.

This is why the teachers are trying to boycott and remove a large data set from the test, it will make it more difficult to validate the test for any use whatsoever.

If teachers at these two schools boycott the test, they are not going to derail anything. The test materials are telling you that the publisher is looking at data from across the country. This boycott would represent no meaningful obstacle to the collection of data for the publisher. Again, they are trying to get about a sample that is less than .5% of a population that includes five million people.

I really want to see some precise metrics,

At this point, you can expect neither precision nor accuracy if you are misusing a test. I understand that it seems like common sense to say "teacher performance=student scores", but that isn't science. To be clear, teacher performance is not the same as student performance and any precise instrument that measures one does not necessarily measure the other. If the test publishers (or anyone else) purports to use this assessment to test teachers then I'd ask why are they not providing the same, basic, methodological information that they do for the students? There is nothing proprietary about data and methodology (its the instrument itself that is copyrighted). It isn't enough to say, "well you can't prove it doesn't test that". Again, that's not science.
posted by Hypnotic Chick at 7:29 AM on January 17 [1 favorite]

I understand that it seems like common sense to say "teacher performance=student scores", but that isn't science.

It isn't what I said either. I cited an example from the NYC School System that uses student scores by making an estimate of expected scores from the students of each teacher, and seeing if actual scores over or underperform the expected scores. That is not used as a direct translation of student scores into teacher performance, it's weighted against other factors. I can't get into more specific detail as that would be informed by knowledge acquired in my current job, which I am not authorized to discuss publicly. Sure, it might be available from other sources.

The Seattle teachers want to boycott the test because it will prevent gathering data that could be used in teacher performance ratings. I noticed the teachers only started a direct action once the possibility of using MAP for teacher ratings came up. Did you read in that article I cited, where the NYC teacher's unions fought to prevent release of teacher ratings, and lost?

Again, that's not science.

You keep saying that. I don't think that means what you think it means.

Science is a process involving the Scientific Method. It is not a product containing 100% Provable Truth at some golden moment of absolute perfection. It is a method of ongoing inquiry to acquire data and compare it to hypotheses, using testing and measurements to create new working hypotheses. The entire system of educational testing is an ongoing experiment, using testing and measurements to help create and continually update a set of best practices. This is not surprising, since the entire US educational system is also a grand scientific experiment in social benefits of an educated populace. This is also not surprising, since the Government of the United States was created as a scientific experiment in political technology, with a hypothesis that a well informed populace would lead to better government.

And ultimately that is where the problem lies. The Seattle teachers oppose the test for reasons that are completely political. The goals of the testing system are not in alignment with their political goals. They may attempt to back up their argument with data, but their argument is completely unscientific and can benefit from scientificicsm only as window dressing, to disguise their political argument as fact.
posted by charlie don't surf at 3:44 PM on January 17

charlie don't surf, what's your explanation for why the teachers oppose the MAP test and not all the other standardized testing their students go through? I see a lot of statements like "The Seattle teachers oppose the test for reasons that are completely political" and "The Seattle teachers want to boycott the test because it will prevent gathering data that could be used in teacher performance ratings," but not a lot of firm backup for them. I know you believe these statements to be true, but I haven't seen any evidence that convinces me (a parent of children in public schools in Seattle) that they actually are.
posted by KathrynT at 3:51 PM on January 17 [1 favorite]

"You keep saying that. I don't think that means what you think it means."

She knows better that you, obviously. A pedantic retread of the scientific method doesn't do anything to rebut her argument that these are flawed inferences.

You're launching into ad hominem attacks on the teachers, appealing to authority in order to justify a testing regime you admit you don't know the details of, ignoring the actual text of the article, and being a condescending dick while you do it.

¿Que pedo?
posted by klangklangston at 4:12 PM on January 17 [2 favorites]

That is not used as a direct translation of student scores into teacher performance, it's weighted against other factors.

I had read the article and it includes an illustrative example from Colorado:

The new Colorado evaluation system was developed in response to a 2010 bill requiring that all principals, teachers and other licensed school staff be reviewed annually. Half of a teacher’s score is determined by student achievement on a range of tests...

All I am saying is where is the research that backs up the hypothesis that 50% of teacher effectiveness can be described by the assessment given. If such research exists it should be publicly available. If that research isn't publicly available, we should be suspicious of the results.

...I don't think that means what you think it means. Science is a process involving the Scientific Method. It is not a product containing 100% Provable Truth at some golden moment of absolute perfection.

What I am describing is basic Methods 101. We aren't talking about "100% truths" or "absolute perfection". Now, you are right when you say that science,"...is a method of ongoing inquiry to acquire data and compare it to hypotheses, using testing and measurements to create new working hypotheses", but you do that in a context of research methods. If you don't believe me, here is a document from a test publisher describing it.

Again, the whole point is that they have an instrument. That instrument was determined to be reliable and valid and the publisher provides sampling and norming data for one group. You then have government officials wanting to use that same instrument to test another group and doesn't provide any data (or even theoretical justification) to do that. This second use is what isn't scientific.

Let me ask you, if we expect publishers to provide rigorous and robust data and analysis to support the use of an assessment to determine student ability, why shouldn't we expect the publishers to provide the same for uses that determine teacher effectiveness?
posted by Hypnotic Chick at 4:40 PM on January 17 [5 favorites]

To me, charlie don't surf reads almost exactly like a clever know-it-all high schooler trying to lecture actual working researchers.

I am most definitely not a scientist. Moreover, I am frankly wretched at stats — I am not proud of this — so I can't speak to anything but rhetoric here. But, well, just on the surface CDS's statements tend to underplay complexity in exactly the places that Hypnotic Chick's statements (for example) reveal new and interesting nuances. This makes CDS's statements seem less trustworthy and HC's seem significantly more trustworthy.

Looking back at the old thread that CDS linked to, I'm struck by how Charlie Don't Surf's first comment looks nothing like real research in rhetoric or composition. This is in sharp contrast to many of the other comments on the thread. People who study (college-level) student writing for a living tend to focus on things like what Rory Marinich talks about, treating writing as something that is good for addressing specific audiences for specific purposes rather than treating writing as abstractly and absolutely good or bad like CDS does.

Like a bunch of people here, I've taught rhet-comp to freshmen in college. One of the big challenges one faces teaching that class, especially teaching it fall quarter to students fresh out of high school, is that their previous education has consistently taught them that meeting CDS's requirements is much, much more important than producing good writing. In real world contexts, it's, well, the other way around.^*

Anyway, I suspect the scientists and statisticians here are reacting to CDS's treatment of statistics and scientific methodologies in this thread more or less like I reacted to CDS's treatment of writing in the old thread.

*: please don't interpret this comment as an endorsement of first-year rhet/comp — I am painfully aware of the flaws of that particular institution.
posted by You Can't Tip a Buick at 12:37 PM on January 18 [3 favorites]

Apparently some people don't understand why I'm not responding to their "argument." They claim I am appealing to (my own) authority, while they appeal to the authority of other people in this thread. Since those other people did not try to assert their own authority, it is unlikely they would condone other third parties who would assume the mantle of authority in their name. And of course there are the straw men, like claiming I slandering the teachers by asserting their actions are politically motivated. I might actually admire their position, if they'd just be honest and admit their goals are political. But instead, we get the same old problem that should be well familiar, a politically motivated group making assertions that their political ideal IS the ideal. And then the amusing other straw men, made out of the merest gleanings they can find from other threads, clutching at straws.

I cannot respond to those arguments because there is no argument, only an assertion of a political position. I don't have a political position other than that I intensely dislike politics disguised as science. However, I can respond to a real argument like:

Let me ask you, if we expect publishers to provide rigorous and robust data and analysis to support the use of an assessment to determine student ability, why shouldn't we expect the publishers to provide the same for uses that determine teacher effectiveness?

I agree, we should expect this data to be published. It is. For example, this document, "Ensuring Fair and Reliable Measures of Effective Teaching." This is the project that Seattle is implementing. It's a research project to determine if these methods are useful in evaluating teachers. This report is from the pilot program, which has enough validation to make it worth further research and evaluation.

Of course the teachers have a political incentive to thwart any data collection that might be used to evaluate them. That isn't the way the tests are currently being used, right now they're only used to evaluate students. But the teachers claim that use of the tests, even for what they were originally intended for, is "flawed" without giving any evidence of the flaws. But the flaws are not what the teachers claim are flaws. For example, they say the tests aren't connected to their curriculum. Well it doesn't have to be. We can test for math proficiency without having to know the specific textbook or methodology. They say they don't use the test results and they don't want them. Well the School Board wants the data. You see, this is why I assert the teachers are politically motivated. They say they don't want the tests, because they don't want the tests. This is not a flaw. This isn't even an argument.
posted by charlie don't surf at 5:56 PM on January 18

I don't have a political position other than that I intensely dislike politics disguised as science.

Ha! Your political position is to negate all efforts that wish to critique the MAP test under the name of SCIENCE by refusing to examine how these 'scientific' tests are instrumentalized. (See: Einstein on the Manhattan Project, etc)

Here's a question. Is it not possible for these statements to be both true?

1) The MAP test is a balanced, scientifically accurate test to evaluate student performance.

2) The MAP test is unnecessary, misused to evaluate teachers, is a severe strain on school resources, and was possibly purchased by superintendent as part of a corrupt deal.
posted by suedehead at 11:45 PM on February 5 [5 favorites]

« Older Cast Iron Pipe Legs? Check. Thick board top? Ch... | Iquitos... Newer »

This thread has been archived and is closed to new comments

Teachers boycott standardized testing January 15, 2013 8:29 AM Subscribe

Teachers boycott standardized testing
January 15, 2013 8:29 AM Subscribe