Last week, this video from The Onion (asking whether tests are biased against kids who don’t give a sh^%t) was going viral among the education social networking geeks like me. At the same time, the conversations continued on the Los Angeles Times Value-Added story, with LAT releasing the scores for individual teachers.
I’ve written many blog posts in recent weeks on this topic. Lately, it seems that the emphasis on the conversation has turned toward finding a middle ground – discussing the appropriate role for VAM (Value Added Modeling) – if any, in teacher evaluation. But also, there is renewed rhetoric defending VAM. Most of that rhetoric seems to take on most directly the concern over the error rates in VAM – and lack of strong year to year correlation between which teachers are rated high or low.
The new rhetoric points out that we’re only having this conversation about VAM error rates because we can measure the error rate in VAM, but can’t even do that for peer or supervisor evaluation – which might be much worse (argue the pundits). The new rhetoric argues that VAM is still the “best available” method for evaluating teacher “performance.” Let me point out that if the “best available” automobile burst into flames on every fifth start, I think I’d walk or stay home instead. I’d take pretty significant steps to avoid driving. Now, we’re not talking about death by VAM here, but the idea that random error alone – under an inflexible VAM based policy structure – could lead to wrongfully firing a teacher is pretty significant.
Again, this current discussion pertains only to the “error rate” issue. Other major – perhaps even bigger issues include the problem that so few teachers could even have test scores attached to them – creating a whole separate sub-class (<20%) of teachers in each school system and increasing divisions among teachers – creating significant tension, for example between teachers under the VAM (math/reading) rating system, and teachers who might want to meet with some of their students for music, art or other enrichment endeavors.
Perhaps most significantly, there still exists that pesky little problem of VAM not being able to sufficiently account for the non-random sorting of students across schools and teachers. For those who wish to use Kane and Staiger as their out on this (without reference to broader research on this topic), see my previous post on the LAT analysis. Their findings are interesting, but not the single definitive source on this issue. Note also that the LAT analysis itself reveals some bias likely associated with non-random assignment (the topic of my post).
So then, what the heck does this have to do with The Onion video about testing and kids who don’t give a sh^%t?
I would argue that the non-random assignment of kids who don’t give a sh^%t presents a significant concern for VAM. Consider any typical upper elementary school. It is quite possible that kids who don’t give a sh^%t are more likely to be assigned to one fourth grade teacher year-after-year than to another. This may occur because that fourth grade teacher really wants to try to help these kids out, and has some, though limited success in doing so. This may also occur because the principal has it in for one teacher – and really wants to make his/her life difficult. Or, it may occur because all of the parents of kids who do give a sh^%t (in part because their parents give a sh^%t) consistently request the same teacher year after year.
In all likelihood, whether the kids give a sh^%t about doing well – and specifically doing well on the tests used for generating VA estimates – matters, and may matter a lot. Teachers with disproportionate numbers of kids who don’t give a sh^%t may, as a result receive systematically lower VA scores, and if the sorting mechanisms above are in place, this may occur year after year.
What incentive does this provide for the teacher who wanted to help – to help kids give a sh^%t? Statistically, even if that teacher made some progress in overcoming the give a sh^%t factor, the teacher would get a low rating because give a sh^%t factor would not be accounted for in the model. Buddin’s LAT model includes dummy variables for kids who are low income and kids who are limited in their English language proficiency. But, there’s no readily available indicator for kids who don’t give a sh^%t. So we can’t effectively compare one teacher with 10 (of 25) kids who don’t give a sh^%t to another with 5 (of 25) who don’t give a sh^%t. We can hope that giving a sh^%t , or not, is picked up by the child’s prior year performance, and even better, by the prior multiple years of value-added estimates on that child. But, do we really know whether giving a sh^%t is a stable student characteristic over time? Many VAM models like the LAT one don’t capture multiple prior years of value-added for each student.
I noted in previous posts that peer-effect is among those factors that compromises (biases) teacher VAM ratings. Buddin’s LAT model, as far as I can tell, doesn’t try to capture differences in peer group when attempting to “isolate” teacher effect (though this is very difficult to accomplish). Unlike racial characteristics or child poverty, whether 1 or 10 kids in a class give a sh^%t might rub off on others in the class. Or, the disruptive behavior of kids who don’t give a sh^%t might significantly compromise the learning (and value-added estimates) of others. Yet, all of this goes unmeasured in even the best VAMs.
Once again, just pondering…
NEW: BONUS VIDEO
School Finance 101 
Wow. Thanks. Are you sure you aren’t a teacher? You make so much sense.
Okay… I’ll admit it… I was a middle school science teacher for a while, and before that a gifted program coordinator (hence my concern over the pull-out issue, and tension between the classroom teacher and other teachers). The middle school teacher in me is what leads me to question the stability of the characteristic of “giving a sh^&t” over time.
On not giving a sh*t:
First, I live with a shrink, who continually points out that at any given moment you have a certain percentage of children and adults who are coping with mental illness. Currently, the NIMH estimates that 1 out of 4 US adults has some form of mental illness , which is up from the more typical 1 out of 5 (NIMH attributes this rise to the economy) . The rate for kids is somewhat lower, but not dramatically so.
And some forms of mental illness look EXACTLY like noting giving a sh*t. Major depression is one, which many parents and school folks don’t recognize in children or adolescents.
So, now you have a certain percentage of your student body, in addition to whatever else they have going on in their lives, now have either a diagnosed (which usually means they have the attendant IEP), or more likely, an UNDIAGNOSED mental illness.
Taking Bruce’s scenario of a principal having it in for a 4th grade teacher, what if her/his classroom is “oversupplied” with kids who don’t give a sh*t, some of whom have IEPs, but most don’t (although many need one)? How do VAMs even begin to address what a very good teacher is doing with children who may be cognitively compromised because of their illnesses? Some mentally ill kids can lose years of schooling until the medications are sorted out. Are we really going to hold teachers accountable for their performances? Seriously?
My understanding of VAMs is they contain heroic, dare I say, INSANE assumptions (apologies to my partner). As Bruce has pointed out in an earlier post, the potential for very expensive litigation seems fairly robust to me (a mere historian).
Well done, Bruce! Keep up the good work. Thanks for giving a …
In reading comments to the LA Times article and elsewhere, several teachers said they felt it was their responsibility to make sure kids did give a sh^&t. Too, the ability to “engage” students (although how exactly that is measured is debatable) is high on many teacher evaluation forms nowadays, so I would bet that a teacher with low VAM might be accused of not being sufficiently motivating. Can’t win for losing…
Throw into the equation that classes are pushing thirty students, if not higher, due to the recent budget woes. You got kids with IEPs or 504s, students who can’t even read a test due to low language proficiency or a learning disability, mixed with students who are well above proficient. Meanwhile, the teacher is being harassed by parents of students with accommodations to chunk tests, supply study guides and maintain contact with them at all times, while the parents of the geniuses want their kids challenged even further. The kids who don’t care are lost because usually the teachers don’t hear from those parents. It’s also a tough task for any teacher to address all those needs let alone shift the paradigm between all three proficiency ranges.
The distribution of students across classes and the distribution of resources (aides, technology, etc.) that will help teachers succeed has always been a contested factor. But these variables are even more difficult for any principal to control and distribute equitably since budgets are being slashed, resources shrinking and teachers being scheduled out due to cuts. What’s left? Principals may give the few resources they do have to teachers who they know will do a good job with those resources and those teachers usually teach specialized advanced or remedial classes.
Lou
It’s a slippery slope into all the other factors that VAM doesn’t even attempt to measure: giving or not giving a @#$!, perseverance, positivity, out-of-school learning, etc…
Slippery slope indeed, and there are many factors that fall unevenly across teachers in schools, potentially affecting their VAM ratings. The very serious point behind this post is that we need to continue to point out the full range of shortcomings of VAM as a major or primary factor for teacher evaluation, not just the year-to-year instability/error rates. Those are a big problem too, but only one of many technical issues, not to mention educational, social and behavioral issues.
I can tell you firsthand that I have watched students go from giving a sh^%t to not giving a sh^%t before my very eyes in the span of half a school year. I had one student who was eager to learn and making great progress and one day he just….stopped caring. Conferences and calls revealed that there were some big issues at home that were clearly affecting him in the classroom. I tried a variety of strategies to engage him; some worked and some did not, but he was definitely not poised to make a lot of progress that year.
And related to my point in this post – let’s say that a good school principal had a pretty good read on the kids and their home lives when deciding how to assign kids to classes for the next year. And let’s say that you were one of those teachers who really cared about working with and helping out kids who are going through difficult family times. If the assignment of students to classes was determined by a meeting with you, the other teachers in your grade level and the principal, you might want to take on more those kids who you think you can really help out. But, if you were going to be evaluated… get your raise based on… or even fired for not producing high enough value-added test scores for your kids… you might actually have to risk your job if you wanted to take on those challenges year after year – challenges that wouldn’t be accounted for in the value added estimates. I’m hoping you’re not a teacher in Colorado, Louisiana, Washington DC or the handful of other locations where they are pushing rigid policies for using Value-added estimates.
This is great piece! You have made a very complicated issue understandable and relatable.
People don’t generally understand the small number of “kids who don’t give a sh^%t” necessary to torpedo an entire school, let alone a single classroom.
An elementary school with a population of around 5 to 7 hundred can take a serious hit from less than five students performing poorly.
Mental illness aside, there are any number of reasons a school full of the children of social anxiety ridden parents will go flat out on a test and a school full of children of parents that are physically exhausted will not take testing as seriously.
Finally let me state unequivocally that though we are fairly sure the kids that score well have learned something, the failure of others to do well tells us nothing of substance. Nothing. Nada. Zip. Zilch.
Can’t emphasize the peer effect and/or disruptive peer influence enough.
If you really want to reach out to students they must understand that “you” care. If you don’t care about what your doing then they won’t care. If you come to school prepared to deliver proper instruction and are excited about the subject matter then they too will be excited about the subject that they are to learn.
I have worked in inner city (both low and high socioeconomic) elementary/middle/ and high schools over the past 20 years and kids are kids as a teacher and as an administrator. If the teacher is excited about teaching then the kids will also be excited about their learning. It is a two way street.
The value added model that is being used in Houston is incomprehensible to the teachers and to school administration. If there was a truly bonafide model that could be explained to everyone and anyone then it would be of interest and it would be of a real value. At present it is a bunch of “goboley gook” which has no credibility.
I totally agree that a teacher plays a role in whether kids give a $#*! or not. That said, when a teacher is assigned a larger share up front, of kids who don’t give a $#*! based on their prior year, that teacher will be at a disadvantage on achieving greater value-added, even if they really help turn around the attitude of many of those kids. My concern here is that teacher may get beaten down by this new system which not only fails to reward them for taking on these challenges, but in fact penalizes them. That’s a big problem.
You raise another issue here that ties back to my legal discussion on this topic from a while back. I’m concerned that teachers will not be provided sufficient comprehensible information on how the value added model generates an estimate of their effectiveness – even as a component of evaluation. And teachers will be dismissed over a) something they have litter understanding of and b) something they have limited control over. These both seem like significant legal/contractual problems. We’ll have to see how this plays out in the courts over time.