On False Dichotomies and Warped Reformy Logic

Pundit Claim 1 – Value added modeling is necessarily better than the “status quo”

There exists this strange perspective that we are faced with a simple choice in teacher evaluation – a choice between using student test scores and value-added modeling, or continuing with the status quo. This is a false dichotomy, false dilemma or logical fallacy. In other words, it’s a really stupid argument in which we are forced to assume that there are only two choices that exist. This argument is usually coupled with an implicit assumption that one of the two must be superior.

“Reformers” continue to press the argument that current teacher evaluations are so bad, so unreliable, that anything is better than this “status quo.”

Expressed mathematically:

Anything > Status Quo

Bear with me while I use the “greater than” symbol to imply “really freakin’ better than… if not totally awesome… wicked awesome in fact,” but since it’s relative, it would have be “wicked awesomer.”

Because value-added modeling exists and purports to measure teacher effectiveness, it therefore counts as “something,” which is a subclass of “anything” and therefore it is better than the “status quo.” That is:

Value-added modeling = “something”

Something ⊆ Anything (something is a subset of anything)

Something > Status Quo

Value-added modeling > Current Teacher Evaluation

Again, where “>”  means “awesomer” even though we know that current teacher evaluation is anything but awesome.

It’s just that simple!

After all, you can’t even measure the error rate in current principal and supervisor evaluations of teachers can you? And if you can’t measure the error rate it must be higher than any error rate you can measure? More really basic reformy logic! That is, the unobserved error rate in one system is necessarily greater than the observed error rate of another – even if we have no way to quantify it – in fact, because we have no way to quantify it?

Unobserved error rate of ‘status quo’ > measured error rate of VAM

Let’s be really blunt here. Both are patently stupid arguments.

And both of these arguments bring to mind one of my favorite analogies related to this issue. If we were in a society that still walked pretty much everywhere, and some tech genius invented a new cool thing – called the automobile – but the automobile would burst into a superheated fireball on every fifth start, I think I’d keep walking until they worked out that little kink. If they never worked out that little kink, I’d probably still be walking. I’ve written previously about how this relates to likely error rates in teacher dismissal (misclassifying truly effective teachers as ineffective) as would occur when using typical value-added modeling approaches.

Pundit Claim 2 – If we get rid of the bad teachers, the system will necessarily be better

The assumption of many pundits is that replacing existing teachers necessarily improves the teaching workforce – that the average potential applicant for any/all available teaching jobs will be better than the average person already there, or at least better than the person we dismiss as ineffective. Now, recall that we have a pretty high chance of misclassifying truly effective teachers and dismissing them.

Now, the math here is similar to that above. The basic premise is that:

Anything > Status Quo

First of all, we know already that schools with more difficult working conditions have a much more difficult time recruiting and retaining quality teachers. Working conditions play a significant role in teacher sorting in initial job matches and in teacher moves over time.

We also know, just by looking at such information as the patterns of higher and lower “effectiveness” scores in the LA Times analysis, that if we dismiss teachers on the basis of their value added scores, we will be dismissing larger shares of teachers in higher poverty, higher minority schools. Or, we can just take the Central Falls, RI approach and declare the entire school failing based on its average performance over time (setting aside demographics and resources) and just fire everyone. Surely the replacements will be better. How could we do worse? Right?

Here’s the thing – even if we assume that some of the lower performance of teachers in poorer LA schools or the lower performance of Central Falls HS is a function of a weaker, less effective teacher workforce, we can only make things “better” by replacing that workforce with “better” teachers.

It is completely arrogant to take the reformy attitude of “how can we possibly do worse?” How could we possibly get a worse pool of teachers than the lazy slugs already in the system?

If the teacher pool in these schools is in fact less effective, and don’t just look that way statistically because of other factors, it may just be that these schools had a difficult time recruiting and retaining teachers to begin with. If we introduce our “game changing” policies – firing all of the teachers for low school performance, or firing individual teachers for bad effectiveness ratings – we will likely make things even worse.

Any teacher wishing to step in line to replace the previous cohort of “failures,” will have to not only consider the difficult working conditions but also the disproportionate likelihood that she/he will be fired a few years down the line, for factors well beyond his/her control (e.g. that pesky non-random assignment problem). That’s a significant change in working conditions – job risk. Without either changing other working conditions or substantially increasing compensation to offset this new risk, the applicant pool is not likely to get better – especially when risk is not increased similarly in other “more desirable” school districts. All else equal, the applicant pool is likely to get worse. The disparity in the quality of applicants for teaching positions is likely to increase dramatically, and the average quality of applicants to high poverty, high minority concentration districts may decline significantly.

Bonus video with thanks to Sherman Dorn: