Dumbest completely fabricated (but still serious?) graph ever! (so far)


Okay. You all know that I like to call out dumb graphs. And I’ve addressed a few on this blog previously.

Here are a few from the past: https://schoolfinance101.wordpress.com/2011/04/08/dumbest-real-reformy-graphs/

Now, each of the graphs in this previous post and numerous others I’ve addressed, like this one (From RiShawn Biddle) had something over the graph I’m going to address in this post. Each of the graphs I’ve addressed previously at the very least used some “real” data. They all used it badly. Some used it in ways that should be considered illegal. Others… well… just dumb.

But this new graph, sent to me from a colleague who had to suffer through this presentation, really takes the cake. This new graph comes to us from Marguerite Roza, from a presentation to the New York Board of Regents in September. And this one rises above all of these previous graphs because IT IS ENTIRELY FABRICATED. IT IS BASED ON NOTHING.

Perhaps even worse than that, the fabricated information on this illustrative graph suggests that its author does not have even the slightest grip on a) statistics, b) graphing, c) how one might measure effects of school reforms (and how large or small they might be) or d) basic economics.

Here’s the graph:

http://www.p12.nysed.gov/mgtserv/docs/SchoolFinanceForHighAchievement.pdf

Now, here’s what the graph is supposed to be saying. Along the horizontal axis is per pupil spending and on the vertical axis are measured student outcomes. It’s intended to be a graph of the rate of return to additional dollars spent. The bottom diagonal line on this graph – the lowest angled blue line – is intended to show the rate of return in student outcomes for each additional dollar spent given the current ways in which schools are run. Go from $5,000 to $25,000 in spending and you raise student achievement by, oh about .2 standard deviations.

Note, no diminishing returns (perhaps those returns diminish well outside the range of this graph?). It’s linear all the way – keep spending an you keep gaining…. to infinity and beyond. But I digress (that’s the basic economics bit above). And that doesn’t really matter – because this line isn’t based on a damn thing anyway. While I concur that there is a return to additional dollars spent, even I would be hard pressed to identify a single estimate of the rate of return for moving from $5k to $25k in per pupil spending.

Where the graph gets fun is in the addition of the other two lines. Note that the presentation linked above includes a graph with only the lower line first, then includes this graph which adds the upper two lines. And what are those lines? Those lines are what we supposedly can get as a return for additional dollars spent if we either a) spend with a focus on improving teacher effectiveness or b) spend “utilizing tech-based learning systems” (note that I hate utilizing the word utilizing when USE is sufficient!). I have it on good authority that the definitions of either provided during the presentation were, well, unsatisfactory.

But most importantly, even if there was a clear definition of either, THERE IS ABSOLUTELY NO EVIDENCE TO BACK THIS UP. IT IS ENTIRELY FABRICATED.  Now, I’ve previously picked on Marguerite Roza for here work with Mike Petrilli on the Stretching the School Dollar policy brief. Specifically, I raised significant concern that Petrilli and Roza provide all sorts of recommendations for how to stretch the school dollar but PROVIDE NO ACTUAL COST/EFFECTIVENESS ANALYSIS. 

In this graph, it would appear that Marguerite Roza has tried to make up for that by COMPLETELY FABRICATING RATE OF RETURN ANALYSIS for her preferred reforms.

Now let’s dig a little deeper into this graph. If you look closely at the graph, Roza is asserting that if we spend $5,000 per pupil either a) traditionally, b) focused on teacher effectiveness or c) on tech-based systems, we are at the same starting point. Not sure how that makes sense… since the traditional approach is necessarily least productive/efficient in the reformy world… but… yeah… okay.  Let’s assume it’s all relative to the starting point for each…which would zero out the imaginary advantages of two reformy alternatives… which really doesn’t make sense when you’re pitching the reformy alternatives.

Most interesting is the fact that Roza is asserting here that if you add another $20,000 per pupil into tech-based solutions – YOU CAN RAISE STUDENT OUTCOMES BY A FULL STANDARD DEVIATION. WOBEGON HERE WE COME!!!!! Crap, we’ll leave Wobegon in the dust at that rate. KIPP… pshaw… Harlem-Scarsdale achievement gap… been there done that! We’re talking a full standard deviation of student outcome improvement! Never seen anything like that – certainly not anything based on… say… evidence?

To be clear, even a moderately informed presenter fully intending to present fabricated but still realistic information on student achievement would likely present something a little closer to reality than this.

Indeed this graph is intended to be illustrative… not real…. but the really big problem is that it is NOT EVEN ILLUSTRATIVE OF ANYTHING REMOTELY REAL.

Now for the part that’s really not funny. As much as I’m making a big joke about this graph, it was presented to policymakers as entirely serious. How or whether they interpreted it as serious, who knows. But, it was presented to policymakers in New York State and has likely been presented to policymakers elsewhere with the serious intent of suggesting to those policymakers that if they just adopt reformy strategies for teacher compensation or buy some mythical software tools, they can actually improve their education systems at the same time as slashing school aid across the board. Put into context, this graph isn’t funny at all. It’s offensive. And it’s damned irresponsible! It’s reprehensible!

Let’s be clear. We have absolutely no evidence that the rate of return to the education dollar would be TRIPLED (or improved at all) if we spent each additional dollar on things such as test score based merit pay or other “teacher quality” initiatives such as eliminating seniority based pay or increments for advanced degrees. In fact, we’ve generally found the effect of performance pay reforms to be no different from “0.” And we have absolutely no evidence on record that the rate of return to the education dollar could be increased 5X if we moved dollars into “tech-based” learning systems.

The information in this graph is… COMPLETELY FABRICATED.

And that’s why this graph makes my whole new category of DUMBEST COMPLETELY FABRICATED GRAPHS EVER!

Advertisements

7 Comments

  1. Something that I think might make this post even more powerful (when linked as a single entity) is a good explanation of what a “standard deviation change in student performance” might actually mean if you did in fact measure it. Truthfully, I have no idea what they’re purporting to measure and so I find moving one standard deviation to the right of kittens and unicorns somewhat difficult to picture.

    Since in general, standard deviation is a measure of the variability of your data, moving one standard deviation – if you could do it for your small cohort among the vast universe of students – might be a relatively small change in performance or it could be a dramatic change in performance. (Note also it did not specify where on the bell curve we started; empirically, moving from the lowest sixth to the next one is quite a bit easier than moving up when you’re above average.)

    And, of course, the corollary that because it’s a measure of variance within the population, if your entire population increases scores by what used to be one standard deviation… you end up on the same place in the curve. No matter how hard the teachers work, 50% of American children will remain below average. It’s just one of those sad truths of mathematics.

  2. I love this article. I can almost guarantee that the “graph” generated few questions about its validity or sense nor did it generate outrage that it was due. Critical thinking skills must be absent in all things reformy!

  3. Hey, I could build worse graphics than that!

    (Slightly) more seriously, el above has a good point. If I were to, er, interpret that graphic (think Martha Graham with two left feet) it would go like this:

    1) If we maintain the current system, the current system is maintained. We can add money and realise gains, but that’s about it. Our kids’s educations will still keep improving about a grade level every five or seven years, and we might be able to live with that.

    2) If we put more money into “improving teacher effectiveness” (which is pretty damned good now, see [1] above), we will create some disruptions in the system. There will be more teacher turnover–probably mosty in areas that have higher poverty rates–less institutional knowledge, more hesitancy on the parts of some to try to expand their students’s horizons (because we’re measuring ‘effectiveness,” and “anecdotes” are not data, except in old Mark Harmon-Kirstie Alley movies), and therefore we will see greater variance (shown here as higher standard deviations, which is the square root of same) in progress.

    3) Finally, if we move to an intensively high-tech environment–major capital expenditures required, not to mention the cost of maintaining umpteen hundred devices, networks, cables, routers, RAID systems, and other stuff–you can’t just pay the janitor to clean the blackboard when you’re using the (POS that is) Blackboard). then we’re dumping a lot of capital that will be outdated within a few years into every school system. And that money has to come from somewhere–I told you we were going to save monies–so we’re going to do that by reducting teaching staff in those schools that cannot afford to make such capital expenditures from their shrinking tax bases, which is probably most of them. So there will be Virtual Teachers in some classrooms, which may impact performance even with all of our great technological advances.

    This suggests that there will be even more variance in performance improvements from the baseline using this method. But note that variance means that the places (few, perhaps, but anyway…) that are able to afford, maintain, and train their staff in the new technologies will outperform in the same manner than (many) others will underperform.

    So that’s what this slide shows you–we can make “improvements” (evidence-based or not) in different areas, but since the cost of those improvements is going to have an impact on other (proven) methods, there will be greater variance in test score improvements, as shown by this graph.

    But more of the “top” kids will show improvements, so you should do this, since all of your kids are, of course, Above Average.

    1. Actually, more than anything, I’ve been pondering what goes on in the range of spending between $0 and $5,000. Do the three lines converge and stay together? It would appear that there is no return to dollars spent up to $5,000. That’s interesting. You’d think you could get something for that. Or, perhaps we actually do harm to student outcomes below that level. And, if I extrapolate to the left here, assuming each to be a continuous linear trend, then tech-based systems and “teacher effectiveness” systems actually do more harm than traditional approaches at spending levels below $5,000. Is this because the minimum threshold start up costs for those systems is necessarily no less than $5,000 (to break even with traditional systems?). And if we implement them severely underfunded that kids just go brain dead. Scary proposition. But it is these very absurdities of this graph that lead me to my very bold statements about it – that it is – it must be – it can only be completely fabricated. I wouldn’t go so far in print if that wasn’t clearly the case.

      Now, you bring up some fun stuff in you tech-based examples above… which of course implies that there might be some serious infrastructure related costs, which require more frequent updating than other infrastructure, all of which might actually dampen that outstanding productivity curve of tech-based learning systems. And, of course, there’s that tricky little bit about this stuff maybe, just maybe working differently for some kids than others. Of course one might expect that if there is going to be experimentation going on, that the experimentation will occur on kids who would be less likely to benefit. This is an important ethical concern that is being completely blown off by individuals like Roza. They are pitching entirely unfounded snake oil solutions as if they fall along this mythical productivity curve, and they are urging policymakers to experiment with these untested remedies on the state’s most vulnerable children. Let’s just say that makes me a bit uncomfortable. And what makes me more uncomfortable is that it doesn’t seem to make them uncomfortable at all?????

      In any case, I look forward to your graphic rendering of an even worse figure. I’m sure it can be done. And I’m sure I’ll see it soon, and likely from someone intending their graph to be serious. Hence my parenthetical in the title. It never ends. It just never ends. Thanks!

Comments are closed.