We all have our flaws. As social science researchers, writers, reviewers, etc. dealing with highly political and politicized topics, we all let our personal biases creep inappropriately into our work at times. In the world of education policy writing and research, there are many occasions where individuals and organizations are provided opportunities to selectively review and present data and summaries of existing literature in order to make one point or another or argue one side of an issue or another. There will always be reasonable differences of opinion on complex social science issues, both with regard to the methods that should be used to provide definitive answers, and the definiteness of the answers themselves.
All of that aside, there are, I believe some really basic rules of thumb that should be followed and some offenses that should be considered so egregious that they warrant public rebuke and removal from the table around which public discourse occurs on key education policy topics.
My latest example of an offense so egregious comes from a review I completed today on a report from Lisa Snell of the Reason Foundation – the Weighted Student Formula Yearbook 2009. Reason Foundation is a Think Tank, so-to-speak, or at least fashions itself as such. There are many “Think Tanks” across the political spectrum and these Think Tanks frequently chime in with well-publicized findings regarding education policy and finance issues. So, this is not new. That said, this particular report is simply so awful, I can’t shake it from my head.
To summarize, the report touts urban school reforms which involve decentralized management and school choice, two issues on which I personally do not have strong opinions one way or the other, from an ideological perspective (rather, my concern is that if one pursues a particular reform, it should be done using appropriate evidence, and should be done well – or as well as can be – and not half-baked). The report inappropriately casts the varied reforms presented as Weighted Student Formulas – a specific reform which involves using a student need focused funding formula to drive financial resources to schools – something which, if done well, could be quite a good idea.
So… I digress… what is so awful then about the Reason report? Why do I think their license to think should be revoked and their seat at the education policy debate table eliminated? In an effort to make their case that the reforms implemented in 14 cities and 1 state have all been highly successful, the authors go so far as to cite evidence of improved student outcomes – in one third of their 15 cases – that occurred in years prior to the implementation of the reforms. That is, the reforms somehow, amazingly, through space-time travel perhaps, had retroactive effects on student outcomes. I’m dumbfounded!
Here’s a section of my review:
=====
Most problematic is the fact that in five of the 15 cases discussed (one third of the cases), outcome successes mentioned actually occurred prior to the implementation of WSF or SBB/SBM (see Appendix A). For example, the report commends NYC for winning the 2007 Broad prize, the year before its Fair Student Funding policy was implemented. The report might arguably attribute this success to mayoral takeover, which began in 2002. But this is the WSF Yearbook, and even the expansive definition of WSF used in the report did not encompass mayoral takeovers.
Similarly, the report commends Hartford for raising test scores in 2008, the year before implementing WSF, and it commends Denver for making strong improvements between 2005 and 2008, whereas WSF was implemented in 2008-09. These successes lead one to question why these districts would want to implement WSF and risk undoing their prior achievements.
The most egregious claim of retroactive causation appears in the press release for the report:
The results from districts using student-based funding are promising. Prior to 2008, less than half of Hartford, Connecticut’s education money made it to the classroom. Now, over 70 percent makes it there. As a result, the district’s schools posted the largest gains, over three times the average increase, on the state’s Mastery Tests in 2007-08 (emphasis added).[i]
Yet, the report itself states that Hartford only began implementing WSF in 2008-09, and only expected to achieve the 70 percent target of available resources allocated to schools and classrooms by 2009-2010 (p. 61). It is difficult to conceive of any defense for Reason’s claims.
=====
Link to complete review: http://www.epicpolicy.org/files/TTR-Baker-Reason-WSF.pdf
I’ve personally come to refer to this offense as the Ouchi Retroactive Causation effect, since the first time I encountered such an absurd argument was in the work of William Ouchi who made much the same analytic gaffe in reference to the success of reforms in Houston. Scott Thomas and I, in our report on Weighted Funding for Hawaii noted: “… most obviously problematic is the authors’ touting of Houston’s success in improving performance through its WSF and decentralized governance, based on performance data from 1999 to 2001, when the formula was implemented (first year of phase-in) in 2001-02” (Baker and Thomas, 2006, p. 9).
http://www.hawaiireporter.com/file.aspx?Guid=1e930a29-3017-40fc-b58c-cf1513149ce7
For a more thorough critique of the work of Ouchi on this topic, see my article with Doug Elmer in Educational Policy (vol. 23 no. 1, 2009)
Look… we all make errors of logic and errors of analysis. So too do relatively safe and conscientious drivers get into an occasional auto accident. However, if a driver drives 15 times in one week, and drives drunk 5 times, that driver should lose his/her license. Claiming retroactive causation 5 times in 15 cases in a report which then proposes “best practices” for schools is comparably irresponsible. Arguably, the first or second offense is bad enough.
If nothing else, the Reason Foundation and the Eli and Edythe Broad Foundation which provided support for the Weighted Student Formula Yearbook should evaluate carefully whether such haphazard and arguably reckless thinking really advances the public policy agenda in any productive way. Whatever good elements there may be in the attempts of these districts to improve educational opportunities for the children they serve are severely undermined by the Weighted Student Funding Yearbook.
School Finance 101 
Ah, but if one is forcing an “ideological saw to sing” there are data and then there are political objectives–and these are occasionally mutally exclusive.
I like the “Ouchi Retroactive Causation Effect.” That will be shared…..