Newsflash: The upper half is better than average!

Dumbest Stuff I've Ever Read! 5 Minutes

I’ve seen many versions of this argument in the past year, but this one comes from Kevin Carey in response to the Civil Rights Framework which criticized the current administration’s overemphasis on Charter Schools as lacking evidentiary support. Carey responds that the Civil Rights Framework selectively interprets the research on Charter schools, noting:

Here’s the problem: the contention that charters have “little or no evidentiary support” rests on studies finding that the average performance of all charters is generally indistinguishable from the average regular public school. At the same time, reasonable people acknowledge that the best charter schools–let’s call them “high-quality” charter schools–are really good, and there’s plenty of research to support this.

http://www.quickanded.com/2010/08/evidence-and-the-civil-rights-group-framework.html

I recall a similar comment in the media a few months back, by a researcher, regarding a national charter schools study – something to the effect of – Charter schools on average performed similarly to traditional public schools, but if we look at the upper half of the charter schools in the sample, they substantially  outperformed the average public school serving similar students.

These statements have been driving me crazy for months now. Here’s why –

To put it in really simple terms:

THE UPPER HALF OF ALL SCHOOLS OUTPERFORM THE AVERAGE OF ALL SCHOOLS!!!!!

or … Good schools outperform average ones. Really?

Why should that be any different for charter schools (accepting a similar distribution) that have a similar average performance to all schools?

This is absurd logic for promoting charter schools as some sort of unified reform strategy – Saying… we want to replicate the best charter schools (not that other half of them that don’t do so well).

Yes, one can point to specific analyses of specific charter models adopted in specific locations and identify them as particularly successful. And, we might learn something from these models which might be used in new charter schools or might even be used in traditional public schools.

But the idea that “successful charters” (the upper half) are evidence that charters are “successful” is just plain silly.

=======

Let’s throw a few visuals and numbers on my whining session above.  Below are some snapshots of New York City Charter schools. First, lets take a quick look at the mismatched demographics of New York City charters compared to same grade level traditional public schools. Here are the Free Lunch rates. I’ve tended to focus on Free Lunch rates rather than Free and Reduced, because Free Lunch falls under a lower poverty threshold, and, as my previous analyses have shown, while charters often serve similar numbers of combined free and reduced lunch children, they tend to serve the less poor among the poor (larger reduced shares, smaller free shares). This graph confirms my previous findings, and is based on data corroborated from both the NCES Common Core, Public School Universe Data from 2007-08 and the New York State Education Department School Report Cards.  Note also that the biggest differences are at the elementary level, which covers most of the charter schools.

Second, let’s look at the rates of children who are limited in their English Language proficiency. Here, the differences at the elementary level are huge! Charters in NYC simply don’t serve limited English proficient children!

Now for a few oversimplified scatterplots comparing charter school performance outcomes to traditional public schools – all “Regular Schools” by the school type classification in then NCES Common Core – and compared against those in the same borough. I’ve focused on Brooklyn and the Bronx here because of the wide variations in student population composition across Manhattan schools.

First note that none of the charters in the Bronx which had 8th grade 2009 test scores available had a free lunch rate over 80%, while several traditional public schools in the Bronx did. This chart shows the relationship between % scoring level 4 (top level) and % qualifying for free lunch. Charters are named and shown in red. Traditional publics are hollow circles. Both groups scatter! In fact, there are a few traditional publics at the top (which may be classified as “regular schools” but may be far from regular). Among Charters, Bronx Prep, KIPP Academy and Icahn 1 do rather well. Hyde Leadership (higher poverty than the other charters) and Harriet Tubman – not so well. But there are plenty of traditional public schools in the Bronx that appear to do well, and others not so well.

Here are the Brooklyn Charters and traditional public schools on the same outcome measure – percent scoring level 4 or higher on 8th grade math.  Here, all but Brooklyn Excelsior Charter have much lower poverty rates and simply aren’t comparable to most Brooklyn traditional public schools. And don’t forget, there are also likely very large differences in rates of children with other needs – like limited English proficiency. Williamsburg Collegiate and Brooklyn Excelsior appear to be doing quite well. But then again, Williamsburg Collegiate starts at 5th grade, so their success is likely at least partly a function of feeder schools.  There are plenty of “high flying” traditional public schools in this picture as well… and likely a few unique explanations as to why they fly so high. There are also plenty of low-flying charters. Here are the Bronx Charters in 2009, on 5th grade math. Again, the charters generally have much lower free lunch rates than the traditional public schools. In this figure, most of the traditional public schools have free lunch rates over 80% while none of the charters do. And again, charter performance, like traditional public school performance is scattered – some low – some high.

And finally, those Brooklyn charters on 5th grade math performance. Low poverty and scattered (except Brooklyn Excelsior which is higher poverty, and seemingly doin’ pretty well).A few new ones – Here are the Bronx and Brooklyn charter 5th grade performance levels based on a regression model controlling for stability rates, free lunch, ELL concentrations, year of data (using 2008 and 2009) and comparing specifically against other schools in the same borough. The performance levels are represented by the residuals of the regression model. Above “0” on the vertical axis is “better than predicted – or better than average at given characteristics, and below “0” is below expected at given characteristics.

In these graphs, most of the highest high flyers are non-charters. Charters are split above and below the “0” line, as one might expect.

Anyway, on this cursory walk through of the relative demographics and relative position of charters in the performance mix, it continues to evade me as to why we should be considering “charters” as a specific reform strategy and one that can raise urban school districts from their dreadful depths of failure. Had I not indicated which schools were charters in these graphs, I wonder how many “reformy” types could have picked out the dots that were charters. I suspect, given a blind sample, they would select the dots that fall furthest out of line in the upper right hand corner of each graph – the highest performing high poverty schools.  In three of the above 4 graphs, they’d have picked non-charters first and would have done so on the misguided perceptions that a) charters are the high flyers in any mix of schools and b) charters serve very high poverty populations. The reality is that charters are as scattered as traditional schools, and in general in NYC, they are serving lower need populations.

=======

A little more fun here. Here are schools in the area around the Harlem Children’s Zone. First, here are the maps of free lunch shares and LEP shares for charter and traditional public schools.  Green dots have lower rates of LEP or free lunch. Stars indicate charters. Names are adjacent to schools. Note that most of the charters are lower poverty and much lower LEP than surrounding schools.

And here are the residuals of the same regression model used above, applied in this case to Grade 5 Math Mean Scale Scores. Red dots are schools that perform less well than expected and green dots are those that perform much better than expected. Note that charters are a mixed bag, and the HCZ charter performs particularly poorly – which caught me off guard.

Published by schoolfinance101

Bruce Baker is an Professor in the Graduate School of Education at Rutgers, The State University of New Jersey. From 1997 to 2008 he was a professor at the University of Kansas in Lawrence, KS. He is lead author with Preston Green (Penn State University) and Craig Richards (Teachers College, Columbia University) of Financing Education Systems, a graduate level textbook on school finance policy published by Merrill/Prentice-Hall. Professor Baker has written a multitude of peer reviewed research articles on state school finance policy, teacher labor markets, school leadership labor markets and higher education finance and policy. His recent work has focused on measuring cost variations associated with schooling contexts and student population characteristics, including ways to better design state school finance policies and local district allocation formulas (including Weighted Student Funding) for better meeting the needs of students. Baker, along with Preston Green of Penn State University are co-authors of the chapter on Conceptions of Equity in the recently released Handbook of Research Education Finance and Policy, and co-authors of the chapter on the Politics of Education Finance in the Handbook of Education Politics and Policy and co-authors of the chapter on School Finance in the Handbook of Education Policy of the American Educational Research Association. Professor Baker has also consulted for state legislatures, boards of education and other organizations on education policy and school finance issues and has testified in state school finance litigation in Kansas, Missouri and Arizona. He is a member of the Think Tank Review Panel, a group of academic researchers who conduct technical reviews of publicly released think tank reports on education policy issues.

Published