Searching for Superguy in Gotham

Who is Superguy? By most popular accounts, Superguy is a figure of mythical proportion (urban legend proportion) capable of swooping down into the poorest of urban neighborhoods of America’s largest cities, gaining immediate access to schooling facilities, rounding up unthinkable private contributions from wealthy philanthropists and quite simply saving the lives of low-income urban school children trapped in bleak, adult-centered, perpetually failing traditional public schools.

Superguy could be found anywhere in the U.S where urban charter schools have proliferated in the past decade – Kansas City, Washington D.C., Chicago, Dallas, Houston, or even more likely, New York City – Gotham itself (yeah… Gotham was a Batman thing, not Superman… but hang in there with me).  I’ve chosen to focus on urban locations here, because who has ever heard of a “rural legend?”

I’ve written on a number of occasions about my general skepticism that Superguy really exists, or that he necessarily exists in the form of an urban charter school operator. My skepticism is based on my own read of the balance of research on charter schools and my own casual analysis  of New York City and New Jersey Charter Schools. New Jersey Charter Schools in particular are pretty average and those that are better than average serve very few of the lowest income children, no special needs children and few or no limited English proficient children. Personally, I’d expect Superguy to be out there fighting for these kids in particular, not just setting up shop in their neighborhood and cream-skimming the less needy among the more needy. But hey, that’s just my notion of what Superguy should be.

For an exceptional review of charter school research, I would recommend Robert Bifulco and Katrina Bulkley’s chapter on Charter Schools in the Handbook of Research on Education Finance and Policy. Neither of these scholars are charter school naysayers, yet they conclude:

Research to date provides little evidence that the benefits envisioned in the original conceptions of charter schools – organizational and educational innovation, improved student achievement, and enhanced efficiency – have materialized.

Of course, the true believers in Superguy (as charter operator) will argue vehemently that the finding that charters, on average, are average does not shake their belief… because the “upper half of charter schools is really good!, better than average, in fact!” Skeptically, I respond – isn’t the upper half of all schools better than average? If so, might Superguy actually be found in any school that’s better than average? But who am I to nitpick?

The most compelling evidence that Superguy exists was provided in Caroline Hoxby’s finding regarding NYC charter schools that:

On average, a student who attended a charter school for all of grades kindergarten through eight would close about 86 percent of the “Scarsdale-Harlem achievement gap” in math and 66 percent of the achievement gap in English.

Who other than Superguy could close the Harlem-Scarsdale gap? However, Stanford University researcher Sean Reardon explains:

Because the report relies on an inappropriate set of statistical models to analyze the data, however, the results presented appear to overstate the cumulative effect of attending a charter school.

Superguy in Gotham is also assumed to have competitive effects, lifting entire neighborhoods wherever he may be present. This evidence is often cited to Marcus Winters’ (Manhattan Institute) finding that:

The analysis reveals that students benefit academically when their public school is exposed to competition from a charter.

But thwarting this Superguy sighting is Wellesley economist Patrick McEwan’s observation that:

The statistical analysis suggests that increasing competition has no statistically significant impact on math test scores, but that it has small positive effects on language scores. The report does not conclusively demonstrate that the results are explained by increasing competitive pressure on public school administrators; they may also be explained by shifting peer quality or declining short-run class sizes in public schools.

Those pesky, curmudgeonly,  academics are at it again… denying the true believers that Superguy comes in the singular form of a New York City charter school operator!

Then there’s the claim that Superguy himself may have been outed in Harlem (is Superguy really Geoffrey Canada?) – as evidenced by Dobbie and Fryer’s studies of the amazing success of Harlem Children’s Zone. But then Russ Whitehurst of Brookings stepped in to rain on this parade, finding the HCZ Promise academy to be relatively average as far as NYC charter schools go.

For several additional curmudgeonly critiques of Superguy sightings, see: http://epicpolicy.org/search/epicpolicy/charter

It is with these contradictory findings in mind that I present the following figures, and we begin our statistical search for Superguy. Now, this is not a really deep, statistically rigorous search for Superguy. The approach here is what some refer to as a “beating the odds” approach (BTO) and is similar to the adjusted performance approach used by Whitehurst in his Brooking critique of HCZ.

It seems that the logical place to start would be New York City, home to the greatest number of Superguy sightings. Let’s begin with a simple flyover of NYC schools, including traditional public schools and charter schools just to get a feel for the demographics of those schools.

Here is the % of children qualifying for Free Lunch across Harlem (and South Bronx) schools. This map does not indicate which schools are charters, but you can click the link below the map to see which ones are.

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS (indicated with an asterisk)

Here is the % of children who are limited in their English language proficiency. Again, this map doesn’t show us which schools are charters, but you can click the link below.

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS (indicated with an asterisk)

Now, here’s our first “Beating the Odds” scatterplot. The predicted performance values expressed on the vertical axis are from a regression equation that accounts for a) limited English proficiency shares, b) free lunch shares, c) mobility shares, d) borough and e) year (includes 2008 & 2009). These graphs look at the adjusted performance levels (not value-added) of NYC traditional public and NYC charter schools (standardized difference between actual and predicted values for 2009). These are illustrative. While outcome levels do go up (are inflated) in 2009, the distributions in these scatterplots don’t change a whole lot if I use 2008 or earlier. (here are the models)

The first Beating the Odds scatterplot looks at the average performance from 2009 (yes, the really inflated test score year) for NYC public schools. Charter schools are not identified in this graph. Schools are displayed from low to high poverty along the horizontal axis. Schools above the red horizontal line are beating the odds, or scoring higher than expected given their location and student population. Schools below the line are, well, not beating the odds. We would, of course, expect our Superguy operated schools to be flying high above the rest and certainly not falling well below… at the bottom of the scatter. So, where is Superguy?

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS

Now, here’s the average of the 4^th and 5^th grade outcomes. Same deal. A good ol’ BTO analysis (yeah… this isn’t really rigorous stuff, but it is illustrative). Again, charters are not identified in this picture. Yes, there are some high and some low flyers in this graph too. But are all of the high flyers charter schools? Is Superguy really here?

CLICK HERE TO SEE WHICH SCHOOLS ARE CHARTER SCHOOLS

While I don’t think we’ve found Superguy here, we are left with some potential clues about the conditions surrounding Superguy sightings – A) that superguy sightings seem more common in the presence of unexplainable deficits in the shares of children who qualify for Free Lunch, B) that Superguy sightings seem more common in the presence of unexplainable deficits in the shares of children with limited English proficiency. Other than that, it seems that Superguy is equally likely to be hiding in a traditional New York public school as it is that Superguy is secretly disguised as a charter school operator somewhere in Gotham.

Alternatively, there exists the depressing but real possibility that Superguy simply doesn’t exist – at least not in the expected form. That there just isn’t a charter school operator out there who can single-handedly swoop into poor urban neighborhoods and save childrens lives – creating results never seen before with a truly representative population of children. Or, at the very least, not all or even the average charter school operator qualifies as Superguy. Yes, some are better than others. And, some are quite good. But you know what I have to say about that argument (see above).

Yeah… I’d like to be a believer. I don’t mean to be that much of a curmudgeon. I’d like to sit and wait for Superguy – perhaps watch a movie while waiting (gee… what to watch?). But I think it would be a really long wait and we might be better off spending this time, effort and our resources investing in the improvement of the quality of the system as a whole. Yeah, we can still give Superguy a chance to show himself (or herself), but let’s not hold our breath, and let’s do our part on behalf of the masses (not just the few) in the meantime.