Still Searching for Miracle Schools and Superguy: Updates on Houston and New York City


I was following a conversation on Twitter a short while back in which one student activist – Stephanie Rivera of Rutgers asked another – Alexis Morin from Students for Education Reform – why SFER chooses to focus almost exclusively on charter schools as beacons of “success” and thus a significant part of the “solutions” for urban education moving forward. Observing this interaction brought me back once again to the astounding gaps in logic which are so pervasive in the current reform rhetoric which seeks to find policy solutions almost exclusively in charter schools and in changing teacher compensation and dismissal policies.  The reformy solutions are pretty much a given regardless of the original question or what the analyses yield.

Too often, the following faulty reasoning is applied in search for solutions in the education reform debate:

  1. Scan the horizon for successful charter schools (even though charters are no more successful, on average than their more dominant counterparts – district schools). [Charter schools on average, are average. Some are above average and others, well, not so much. Because charters are still much smaller in total numbers, if one was to simply look for “good” schools, the odds of picking a charter would in fact be smaller.]
  2. Assume that better-than-average charter schools – successful ones – are better than traditional public schools. [ignoring that while, by virtue of being the upper half of a similar distribution, they are really no different from the upper half of traditional public schools].
  3. Assume that because the better-than-average charter schools are better than the average traditional public school, that being a charter school  – bearing the label/classification “charter” – has something to do with it. [even though a comparable – or even larger – share of other schools bearing the label “charter” are actually doing worse than the average traditional public school].
  4. Assume that bearing the label – “charter” – necessarily means that these schools have and use creatively and inventively the substantially greater autonomy granted to them. [That is, they certainly don’t waste their time on stuff like spending more money, providing smaller class sizes and paying teachers more].

I have seen this utterly ridiculous stream of contorted logic rolled out on numerous occasions in the past few years, and even in the past few weeks.

A while back, I posted two separate entries called “Searching for Superguy” – one for New York City and one for New Jersey – in order to display the distribution of performance, corrected for demographics, for New York City and New Jersey charter schools.

Since that time, I’ve compiled quite a bit more data on charter (and other) schools in a variety of settings. I’ve also developed a clearer vision of exactly what constitutes one of those “miracle” schools we’re all searching for. A miracle school is characterized by at least the following four factors:

  1. Serves the same kids (poverty, language proficiency, disability)
  2. Spends less than other schools serving similar kids
  3. Has high average outcomes compared to schools serving similar kids
  4. Achieves better value-added on measured student outcomes than other schools

For today’s post, I offer you a tour of charter schools in New York City and in Houston Texas – two cities with significant concentrations of charter schools and two cities with significant numbers of charter schools affiliated with major charter management organizations.

A Tour of Houston and New York City Charter Schools

Serve the Same Kids?

The first question, of course, is do charter schools in these cities serve the “same kids?” as traditional public schools in the same city/borough and at the same grade level. To answer this question, I estimate regression models to three years of data (2008 to 2010) where the population characteristic of interest is the dependent variable, and a)year of data, b) location of school (city) and c) grade level/range are the independent variables. Houston is treated as a single city (uh… because it is) and New York is carved into boroughs for this analysis. That is, schools are compared with same grade level schools in their borough. Charter Schools are lumped together by CMO, with schools not belonging to major CMOs lumped together for this analysis (they are indeed a very heterogeneous group). The analysis is weighted by the enrollment of the schools.

Figure 1 shows that in New York City, Charter Schools serve a) far fewer children who qualify for free lunch , b) far fewer LEP/ELL children and c) far fewer children with disabilities than school serving the same grade level in their borough. Uncommon schools are indeed the least common. But Success academies have particularly large deficits in LEP/ELL children. These schools simply aren’t comparable in terms of student populations.

Figure 1. New York City Demographics

Figure 2 shows the Houston schools by CMO. With my present data I was unable to parse free from free or reduced price lunch, which may be important. But, this figure and some of my previous analyses confirm that at least in Houston, charters are doing a better job of serving low income kids (than NYC charters). However, most Charter CMOs still seem to have a significant aversion to children with disabilities and in most cases, also to children who are LEP/ELL.

 

Figure 2. Houston Demographics

Spend Less?

The spending analysis is conducted similarly, using 3 years of data and using a regression model where spending per pupil is the dependent variable and where student characteristics, grade level and year are the independent variables. This analysis plays off our recent NEPC report, using the same data (expanded to include all NYC charters, and cleaned&merged with additional measures) and same methods.  I have stayed with my original expenditure measure for BOE schools (defended here) and have used Annual Financial Report (nor IRS 990) data for charters.

Figure 3 shows that in New York City, charters are generally outspending traditional public schools serving similar student populations. KIPP and Uncommon schools are outspending BOE schools by over  $4,000 and $3,500 respectively and Harlem Childrens Zone schools are spending similarly (and that’s not even counting all of the additional money flowing to/through the parent organization. it’s just the annual financial report data!).

Figure 3. New York City Spending

Figure 4 shows more of a mixed bag in Houston. Non-major-CMO charters spend less than district schools. KIPP’s elementary schools spend less, but not consistently/significantly so (some do). Cosmos/Harmony schools also spend less, but through different means.  Others are indistinguishable, with some network schools spending more and others less (some Yes Prep schools outspend districts schools serving similar students).

Figure 4. Houston Spending

Have High Average Outcomes?

The following several graphs explore ht distribution of spending and outcomes for individual schools. Note that these graphs don’t include all of the other stuff that might need to be included to parse whether spending differences actually help produce outcome differences. That’s not the point here. Rather, these are just descriptive graphs of the relative spending and outcomes of these schools – focusing on schools serving middle grades.

Figure 5 shows that each of the charters spends quite a bit more than otherwise similar district schools. Each charter also has higher average performance (except St. Hope) than district schools. But, as shown above, they also have less needy students.  In other words, no miracles by these measures. Higher average outcomes yes. Lower spending? NO. Same kids? NO!

Figure 5. New York City Average Outcomes

Figure 6 shows the average spending and outcomes for schools in Houston. Among KIPPs, all spend much more than district schools and two have higher than average outcomes and the other two have average outcomes. The Yes Prep school in the sample spends more and has higher than average outcomes. Meanwhile, some other charters spend less and do less well, and one spends less and has somewhat higher than average outcomes. Notably, however, the population characteristics of these schools were also mixed. It may be the case that these KIPP schools have more needy populations than the average district school, and on average are doing average to better. That’s not bad. However, we must acknowledge they are doing this at a much higher price! Similar kids? Mixed (more low income, fewer ELL or special ed). Better outcomes? Also mixed, but okay. Less money? No, actually more… a lot more.

Figure 6. Houston Average Outcomes (standardized)

Have Greater Gains?

Okay, here’s one last shot. Let’s look at achievement gains instead of just level of performance. I’ve constructed school level average gains for NYC schools by aggregating the teacher value added data for teachers in each school (weighted for the number of students who contribute to their scores in English Language Arts or Math). In other words, in this analysis a school is only as good as its teachers (consistent with reformy wisdom) and, for that matter, the children served by (linked in data records to) those teachers.

Figure 7 again shows that in New York City, charters tend to significantly outspend district schools with similar populations – well except Equality charter which is somewhat closer. On average, the average gains are indeed higher in these higher spending charters – actually moving upward in sort of a pattern. But remember, the peer groups in these schools also aren’t particularly comparable. KIPP AMP and Brooklyn prospect, however, don’t do so hot.  But, if there’s any case to be made here with these charters, resources just might matter. Not the same kids. More money. Some reasonable outcomes.

Clearly, some deeper investigation is warranted. But, in each case there are also district schools, including lower spending district schools that outperform most of the charter schools.

Figure 7. New York City Value-Added

Finally, we’ve got Figure 8, showing the distribution of school level value added ratings from the FAST TEXAS system. Here, the KIPPs in particular are more of a mixed bag. Some have higher and others lower value added.  Note that in Texas these “progress” metrics seem to be associated with student characteristics.  All of the KIPP schools and the Yes Prep school spend more than district schools. Clearly, some deeper investigation is warranted. But, in each case there are also district schools, including lower spending district schools that outperform all of the charter schools.

 

Figure 8. Houston Value-Added

As a bonus, I also have this slide on class sizes (8th grade math) for NYC schools serving 8th grade. I’ve related class size to spending here, showing that each of these higher spending charter chains (except for Democracy Prep) seem to be leveraging at least some of that funding to provide much lower class sizes.

Bonus Slide: Class Size and Relative Spending in NYC

Closing Thoughts

In closing, we all really need to step away from the misguided logic I laid out in the beginning of this post – that “charter” in and of itself is meaningful  and that the only answers for the future of education must be found among successful charter schools – especially miracle charters under the watchful eye of Superguy. If there’s anything we know by now about Superguy is that he’s got some pretty nice financial backing!

Superschools & Miracle-Guy… uh… wait… Superguy and Miracle Schools, if they do exist, are a rarity. Further, if they do exist, they are as likely if not more likely (by sheer numbers) to be traditional public schools and not charter schools. But regardless of the governance of the schools, when looking for ideas (not “solutions”) for how to improve educational opportunities, we should be focused on the following:

  1. Attempting to learn what we can from all types of schools and not predetermining that we should look at successful “charter” schools. This is especially true since charters are proportionately no more successful than other types of schools and, given that they are smaller in total numbers, they are in total, less numerous among all successful schools.
  2. Being really, really careful about parsing out reasons for success before declaring schools to be miracles (and we should avoid the whole notion of “miracle schools”). We must look closely at population characteristics and population dynamics (mobility/attrition/neighborhood changes).
  3. When we do find schools – charter or other – that appear to be making unexpected gains, we should explore what it is that they are doing. We should explore their resource use. We should explore the strategies they employ and we should figure out not just what it is costing them to adopt these approaches but what it would cost to adopt these approaches in other settings and more broadly.

Finally, publicness and true public access matters. While we may analyze and compare schools more thoughtfully regardless of their governance. I would now argue that when we consider the policy path forward we should actually give serious consideration to their governance.

All else equal, I am increasingly uneasy with the notion of creating larger numbers and shares of schools that are LIMITED PUBLIC ACCESS as I described in a previous post on charter schools. Intended or not, this is what has become of large segments of charter schooling.

I am concerned with the effect of expanded limited public access schools on those truly public schools around them. Intentional or not, this is how charter expansion seems to be playing out.

I am equally if not more concerned with the idea of shifting larger shares of children into schools where those children and their parents may forgo their constitutional and statutory protections, except where explicitly laid out in state charter statutes. Intended or not in the letter of state charter statutes, it is the charter operators themselves who invariably invoke their “private” status when defending the stifling of teachers’ free expression, or teachers’ claims seeking damages (under federal law) for mistreatment by a state actor, or students/parents claims regarding strict enforcement of discipline codes.

The maintenance of constitutional protections and true public access – non-exclusion – MUST play a significant role in the determination of the path forward. We should not be too quick to trade constitutional protections to employee or student free speech, privacy rights, protections from unreasonable searches and various statutory rights for a few additional points on state assessments or for a few dollars cut from school spending.

That said, the figures laid out above suggest that we likely aren’t even getting systematic or sizable bang for the buck when/if we do trade these constitutional protections. So perhaps that point is moot.

Related Resources

Baker, B.D., Libby, K., & Wiley, K. (2012). Spending by the Major Charter Management Organizations: Comparing charter school and local public district financial resources in New York, Ohio, and Texas. Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/publication/spending-major-charter.

(Follow up: https://schoolfinance101.wordpress.com/2012/05/07/no-excuses-really-another-look-at-our-nepc-charter-spending-figures/)

Baker, B.D. & Ferris, R. (2011). Adding Up the Spending: Fiscal Disparities and Philanthropy among New York City Charter Schools. Boulder, CO: National Education Policy Center, 33. Retrieved April 24, 2012, from http://nepc.colorado.edu/publication/NYC-charter-disparities.

Advertisements

20 Comments

  1. I always applaud this work as it is useful when talking with anyone who might actually care. The problem is you are talking now about a field that is inundated with ideologues and believers.

  2. In addition to the items listed in this post, charter operators also invoke their “private” status to cover many other items. Depending on state laws and chartering rules, they can shelter financial records, personnel actions, and pupil admittance and dismissal decisions. They also can evade laws that traditional neighborhood schools have to follow including but not limited to: evaluation, hiring and firing (including nepotism), curriculum, etc.

    There is no single silver bullet or miracle snake oil cure.

    Keep up the great work Dr. Baker.

  3. Bruce — Although I am quite busy at the moment and although I glanced at this posting on Miracle Schools in Houston and New York, I must make a quick comment. In my analysis of what I will call high-flying schools to equate your notion of Miracle Schools (high need/high performing schools), I found only two NYC high schools in my dissertation research that fit this measure in 1999 and 2000. Here, I will only confine my quick comments to NYC high schools. And might I add, these two schools were NOT charter schools, but rather traditional public schools. Now, I realize that the number of NYC high schools have considerably grown in the past decade, and many of these high schools are charter schools. But I doubt very much that high flying schools prevail in NYC , and these do not typify charter schools. Historically, the successful schools in NYC have been the specialized schools – Science, Stuyvesant, Brooklyn Tech, LaGuardia — , which an entrance exam is required. A school’s ability to select students – as proven by these specialized schools — is a key to its success. And I would argue that selection is highly correlated with family affluence, and these two factors explain why these four schools are high-flying institutions. Admittedly, these two factors overlook the key definition requirements of a high flying school.

    In closing, has the rise of charter schools in NYC raised the mean student performance of NYC high schools when compared to the mean student performance of the rest of the state? I am confident that the answer is a resounding NO. If a reasonable statistical analysis derives that this is the case, then it stands to reason that there are probably not even a hand-full of charter schools (especially high schools) that can be considered Miracle schools — or what I call high-flying high schools in NYC.

    Next week sometime, I will read your blog on this topic more closely. But I could not resist this initial posting. In my next posting, I will share more reasoned thoughts, featuring focused analysis.

    Tyrone Bynoe

  4. Question about the success/statistics of one charter school in Brooklyn,N.Y..
    IN N.J. the Directors of the Regional Achievement Centers were recently chosen.
    The director of Region Six is David Hardy..founding principal of the Achievement First East New York Middle School.
    I believe one of his responsibilities is to help the Camden City School District.
    How did his Middle School fare in your statistics???
    I hope his experiences will help our region.

    1. Hmmmm… I’ll see what’s in the data… but my point is to some extent that it’s hard to determine much from these limited data snapshots of schools.

  5. Of Houston schools, you say, ” But, this figure and some of my previous analyses confirm that at least in Houston, charters are doing a better job of serving low income kids (than NYC charters). However, most Charter CMOs still seem to have a significant aversion to children with disabilities and in most cases, also to children who are LEP/ELL.”

    Aversion? This is an interesting choice of words. Let’s take YES Prep, as I happen to know their demographics: 86% Hispanic kids, 11% black, 1% white; 78% economically disadvantaged. Yet their LEP/ELL rate is much lower than Houston public schools, as you correctly note.

    This can’t possibly be because they have an “aversion” to Hispanic kids. Rather, it’s almost certainly because Texas doesn’t provide additional funding for LEP/ELL kids after 4th grade, and given that YES doesn’t have elementary grades, it has no incentive to classify kids that way. (Same for KIPP middle schools, by the way).

    1. Clearly, each of these cursory runs of the numbers do require additional exploration. One thing I do need to do with these data, as I’ve done with Newark data is to explore the immediate locations and neighborhood demographics. In Newark, for example, some charters are in predominantly Hispanic neighborhoods and others predominantly black neighborhoods. Those in predominantly black neighborhoods serve less than the citywide average LEP/ELL population, which is understandable. So, more local analysis would be relevant here.

      That said, as I understand it, the bilingual program weighting in the first Tier of FSP does not only apply through grade 4: http://www.lbb.state.tx.us/Public_Education/FoundationSchool_FiscalPolicy_0309.pdf (though Figure 10 does show that the largest shares of kids are in K-3) Further, even if this is the case, the incentive (or lack thereof) would apply similarly to the district and charters unless the formula differed in that regard. Parsing the LEP/ELL populations by grade level would certainly be a useful extension here, whether tied to the funding issue or not.

      Beyond that simply having high Hispanic concentration does not mean that they are not LEP/ELL averse (as I’ve found w/some NJ charters.. though NJ & TX demography are quite different). In fact, it is quite possible that charters in predominantly Hispanic neighborhoods are simply more likely to enroll larger shares of the children from families who do speak better English – perhaps being second generation, etc. This may have less to do with the charter’s own recruitment/enrollment practices and more to do with parental awareness/involvement because language barriers are less significant.
      ===
      Note that schools in the regression models of either demographics or spending are compared by “grade type” so they compare elementary to elementary, middle to middle, etc. So, the KIPP contrast is kind of fun, because I’ve split their schools by grade levels rather than averaging them. Their lower schools have higher than expected LEP/ELL (compared to elementary schools) and middle schools have lower. Of course, the data provide few insights as to why.

      1. I suppose it’s theoretically possible that YES Prep has stumbled onto a heretofore-unknown demographic: Hispanic kids who are more likely to be poor but at the same time are better at English. But in any event, using words like “aversion” seems premature, unless one also wants to accuse Houston public schools of being averse to poor kids (after all, the disparity is far greater on that measure).

      2. 1. there exists significant heterogeneity in urban Hispanic populations. And when we are using such broad brush measures of “poor” which cover 80% or so of the population, one can expect substantial heterogeneity in such factors as language proficiency (generational status, etc.). So yes, single schools serving a small share of the total population can have a very different segment of it. The grade level/range issue is the more relevant one to explore further.

        2. First, note that the low income differences are against a much higher average (average free/reduced lunch much higher than average LEP/ELL or SE), so proportionately, they aren’t necessarily larger differences. Again, comparisons by neighborhood are warranted. Second, the district must be open to all comers and continues to serve the vast majority of residents, and must take on those who exit from charters regardless of timing. So your statement about the public system is completely foolish (given our previous conversation about “limited public access” versus true public access). I guess they could manipulate housing policy (as many fine suburban districts do/have).

        Indeed, “aversion” may be premature, as I acknowledged in my previous response. But it’s certainly worth further exploration.

  6. I also have heard from a parent of a very successful charter school run by a company that has been in the business since 1996. The company is called the The National Heritage Schools and the particular school is called the Rivington Street Academy in St. Albans, Queens. I know that they don’t have spec ed students, but supposedly they have consistently great scores. Her first grader knew his times tables up to 5 and knew what nouns, pronouns, verbs, adjectives and idioms were. Supposedly they have art and music and gym for all students and a thriving afterschool program. I’d like to know about that school. Is it cherrypicking? Is it getting super funded? Is it showing a large number of students being counseled out?

  7. The REAL story on YES prep:

    In my study of charters in Texas that will be released around the 23rd of this month, I examined the 5th grade characteristics of kids that entered charters in the 6th grade. When compared to schools in the same zip code, students entering Yes Prep were less likely to be identified as ELL or as special needs when enrolled in the 5th grade. Thus, YES Prep decisions about identification of students as ELL or special needs does not play into this finding at all. Now, I don;t know of YES Prep has an aversion to ELL or special needs kids or the other way around, but YES Prep enrolls a statistically significantly lower percentage of those kids. And that covers five years of student cohorts, so it is not a one-year aberration in the data.

    Further, students entering YES Prep have statistically significantly greater 5th grade math and reading scores than students not entering YES Prep. When compared to students in the same zip code as YES Prep schools, students entering YES Prep score around 0.2 standard deviations greater than kids not entering YES Prep. And, the difference in the percentage of poor kids is only a marginal 3.8 percentage points. Further, when you look at the scores of poor kids, those entering YES Prep had 5th grade math and reading scores around 0.3 standard deviations greater than poor kids not entering YES Prep.

    Moreover, when it comes to attrition, YES Prep has statistically significantly greater attrition rates for lower performing students and statistically significantly lower attrition rates for higher performing kids.

    Finally, YES Prep graduates that enter Texas universities perform abysmally in college-level courses. Nearly 50% of the YES Prep grads had college GPAs less than 2.0.

    1. So, I believe Stuart is correct in that “YES Prep has stumbled onto a heretofore-unknown demographic: Hispanic kids who are more likely to be poor but at the same time are better at English.” Of course, that statement is only true relative to students in elementary schools in zip codes containing YES Prep schools and contiguous zip codes, but not in relation to the all students in the state. But, as Bruce has pointed out repeatedly, a state comparison is foolish.

Comments are closed.