After having a series of conversations with Star Ledger reporter Julie O’Connor about her desire to write a cover story about how TEAM Academy is producing miracles in Newark, I wrote this post:
The reason for this post is explained in this paragraph:
Well, one reason I’m going there is that I’m sick of getting e-mail and phone inquiry after inquiry about the same charter schools – and only charter schools – asking how/why are they creating miracle outcomes. I try to explain that there may be more to the story. The reporter then says that the charter school’s data person says I’m wrong – validating their miracle outcomes (despite their own data not being publicly available/replicable, etc. and often with reference to awesome outcomes reported in popularly cited studies of totally different charter schools).
For a while after writing this, I figured that the NJ Star Ledger reporter who was so insistent on writing her rah rah TEAM article had simply given up. But alas no. The puff piece finally arrived today: http://www.nj.com/opinion/index.ssf/2015/05/beating_newarks_odds_kipp_charter_network_is_poise.html#incart_river
Now, it’s written as an editorial, so I guess that means it’s okay to make stuff up, ignore lots of stuff, and just generally roll with a combination of propaganda provided to you by the school and your own personal predisposition.
What’s so disturbing about this all is that the title of the editorial itself is directly refuted by the statewide analysis I provided. That TEAM relatively marginally beats expectations, and in fact, several Newark Public schools and a few other charter schools in Newark “beat the odds” so to speak, by much more. AND THE AUTHOR OF THE EDITORIAL WAS FULLY AWARE OF THIS.
I refused to call the reporter in part because I wanted there to be a full, complete transcript of our e-mail conversations. I’m sick of banging my head against this wall.
Below is a transcript of the conversation that started with an inquiry to Diane Ravitch from Julie O’Connor. Others were included on the e-mail chain and jump in at various points.
I’m on the editorial board at The Star-Ledger in New Jersey, and I’m working on a cover story for our Perspective section about the KIPP schools in our state. The college attendance stats of KIPP seniors in Newark seem pretty impressive, and I was wondering if you have the same reaction, and what you think of KIPP’s forays into Camden.
Would really appreciate it if you could give me a call at . Would like to discuss KIPP in the context of your criticisms of the broader charter school movement, and whether or not you think it is an exception.
I suggest you talk to Mark Weber and Bruce Baker at Rutgers, who have studied charters in NJ. I lean on their research. The question is not whether one chain can produce successful graduates, but whether charters in general are helping the most vulnerable schools, whether they are reducing the funding and capacity of public schools, and whether their success-when it exists–is the result of selection and attrition.
Ok, thanks for your prompt reply.
Prof. Baker emailed me his report on free/reduced lunch and the TEAM schools, but I have been unable to reach him on the phone to discuss KIPP or my follow up questions.
Basically, I am looking for a reaction to two claims from KIPP that seem impressive: The college attendance rates (last year, 95 percent of KIPP seniors went to college, 89% to a 4-year, 6 percent to a 2-year), and the fact that KIPP kids in elementary and high school equal or outperform the average for the state of NJ (some years they do in middle school, too, though this year they didn’t).
KIPP kids are 87% free/reduced lunch and the state is in the 30s. I understand that Baker and others are skeptical about comparing KIPP kids to their peers in the Newark district. But what about comparing them to the state average? And what about their college attendance rates?
I would like to discuss the criticisms of the charter school movement and whether you view KIPP as an exception, or more of the same. Prof. Baker, can you please give me a call as soon as you get a chance? . We are hoping to run the story in the next week or so.
Baker to Reporter
My point is, and shall continue to be that news stories on education should NOT be driven by some PR prompt from specific schools touting their “successes” through anecdotes. Thus, my only reaction is the reaction I posted previously about school performance, given analyses across all schools, using comparable, publicly available data:
The bottom line is that KIPP schools performance on comparable measures of student growth, controlling for demography, resources, etc., are relatively average (marginally above average). Many district schools, including ones in Newark, far outperform them.
Reporter to Baker
Ok. Even if KIPP students aren’t representative of their district, isn’t it still impressive that they are beating the state average, given that their student population is significantly poorer?
KIPP says 93 percent of their students stay with them (7 percent leave their schools each year for any reason).
If what this tells us is that KIPP students have high scores and go to college, how do they fit into criticisms of the larger charter school movement? And what do you think of KIPP’s expansion into Camden?
Prof. Baker, read your blog post and would like to discuss. I am not sure how you are measuring growth in these ranked schools. Are you skeptical about the accuracy of the college attendance rates and performance numbers reported by KIPP? If so, why? Please give me a call. 
Baker to Reporter
Not without running a model of demographics against the same outcome measures across all schools, to see how/whether they truly deviate, statistically, from expectations. Anecdotes of this type are unhelpful for understanding what’s “impressive” statistically or not.
For measuring growth, I’m using the state’s own reported school Median Growth Percentile – for 2012, 2013 and 2014.
Skeptical or not, context is what’s needed for them to really mean anything. The context of all other schools, and their demographics, to evaluate statistically whether the KIPP schools actually deviate from what would otherwise be expected (given enough schools to estimate a model of expectations).
Reporter to Baker
Ok. Is the state average not considered a good measure of how schools are doing?
Is your central point in creating your own measurement for whether schools deviate from expectations that KIPP schools have more resources and classroom time and better class sizes, and that’s why their students are doing so well?
Are you trying to account for those factors in your outcome measure, since you might not find such conditions in traditional district schools? That seems to be your argument in this blog post:
Trying to understand your general view of KIPP’s performance.
Baker to Reporter (w/head banging against desk)
No. State average is NOT a useful comparison. Given the number of things that vary across schools, one needs to look at any given school in the context of all schools, with all available measures. Not just compare one school to the state average and say, for example, “it’s got higher poverty, and higher outcomes than the state average.” That comparison misses a lot of other factors that may vary across schools. One needs to see how those factors affect the outcome measure across schools and then compare against the overall pattern.
Second – I’m not “creating” my own measurement. I’m doing what I describe above. Taking the state’s measures, and making comparisons among “otherwise similar” schools along the trend of schools, given their various attributes. That is, how much higher, or lower than expected, does a school score (on growth) given all of those factors that vary.
Now, I also use the state’s growth measure, because, for all its shortcomings, it is actually the best available New Jersey measure of what a school might be contributing to student outcomes (rather than what kids come in with, or who leaves and when). But that measure too is ONLY useful if you control for/account for the various factors. Quite simply, this is how credible analysis of this type is done, knowing full well that even this approach can’t capture some factors that affect outcomes that really aren’t about how good/bad a school is.
Their performance tends to be marginally above average, to about average, considering all schools including district schools. For that matter, several Newark district schools have higher performance. Discovery Charter school is the standout among charters. North Star seems to do well, but I believe that the model isn’t really capturing the effect of their substantially greater attrition, or different student population. But who knows. But then again, Robert Treat has very different student population and tends to show very weak gains with adjustment for the included factors.
Reporter (who clearly never bothered to read the original post)
What factors that vary are you trying to account for? It is things like resources, classroom time and class sizes?
Baker to Reporter (direct response to ignorant question)
They are all listed in the blog post!
Outcome is Growth
- prior average scale score level
- % free lunch
- % disability (because I cant’ break out by severity, charters like TEAM actually get an advantage here)
- % Ell
- total staffing expense per pupil
- school grade range served
- school size
More Exasperated Baker to Reporter
Schools in Newark: https://schoolfinance101.files.wordpress.com/2015/01/slide18.jpg
Charter Schools Statewide: https://schoolfinance101.files.wordpress.com/2015/01/slide24.jpg
So again, I ask, why do you feel the necessity to write a story on KIPP schools? And why the apparent obsession on trying to find a miracle in KIPP? How do these supposed miracles (that generally aren’t) come across your desk?
An objective statistical run of all schools in the state, using the state’s own best available measure as the outcome, finds TEAM in Newark to be a decent – relatively above average – school, but no miracle. There are no miracles in this complex endeavor. That’s fine. They do a pretty good job, and seem to do a better job of serving a more representative student population than some others (see also: https://schoolfinance101.wordpress.com/2013/11/27/where-are-the-most-economically-segregated-charter-schools-why-does-it-matter/)
I’m not trying to rain on their parade. I’m just pointing out that if we take all of the data from schools around the state and try to figure out who’s actually “doing better than expected” given who they serve and the resources they have, we don’t identify KIPP as the standout.
Weber to Reporter
Julie, I am going to encourage you to read Bruce’s entire post, as it is far more sophisticated and comprehensive than what I am going to include here.
That said, let me put this in very simple — admittedly, TOO simple — terms:
This is a very quick and very dirty scatterplot that shows the average scores on the NJASK Grade 8 English Language Arts (ELA) exam from last year for every school in the state. I’ve highlighted TEAM on this graph.
The NJASK score is on the vertical or y-axis. On the horizontal or x-axis is the percentage of students who qualify for free or reduced price lunch, a proxy measure for student economic disadvantage (a student’s family has to be at 185% or below the poverty line to qualify for FRPL).
The first and most obvious thing to notice is the relationship between how many FRPL kids a school has and its average test score. Clearly, when FRPL goes up, test scores go down. 70% of the variation in these scores can be statistically explained by the percentage of FRPL kids at the school.
We all know this. Poverty matters.
The green line through the middle is called a regression line: it’s a kinda-sorta “average” that predicts how well a school will do given its FRPL percentage. If you’re above the line, you’re doing better than prediction; if you’re below the line, you’re doing worse.
TEAM is above the line – hooray for them. But how many other schools do you see across the state that are at least as far above the line as TEAM? How many are way, way further above that line compared to TEAM?
Again: what Bruce did in his post was far more sophisticated than this, because he’s using a statistical model to account for other things that will affect student outcomes, like percentages of special education kids and how much a school spends per pupil on staff (yes, money does matter). He’s also judging outcomes on SGPs, which is arguably a better measure of a school effectiveness.
I’m boiling this down, however, to reinforce his point: yes, TEAM is a better-than-average school. Again, good for them… but why all the outsized attention? Why are you writing a story about them and not the many, many other schools that “beat prediction” much better than TEAM? How many district schools could be considered “miracles” relative to TEAM that get ignored by the op-ed pages of your newspaper?
Julie, you and I both know I have been the Star-Ledger Editorial Page’s harshest critic on education. I’ve admitted before that sometimes I have gone too far… but can you understand my frustration? Can you understand how unfair it appears to those of us who have taken the time to study Bruce’s work that TEAM gets all the accolades while many schools that — by TEAM’s own standards — are doing a BETTER job than they are, yet continue to be ignored?
I am asking you to listen to Bruce carefully and take the time to understand what he is saying. This stuff matters. You control arguably the most important space for punditry in the state. You owe it to your readers to get this stuff right.
If I can help further, let me know.
Reporter (still not bothering to read, and returning to anecdotes provided by school)
What about the 95 percent of KIPP seniors that went to college last year? That seems impressive to me.
Also, when you say comparing KIPP to the state average doesn’t mean anything without “running a model of demographics against the same outcome measures across all schools, to see how/whether they truly deviate, statistically, from expectations” — isn’t that what the Mathematica study does? Control for any differences in student population?
Baker (even more exasperated) to Reporter
Why don’t you write it that way then – that it seems impressive to you. I’m not going there, with your representation of data, passed along to you most likely by the school, without opportunity run appropriate models on the data. And I don’t have time to be doing that right now, or quibbling with you over your strange incessant desire to write a story on how awesome you think these schools are, without ever bothering to look at the schools in the context of all schools, where many others may, in fact be even more impressive.
And are you speaking of some Mathematica study of TEAM Academy specifically, and their graduation and college matriculation rates? Or Mathematica studies of KIPP schools generally/nationally ? [I believe only the latter exists –http://www.mathematica-mpr.com/~/media/publications/PDFs/education/kipp_middle.pdf] Yes, the network’s results are solid. Not miraculous. But solid. Driven in part, perhaps by selection issues (see methods critiques below), and in part by resources. KIPP schools in many contexts substantially outspend their “competition” offering higher salaries, much smaller classes, longer days/years, etc. Certainly won’t deny that those types of resources matter.
Comments on related methods here: https://schoolfinance101.wordpress.com/2012/12/20/thoughts-on-randomized-vs-randomized-charter-school-studies/
There are indeed limitations these methods.
Some information here on where TEAM fits on resource/demographics, etc in Newark: https://njedpolicy.wordpress.com/2015/01/13/research-note-resource-equity-student-sorting-across-newark-district-charter-schools/
Weber to Reporter
Related to the issue of resources:
Find attached the 2012 tax forms for TEAM, Friends of TEAM, and KIPP. You can access these easily at guidestar.org.
You will notice on page 42 of the KIPP 990 that TEAM received $1,053,147 in direct support from KIPP. This likely does not include all sorts of administrative, logistical, marketing, lobbying, etc. activity KIPP undertakes on behalf of TEAM.
On page 21 of the Friends of TEAM 990, you’ll find a $1,005,332 grant to TEAM. On page 9, you’ll see the group took a rental income loss of $1,813,501, likely to the school’s benefit (were I you, I’d certainly ask them about this).
In 2011-12, TEAM enrolled 1,504.5 students. If you take the grants from KIPP and FOT together, that comes to $1,368 additional expenditures per child, not including the rental loss that FOT took. So far as I know, this extra funding is not reported in the NJ Taxpayers Guide to Education Spending.
Let me be clear: it is, in my opinion (an opinion backed up by a substantial and growing body of research) that spending this extra money on behalf of these students will help their academic growth. This is a good thing.
But it is exactly the sort of issue that is not addressed by the Mathematica report, nor by any number of other “studies” that purport to show the superiority of KIPP’s methods by holding all things constant.
So how does TEAM spend all this extra money? Well, here’s one way:
At all stages of a teachers career, TEAM pays a higher salary, even when adjusted for experience, than NPS (and way more than Newark’s “local” charters). When you pay more and offer better working conditions, you can attract people who are willing to work longer hours (to a point).
But they manage to keep salary costs low by also doing this:
Notice the high number of teachers with only one year of experience at TEAM? Notice how they barely have any teachers with more than 15 years of experience? That’s when the NPS salary guide gives veteran teachers a big boost.
Is this a smart strategy? Absolutely. Is it sustainable? I say almost certainly not. Does TEAM really think they can keep recycling their staff AND expand the number of students enrolled? Are there really that many young people out there willing to make teaching at TEAM a temporary career? And is that really good for the city and its students?
As Bruce says: TEAM does a good job. They are, by the numbers, a good school. But I would argue KIPP’s methods are not replicable at a large scale. In fact, THEY’D probably agree with me, because they have said over and over again that they are not interested in taking over an entire district.
Julie, if you are willing to dig into this and go behind the talking points the KIPP publicity machine feeds the press, I think you will find TEAM’s “success” raises more questions than it answers:
– If more money is good for charter schools, why isn’t it good for pubic schools?
– Is it good for the teaching profession to encourage the growth of schools that appear to run on a policy of churning much of their staff?
– When we get past the issues of different student populations, attrition, extra resources, hiring practices, test prep, etc., what, exactly, is so special about KIPP/TEAM?