There’s no mystery or proprietary secret among academics or statisticians and data geeks as to how to construct simple comparisons of school demographics using available data. It’s really not that hard. It doesn’t require bold assumptions, nor does it require complex statistical models. Sometimes, all that’s needed to shed light on a situation is a simple descriptive summary of the relevant data. Below is a “how to” (albeit sketchy) with links to data for doing your own exploring of charter and traditional public school demographics, by grade level and location.
Despite the value of a simple, direct and relevant comparison using accessible data providing for easy replication, many continue to obscure charter-non-charter comparisons with convoluted presentations of less pertinent information. Matt DiCarlo recently published a very useful post (at Shanker Blog) explaining the various convoluted descriptions from Caroline Hoxby’s research on charter schools that make it difficult to discern whether the charter schools in her comparisons really had comparable student populations to nearby, same grade level traditional public schools.
As I’ve discussed in the past, charter advocate researchers tend to avoid these basic comparisons, instead showing that students selected through the lottery were comparable to those not selected but who still entered the lottery (excluding all those who didn’t enter the lottery). While this information is relevant to the research question at hand (comparing effectiveness among lottery winners and losers), it skips over entirely another potentially relevant tidbit – whether, on average, the charter students are comparable to students in surrounding schools.
Alternatively, charter advocate researchers will compare charter characteristics to district wide averages, or whatever comparison sheds the most favorable light. For example, Matt DiCarlo explains of Caroline Hoxby’s NYC charter research that:
“The authors compare the racial composition of charter students to that of students throughout the whole city – not to that of students in the neighborhoods where the charters are located, which is the appropriate comparison (one that is made in neither the summary nor the body of the report). For example, NYC charter schools are largely concentrated in Harlem, central Brooklyn and the South Bronx, where regular public schools are predominantly non-white and non-Asian (just like the charters).”
The better approach is, of course, to compare against the, well, most comparable schools – or those serving similar grade levels in the same general proximity – or even to be able to identify each individual school (such that one can determine comparable grade levels) among districts in similar locations.
Here’s my general guide to making your own comparisons using a readily available data source.
Go to: www.nces.ed.gov/ccd
Use the Build a Table function: http://nces.ed.gov/ccd/bat/
- Select as many years of data you want/need (first screen toggle)
- Select the “school” as your unit of analysis for your data (first screen, drop down)
- Select “contact information” from the drop down menu on next screen
- Select location zip code
- Select location city
- Select “classification information” from the drop down menu
- Select the “charter” indicator
- Select the “magnet” indicator (in case you want to include/exclude these)
- Select “total enrollment” from the drop down menu
- Select total enrollment
- Select “students in special programs” from the drop down menu
- Select students qualifying for free lunch
- Select students qualifying for reduced price lunch
- Select “Grade Span Information” from the drop down menu
- Select “school level” identifier
- Select “High Grade” and “Low Grade” indicators if you want more flexibility in comparing “like” schools
- Pick the state or states you want (you can’t use this tool to pull all schools nationally because the data set will be too large for this tool. Complete data are downloadable at: http://nces.ed.gov/ccd/pubschuniv.asp )
Calculate Percent Free Lunch and Percent Free & Reduced Lunch (divide groups by total enrollment)!
Play…
Here are some examples…
First, here are a handful of New Jersey Charter Schools compared to other schools (comparable and not) in their same zip code.
In this first figure, from a Newark, NJ zip code, we can see quite plainly and obviously that the shares of children qualifying for free lunch in Robert Treat Academy are much lower than all other surrounding schools, including the high school in the zip code (Barringer), where high schools typically have lower rates of students qualifying (or filing relevant forms) for free lunch.
Here are a few more.
Other “high flying” charters in Newark including North Star Academy, Gray Charter School and Greater Newark Academy, in a zip code with fewer traditional public schools, tend to have poverty concentrations more similar to specialized/magnet schools than to neighborhood schools in Newark. Other charter schools like Maria Varisco Rogers and Adelaide Sanford have populations more comparable to traditional neighborhood schools. But, we don’t tend to hear as much about these schools – or their great academic successes.
Things aren’t too different over in Jersey City. In the area (zip code) around Learning Community Charter School, other charters and neighborhood schools have much higher rates of children qualifying for free lunch than LCCS. Only the special Explore 2000 school has a lower rate.
Ethical Community Charter also stands out like a sore thumb when compared to all other schools in the same zip code, including those serving upper grades which typically have lower rates.
But what about those NYC KIPP schools? How about some KIPP BY ZIP?
So much has been made of the successes of KIPP middle schools, coupled with much contentious debate over whether KIPP schools really serve representative populations and/or whether they are advantaged by selective attrition. I included some links to relevant studies on those points here. But even those studies, which make many relevant and interesting comparisons, don’t give the simple demographic comparison to other middle schools in the same neighborhood. So here it is:
School Finance 101 
This is really interesting. Question: how do zip codes align with districts or school zones within districts. Also, many middle schools in NYC are not representative of their local neighborhood; is there any way to screen for the composition or selectivity of the district schools near the charter schools?
Zip codes are not typically aligned with any type of attendance boundary, especially within district, school attendance boundaries. But zip codes represent smaller areas likely to be more representative of neighborhoods & collections of neighborhoods. Ideally, one would have access to child level data including neighborhood of residence (specific address), and could use those data to map the implicit attendance zone/radius around each school by the actual attendance patterns. That said, even in choice models, it does tend to be a radius of attendance with greatest density nearest the school, so using localized comparisons is usually reasonable at the elementary and middle school level. Clearly, zip codes and collections of adjacent zip codes provide more relevant comparisons than city wide comparisons when a city as large as NYC is involved. New York City is more complicated in this regard.
Thanks for this. I’m going to send a link to my PTA-mom friend in Jersey City who’s been trying to make apples-to-apples comparisons.
It’s sad that zip code is even used as the size of a zip code area represents the population density to postal worker delivering the mail ratio. Census based average income boundaries would be so much nicer. In addition, many school boundaries are gerrymandered to please parents.
I will likely take a stab at doing these based on a radius around the latitude and longitude (available in the NCES data set) location of each charter. I just wanted to do the zip thing as an easy demonstration that many could replicate on their own. Zip Codes do work pretty well though for simple illustrative purposes, and one can pull together clusters of adjacent zip codes. The point here is that localized comparisons within large cities are far more relevant than city wide comparisons.
I would love to see the comparisons within the actual city “districts”. There is a huge difference between Greenville, Down Town, and the Heights in Jersey City, and many of those VERY different neighborhoods are in the same zip code. In fact a Private school in Greenville is becoming a charter school and siphoning kids from the neighborhood school. Needless to say people are happy about the supposed change in law that says if the charter school kicks the kid out before Columbus Day that the charter NO LONGER gets to keep the money. We’ll see how that works.
If I find the time, I’ll try to construct some “neighborhood” files based on clusters of adjacent zip codes. I’d be happy to take a run at some comparisons based on zip code clusters you might be interested in looking at. Fire away. Which JC zips would you like to group together for comparison? I can actually take it down to Zip + 4 for greater precision. But, in any given Zip +4 there aren’t many schools. The better approach would be to use the latitudes and longitudes and come up with some space that best represents the neighborhood around a given charter. I just haven’t had the time to go to that level with these data yet. Buried under several large projects this summer.
How do comparisons of charter school composition with nearby public schools help us improve charter school policy? Why are they more relevant than than citywide comparisons?
The point in this post is that many other times on the blog I’ve done the comparisons against the whole city, but in many cases, the city is a collection of diverse neighborhoods. New York is obviously very many different neighborhoods, and Newark includes at least a handful of vastly different sections, racially and socioeconomically. Comparing against citywide averages really hides these neighborhood differences. Zip codes aren’t great, but were available for simple illustrations.
On average, whether choice schools or neighborhood schools, schools tend to draw locally – from nearer neighborhoods – more so than form further ones (though you might be able to provide additional information in this regard?).
In New Jersey and at times in NY the policy conversation has carried on at a very crude level – arguing that average performance of charters is simply better than average performance of public schools in the same neighborhood, and that because they serve kids from the neighborhood they therefore must be representative. I know it sounds silly that anyone would frame it that way, but that has been the gist of the policy conversation, more so in NJ. If most attendance is more local than not, in a diverse city it matters to compare charter enrollments with other schools drawing from nearby populations a) to determine the extent (or not) of local cream skimming) and b) the potential disadvantage created for the traditional public school serving an increasingly poor population (even more so with special ed and ELL) as the charters cream skim.
I’ve also pointed out previously that if we did a good lottery study of NJ charters, we might still find that charters like Treat or North Star outperform other schools (attended by lottery losers). But, given how different their populations are, the difference might be predominantly peer effect not school quality effect, suggesting severe limitations for scaling up. And the peer effect benefits for these charters leave behind a negative peer effect for publics drawing from the same area. Clearly, this Zip Code level stuff in the CCD is way too crude to get at the implicit attendance zones for these choice schools. Individual address files are obviously necessary.
One of my other big points here is that I really don’t mind so much that Treat and North Star are what they are, and in my own opinion do well doing what they do. But, they don’t serve comparable populations. In effect, they are more like academic magnet schools. That’s fine, as long as we call them what they are, and don’t argue that they are serving kids comparable to those in the neighborhood and doing better with them. You found positive effects of magnet schools in Hartford. Question: I can’t recall if you were able to tease out peer effects? But that wouldn’t necessarily be the point anyway. I believe it is assumed that much of the effect is created by peer effect. And that’s fine if we’re open and honest about it, and acknowledge that this limits scalability.
So I see three points. Let me consider whether I agree with them.
1. Comparing achievement of charters with nearby publics does not tell us whether charters do better or worse than publics with the same population of students. I agree, and if people are making that mistake, it is worth pointing it out.
2. Comparing charter composition to those in nearby charter schools tells us whether charter schools are cream-skimming. Maybe. As you point out, it depends on how well neighborhoods (here, zip codes) line up with school catchment areas. I would recommend caution on this point. If charters are able to achieve lower concentrations of poverty because they attract non-poor students from outside the neighborhood this might be a good thing. In addition, as you also point out, cream-skimming may not be all bad. It might be good for the kids that are skimmed, even if it is bad for those left behind (and we don’t really know if it is bad for the kids left behind).
3. If charter and magnet schools are able to provide better learning environments because they assemble motivated students with supportive home environments, they might not be scalable. I agree going to scale would be difficult. So the key question is not just whether charters and magnet schools do better, but when they do do better, we need to ask why and then think about how we can design policies to replicate it. We also need to think about whether kids doing better in charters and magnets comes at the expense of kids in other schools. I don’t know the answers to any of these questions.
So it seems I more or less agree. My one caution is that saying comparisons of charter school composition with that of nearby publics can provide useful information, does not imply that we should demand that each charter school match the composition of surrounding public schools as a policy goal (a move the NYS Legislature recently made with impetus from the teacher’s union).
” If charters are able to achieve lower concentrations of poverty because they attract non-poor students from outside the neighborhood this might be a good thing. ”
Indeed, isn’t that the objective of economic integration?
I believe that’s what bob is suggesting. It would be a good thing. The extent to which charters accomplish this goal, however is questionable. I believe bob was speaking based on his experience evaluating Hartford magnet schools which do accomplish this to some extent, if I recall.