On “Dropout Factories” & (Fraudulent) Graduation Rates in NJ


This NJ Star Ledger piece the other day reminded me of an issue I’ve been wanting to check out for some time now. I’m skeptical of graduation rates as a measure of student outcomes to begin with, because, of course, graduation can be strongly influenced by local norms and practices. As such, it’s really hard to validly compare graduation rates from one place to another or even over time, as graduation standards may change. Notably, arbitrary assignment of “passing” cut scores on high stakes assessment isn’t particularly helpful and can be quite harmful. But I digress.

What piqued my interest a while back was the apparent disconnect between cohort attrition measures from 9th to 12th grade, or 10th to 12th grade, and reported graduation rates. Indeed, these are two different things. BUT, it seems strange for example that North Star Academy in Newark could report a 100% graduation rate! and .3% dropout rate! while having approximately 50% attrition rate between grades 5 and 12! How can you lose half your kids over time and still have 100% graduation and effectively no dropouts. Of course the answer is that none of these are dropouts, but rather they are voluntary transfers (with no follow up to determine where they’ve gone or what happened to them).

In any case, it seemed at best, a bit disingenuous and at worst, outright fraudulent for North Star to present itself as near perfect, when a deeper dive into the data (something North Star’s own data driven leaders fail to ever report) suggest otherwise.

Here, I quickly explore the significance of this issue across charter and district schools statewide.

First, let’s look at 2013 graduation rates and the 2012-13 fall enrollment cohorts as seniors relative to themselves as freshman.

Slide1As the key indicates, orange dots are district cohort ratios – representing the senior class of 2013 as a percent of who they were as a freshman class of 2009-10. Green dots are graduation rates for all of the same district schools. Blue circles are reported graduation rates for charters and red squares are cohort ratios. The trendline is fit to charter and district school cohort ratios. In most cases, the cohort ratios are lower than the reported grad rates. But not by a whole lot. For TEAM Academy the two are close enough to overlap. For Central Jersey College Prep and University Academy, there is what appears to be a differential of about 5 to 7% or so.

But for North Star, the gap is huge. If we evaluate North Star on its reported graduation rate, the school looks great. Nearly perfect! But even compared to other schools statewide, on the same measure of cohort loss, North Star is no leader. Rather, it’s a laggard. (not a Paterson Sci/Tech or Hoboken laggard, but a laggard nonetheless).

Let’s take a look now at the 2012 and 2013 graduation rates, averaged, and the last 3 cohorts of sophomores to seniors, averaged just to see if the above single year estimates are anomalous.

Slide2Taking the two years of grad rates and three cohorts actually reveals that North Star a) falls even further below the trendline (worse than the average district school) AND b) still has a massive gap between reported grad rate and cohort loss. TEAM now also has a gap, but that gap is smaller than for North Star. Indeed, it is possible that TEAM is back filling enrollments (adding kids in high school to fill empty seats), but I’ll leave Ryan Hill, chief exec of TEAM to let me know if that’s the case.

Now, it’s certainly also possible that district schools in Newark are adding kids in upper grades, as they exit from North Star, or other charter or magnet schools. It is far less likely that many of these students are shifting to selective private schools (and upward, outward transfer) after the 10th grade.

Finally, let’s take a look at the gaps between reported graduation rates and cohort ratios, again using the last three sophomore to senior cohorts and last two years of graduation rates.

Slide3Consider this a test of the legitimacy of using the graduation rate to characterize the extent to which schools actually help students persist toward high school completion. The above graph suggests that North Star’s graduation rate is overstated by 10 to 15% averaged over time and that their graduation rate is far more inflated than nearly anyone else except American History High. TEAM is actually quite low in average difference between cohort attrition and reported graduation rate. The other high outlier here is Central Jersey College Prep.

Again, there can be a number of enrollment flow/transfer reasons for the gaps between cohort attrition and reported graduation rate. But, at the very least these figures should be regularly reported and used as a basis for evaluating the validity of reported graduation rates.

 

 

 

Advertisements

15 Comments

  1. “High school graduation rates at historic high” – Lyndsey Layton – Washington Post – April 28, 2014

    It would appear that Arne “Donuts” Duncan is standing in another steaming pile of fraudulent information. All that is necessary to inflate the value of a ratio is to artificially reduce the size of the denominator.

    Rob Bligh

  2. I also am fairly dubious about charters and loved the previous posts on the subject. With graduation rates however, I would have several reservations because I think that student mobility/turnover is an unappreciated problem. The old state report card mobility rate only reported students leaving after Oct 15th, missing summer transfers. Given the existence of twice yearly NJSmart snapshots, we can now get a better sense of how many students are with us the entire time over period. The two times I looked, the results were similar. Roughly 50% of our eight graders had been with us for every submission over five years. 28% joined us after the first submission, while 22% started with us but missed some submission, even if they were with us in the end. And we are in Hudson County, a poor county but not the poorest and most mobile.

    The current way original cohort – verified transfers out + new transfers-in has a lot of holes, but it is better than the old state report card rate

    1. Obviously, having even more precise mobility tracking data over time would be that much better. I acknowledge that what I’ve done here is a rather sloppy, but still worthwhile check on inconsistencies. A major problem with the current method is that it allows charters to count every kid pushed out as a transfer, creating no consequence for charters that push out very large numbers of kids they believe are unlikely to graduate.

  3. One of the schools, Weehawken High, stands out in your last graph and brings back memories of a conversation I had with a Teacher from there.

    The teacher told me that at times during the school year, immigration would swoop in and many families of school children would leave in the middle of the year!

    1. Indeed, there are likely many different stories behind which schools stand out, all begging for much greater detail in data that should be made generally available for evaluation/analysis.

  4. There are many ways to “juke” the data as most experienced H.S. Principals know. To lower your drop out rate you indicate a student as a transfer to an Alternative School Program, districts that have them use them to keep the numbers at the H.S. at a respectable level. They sacrifice the numbers at the alternative school thusly keeping the H.S. cleaner.

  5. If in fact all student data were to be kept by the DOE then students could then be tracked if transferred and listed as completing the program they were enrolled in. The data should come from the State of NJ DOE not from the school, the district or the principal. But alas that may never happen. Garbage in = garbage out.

    1. For the last couple of years, the state now does calculate the graduation rate using NJSmart submissions. In NJSmart, you either transfer the student out or mark them as graduated. Only transfers that are verified, i.e., another school in the state transfers them in, remove them from a school’s cohort. There is a whole appeals process within NJSmart. There are still weaknesses but it is better than when schools did it themselves

      1. while NJ Smart does permit the state to calculate a better grad rate than previously, it still does not resolve this other issue- that a school of choice can squeeze out large numbers of students prior to their senior year and still achieve a perfect graduation rate. Thus, looking at the gap between cohort reduction over time and graduation rates can provide useful insights.

        I guess this all speaks to the need for a persistence measure of a different sort.

      2. Schools can “squeeze them out” if they transfer to another school AND that school picks them up, otherwise they stay in the cohort and are counted as not graduating. Another problem with the gross comparison is that there are also transfers in and those students are added to your cohort based on the year they first attended 9th grade

      3. Except that not all charters will accept transfers in at all grade levels, or don’t seem to. In this case, a “transfer” out of North Star, for example, will at least for a moment be picked up by an NPS school. So, as I note in the post, the outflow of students from North Star will add to cohorts in upper grades in district schools. But, North Star’s overall enrollment is still small enough not to have substantive effects on district upper grades enrollment. So yes. North Star can squeeze them out. NPS schools must take them. And under the current system, North Star is still able to claim great success, by keeping only those with whom they will/can succeed. True, a magnet school could do the same. But we do not pretend that magnet school accepts and serves all comers.

  6. Mobility rates at public high schools are at least as high and likely higher than at charter schools.

    1. The figures above certainly show that charters fall throughout the distribution. The point is that there is variation, and for some schools the differential between cohort reduction and graduation rates is smaller and for some larger. Further, there are many types of mobility which should not all be lumped together. So, to assert a generalization like that above is pretty pointless. But, what we see here is that some schools of choice – specifically North Star, have very high rates of cohort reduction over time, yet are still able to claim near 100% “success.” This is highly deceptive, as would it similarly be with a magnet school under the same circumstances.

Comments are closed.