How NOT to fix the New Jersey Achievement Gap

Late yesterday, the New Jersey Department of Education Released its long awaited report on the state school finance formula. For a little context, the formula was adopted in 2008 and upheld by the court as meeting the state constitutional standard for providing a thorough and efficient system of public schooling. But, court acceptance of the plan came with a requirement of a review of the formula after three years of implementation. After a change in administration, with additional legal battles over cuts in aid in the interim, we now have that report.  The idea was that the report would suggest any adjustments that may need to be made to the formula to make the distributions of aid across districts more appropriate/more adequate (more constitutional?). I laid out my series of proposed minor adjustments in a previous post.

Reduced to its simplest form, the current report argues that New Jersey’s biggest problem in public education is its achievement gap – the gap between poor and minority students and between non-poor and non-minority students.  And the obvious proposed fix? To reduce funding to high poverty, predominantly minority school districts and increase funding to less poor districts with fewer minorities.

Why? Because money and class size simply don’t matter. Instead, teacher quality and strategies like those  used in Harlem Childrens’ Zone do!

Here’s my quick, day-after, critique:

The Obvious Problem? New Jersey’s Huge & Unchanging Achievement Gap

The front end of the report provides lots of nifty graphs based on cohort proficiency rates on tests which change substantially in some years. The graphs are neatly laid out to validate the argument that New Jersey’s achievement gap is large and hasn’t changed much.  First, on the point of the largeness of the gap, in national context. I’ve explained here how the NJ poor-non-poor gap is actually relatively average nationally. That’s not to say that it’s acceptable, we ought to work on this, by whatever reasonable means we can.

Thankfully (so I don’t have to revisit all of the problems here), the remainder of the achievement gap analysis presented by NJDOE is thoroughly critiqued in a recent post by Matt Di   Carlo at Shanker Blog. DiCarlo summarizes some of the NJ achievement gap and trend data to point out:

The results for eighth grade math and fourth grade reading are more noteworthy – on both tests, eligible students in NJ scored 12 points higher in 2011 than in 2005, while the 2011 cohorts of non-eligible students were higher by roughly similar margins.

In other words, achievement gaps in NJ didn’t narrow during these years because both the eligible and non-eligible cohorts scored higher in 2011 versus 2005. Viewed in isolation, the persistence of the resulting gaps might seem like a policy failure. But, while nobody can be satisfied with these differences and addressing them must be a focus going forward, the stability of the gaps actually masks notable success among both groups of students (at least to the degree that these changes reflect “real” progress rather than compositional changes).

Revelation? Gaps are a function of the height of the highs as much as the depth of the lows. If both get better, gaps don’t close as much. Gaps are still a problem, and must be addressed even if the highs get higher, because opportunity for access to college and on the labor market is relative. But, the framing of the NJ achievement gap by NJDOE is unhelpful in this regard, and the proposed solutions harmful. How does it make sense then, to provide greater increases in state aid to those students in districts at the highs and less to the lows?

Supporting Claims for Solutions?

Of course, to support the eventual pre-determined (utterly absurd) conclusion that the way to close this achievement gap is to cut aid to the poor and give it to the less poor requires that the report validate that money really has nothing to do with it. That, arguably, all of that money and increased staffing actually made things worse. Further, that cutting money from poor districts is what will make them better. I guess it also then stands to reason that giving larger aid increases to less poor districts might also make them worse, and viola – the achievement gap shrinks!

  • Claim 1: Money Has Nothing to do with It

The claims that money doesn’t matter are built on some graphs which could easily make my list of dumbest graphs (or at least most pointless, deceptive, meaningless ones). Here’s one which is intended to convince the reader that all of that money sent to Abbott districts was for naught:

The report uses the graph to conclude:

While the above analysis is not sufficient to say whether new spending has had a positive impact on student achievement, it makes clear that financial resources are not the only – and perhaps not even the most important – driver of achievement.

If the graph isn’t sufficient to make this point, then why use the graph to try to make this point? Clearly, looking only at two variables – percent change in revenue and percent change in proficiency rates – is not even sufficient to make the softened claim “perhaps not even the most important” factor in improving student achievement.  These assertions can’t be supported in any way by this graph.

But even more suspect is the assertion embedded in the policy recommendations  that therefore, cutting aid from high poverty districts will cause no harm.

Better research on whether and to what extent school finance reforms improve student outcomes &/or equity of outcomes shows that in fact, school finance reforms can and do improve both the level and distribution of student outcomes:

Higher quality research, in contrast, shows that states that implemented significant reforms to the level and/or distribution of funding tend to have significant gains in student outcomes.

Further, research on the broader question (based on real analysis) of whether and how class size and money matter indicates that, in simple terms, money does matter, and that things that cost money, like class size reduction and improving teacher quality (which does cost money) matter:

Perhaps most importantly, even the research that has cast doubt on the strength of the positive influence of money on student outcomes has never validated that cuts to funding are not harmful and may be helpful. This is an absurd and unfounded claim.

Richard Murnane of Harvard said it well enough back in the early 1990s:

“In my view, it is simply indefensible to use the results of quantitative studies of the relationship between school resources and student achievement as a basis for concluding that additional funds cannot help public school districts. Equally disturbing is the claim that the removal of funds… typically does no harm.” (p. 457)

Murnane, R. (1991) Interpreting the Evidence on Does Money Matter? Harvard Journal of Legislation. 28 p. 457-464

Though not directly stated in the NJDOE report, it is implicit in the recommendations.

  • Claim 2: Teacher Quality & Harlem Childrens’ Zone-Style Strategies Can Close the Gap

Deeply embedded in the NJDOE report, making the transition from claims of dire achievement gaps toward how to fix them, is a discussion of how the obvious solutions based on current research must have to do with improving teacher quality and doing stuff like Harlem Childrens’ Zone does.  The NJDOE report includes two particularly bold statements that these two strategies alone – but certainly not money – can close the black-white achievement gap:

Having a highly effective teacher for three to five years can erase the deficits that the typical disadvantaged student brings to school.xxiii

Evidence from the Harlem Children’s Zone provides a similar demonstration of the power of schools to close the black-white achievement gap existing in New York.xxiv

Needless to say, these interpretations of the existing research are a massive unwarranted stretch. Matt Di    Carlo addresses the issue of  how many teachers does it take to close the achievement gap?

Even then, the implicit assertion of the report in general, that money has nothing to do with teacher quality or the distribution of teacher quality, is ridiculous. As I explain here:

A substantial body of literature has accumulated to validate the conclusion that both
teachers’ overall wages and relative wages affect the quality of those who choose to enter the teaching profession, and whether they stay once they get in. For example, Murnane and Olson (1989) found that salaries affect the decision to enter teaching and the duration of the teaching career, while Figlio (1997, 2002) and Ferguson (1991) concluded that higher salaries are associated with more qualified teachers.

And further, on the flip side, cuts to funding and severe constraints on spending growth can reduce teacher quality:

Research on the flip side of this issue – evaluating spending constraints or reductions – reveals the potential harm to teaching quality that flows from leveling down or reducing spending. For example, David Figlio and Kim Rueben (2001) note that, “Using data from the National Center for Education Statistics we find that tax limits systematically reduce the average quality of education majors, as well as new public school teachers in states that have passed these limits.”

And, if we are interested in achievement gaps, and better distributing the quality of teachers across richer and poorer districts and children:

Salaries also play a potentially important role in improving the equity of student outcomes. While several studies show that higher salaries relative to labor market norms can draw higher quality candidates into teaching, the evidence also indicates that relative teacher salaries across schools and districts may influence the distribution of teaching quality. For example, Ondrich, Pas and Yinger (2008) “find that teachers in districts with higher salaries relative to non-teaching salaries in the same county are less likely to leave teaching and that a teacher is less likely to change districts when he or she teaches in a district near the top of the teacher salary distribution in that county.”

But even more strikingly, these interpretations ignore entirely that what Harlem Childrens Zone does, above and beyond anything else is to spend a ton of money (raising as much as $60,000 per pupil in private giving in some years, for additional information, see this post) and spend much of that money on providing smaller class sizes than surrounding NYC district schools.  So, in effect, what Harlem Childrens Zone shows us (in its best light) is that we can make modest progress toward closing achievement  gaps by leveraging substantial additional financial resources to provide comprehensive wrap-around community resources coupled with small class sizes.

The Proposal: Cut Aid to the Poor and Give More to the Non-Poor (& Less Poor)

After the rather predictable preamble about New Jersey’s achievement gap, coupled with classic claims that money clearly isn’t the answer, and things that actually cost money, but we’ll pretend don’t really cost money are the answer, the obvious recommendations for changes to the school finance formula are to reduce aid to the poor and give it to the less poor.

Here are the distributions of the percent change in state aid for 2012-13 across K-12 districts and the per pupil (preliminary estimates in need of updated enrollment figures) by districts arranged from lower to higher concentrations of low income children:

K-12 Unified Districts Only

K-12 Unified Districts Only

The report argues specifically that the adjustments in the aid formula for low income children should be reduced. That they should be reduced because they were increased without basis, over original recommendations provided to the state board back in 2003 (but hidden until 2006). In short, that those low income kids really don’t need that much and will be better off without it.

I critique those original recommendations in this report. Essentially, the argument is that there is simply no basis for providing as much as an additional 57% per low income child in high poverty concentration districts, therefore we should reduce it. The icing in the cake in this argument is a table in which the report points out that Texas, Vermont and Maine provide less than this. How in the heck they chose Texas, Vermont and Maine is beyond me. These states are at least a little different from NJ… and… from each other.

Beyond that, it should go without saying that the decisions of policymakers in three completely different states that aren’t New Jersey really have little or nothing to do with the cost of providing equal educational opportunity to low income kids in New Jersey.  Are we going to base all of our policies on Vermont… and Texas simultaneously? That would be a real trick? Consider the possibilities?

As my reported linked above points out, the weights in the original analysis were too low, and were thus adjusted upwards, though not necessarily far enough? On what basis? Well, the actual research on the costs of providing equal educational opportunities for low income children points to weights nearer to double, not 40% or 50% higher than average.  Here’s the most directly relevant article, from the Economics of Education Review, and here’s a link to a National Research Council report on the subject.

In a further effort to reduce aid to poorer districts (in a way that will have multiplicative effects throughout the formula) NJDOE proposes to base the allocation of aid on Average Daily Attendance. This is actually a classic, well understood Trick of the Trade for shifting aid away from poorer districts which for a variety of reasons outside their control have lower attendance rates. Way back when I started this blog, one of the topics I wrote about was these seemingly innocuous tricks (a subject of my research).  While other states do continue to use these policies, since their effects are well understood, to recommend such a change is shameless.

But even setting aside the empirical evidence on “costs,” how can it possibly make sense that achievement gaps between richer and poorer districts will be moderated by taking money from poorer districts and redistributing it back to less poor ones?

That’s the report in its essence.

We’ve got big achievement gaps.

Money doesn’t matter – in fact – it must be making things worse not better.

Therefore, to close the gaps, we need to give less of that harmful money to the poor, and more to the non-poor.

Go figure?

13 thoughts on “How NOT to fix the New Jersey Achievement Gap

  1. The logic of the proposals is clearly illustrated in quotes provided this article:

    The Christie administration hopes to use these changes to shrink the “achievement gap” between poor and wealthy students.

    “While money certainly matters, there is no evidence money alone will close the achievement gap,” said acting Education Commissioner Christopher Cerf.

    And here:

    “These recommendations, along with the new policy aspects, will serve disadvantaged children in the state better than they’ve ever been served before,” Cerf said. “We are all living in a universe where we’ve been led to believe that you equate effectiveness with dollars. I don’t think the evidence supports that at all.”

  2. Bruce, great analysis per usual. My unkind political analysis is that the 2013 election is coming and Christie has to throw some REAL political bones to the white suburbs to be re-elected. And what better way to do this via school finance? That he also gets to politically punch the Abbott districts, whom white suburbanites view with great disdain anyway, is gravy. Win-win for him.

    Too bad about the kids, though. (And I HATE this cynical calculation to all of this). But clearly, the latest proposal has nothing to do whatsoever about ameliorating the achievement gaps.

    At least you’ve “de-pantsed” the Christie administration’s supposed justifications. So, “thank you.”

    1. Here are my thoughts on this.

      It’s clear that there will always be pressures to redistribute aid from one group to another. That’s just how these aid formulas work. When SFRA was adopted under the Corzine administration it was actually designed to spread aid more widely across districts and reduce the targeting to Abbott districts. And, there were a few tricks in the original formula that accomplished just that, including the distribution of special education funding on a flat basis, and the reduced “combination” weighting for kids who are both ELL and low income. I totally expect formula recommendations, from either party, that attempt to spread the aid more widely, or give to those who haven’t gotten much in the last cycle, or few cycles. That’s the nature of the politics of state aid formulas. It’s all about getting enough votes through political tradeoffs.

      What irks me is when politicians (be it in the original design of SFRA or now) try to frame their redistribution preferences as being a technically more correct answer… that their preferences are more rational, more sound, empirically based policy decisions. Quite honestly, that’s the fatal flaw in this report. It would be one thing for the current administration to simply come out and say… hey… we want to spread the aid more flatly across districts and to do that we’re going to take it from Abbotts, and that’s just a matter of political preference.

      It is yet another thing for them to assert that the problem that must be addressed is the achievement gap, and that reducing aid to higher need districts by taking such steps as funding on the basis of Average Daily Attendance and reducing student need weights are a logical step toward achieving that policy goal, and that these steps are rooted in widely accepted empirical research?

  3. First..could you post a chart with the names of the districts and their dollar per pupil increases..

    As I am sure you realize, .The Gov. is using the acheivement gap argument to further his privatization of education agenda.

    So,reducing direct state aid to education in High Needs Districts will help increase the achievement gap.

    What is the obvious answer….More charter schools under State control…..and

    County School Districts which will be funded by local taxes….and allow the State to cut their percent of funding true public education in our State…..

    Its really not about helping the suburbs…

    1. Will do when I get a chance. Yes, clearly there are multiple agendas at work here. I figured I’d make a rather simple obvious point that achievement gaps between poor and non-poor will not be closed by moving the money the other direction. If you look at my previous CT post, you’ll see some discussion about the role of socioeconomically segregated charters and magnet schools in the achievement gap.

  4. As usual, great post. But, you fail to mention another glaring deficiency in the NJ report: every single respected researcher knows full well that the achievement gap CANNOT be measured using proficiency measures and especially not changes in proficiency measures. Koretz (2008) makes this point abundantly clear in his book. You can change the achievement gap simply by changing the cut score? Make the gap bigger? Move the cut score higher. Make the gap lower–let a ceiling effect creep in or move the cut point lower. Moreover, changes in proficiency are driven more by the distribution of the scores around the cut point than by advances in actual achievement. The only ACCURATE manner to measure the achievement gap is to use scale scores as DiCarlo does using the NAEP scores.I’m using this example in my class on “understanding Education Data.” A classic misuse of data to make an ideological point.

    1. I posted a link to a post that explains that. Not enough time for everything. Thanks Ed!

  5. Bruce,
    If you look at the oft quote 8th. Grade LAL achievement gap statistic ( NJ is 50 th out of 51), is that a fair presentation? If you look at the NAEP scores with an eye towing the Confidence Intervals, I do not see it that way? If you have such a broad based test with the averages reported to 6 decimals, I think it gives an impression of precision that does not exist? Your thoughts?
    Thank you.

  6. Bruce, sorry for all the typos above. This new iPad has an annoying way of anticipating my keystrokes and editing my text in ways that I have not yet figured out.

  7. Bruce,
    Thank you, I am looking for a bit of an education on this issue though. Irrespective of the gap issue, is it fair to say that any given state ranks higher than any other if the confidence intervals overlap? Just looking for some help.
    I read your reference above and, as usual, it was enlightening. Thank you.

    1. I would agree that any ranking of that sort, where there’s so little certainty that one state (country, school district, whatever) is ahead of any other is problematic. So, you effectively have large groups that aren’t statistically differentiable from one another, but their means differ (or in this case, more problematically, differences between means for two groups within states). I’ve not checked how this really plays out with the NAEP gap rankings, since there are other conceptual & measurement problems that undermine the comparisons.

      In most such cases, you would at least be able to say that it is more likely than not that this state, or these states, with higher means, do rank higher than these states with lower means. Remember, we might be setting a pretty high standard of confidence w/those intervals. That, for example, we can be 95% confident that one state ranks above another, or a group of others, or the national average. A simple “more likely than not” standard would lower that confidence level considerably. So, we could still rank states by their mean, but then add the caveats of how certain we are that the mean for any given state is above the next state behind it, the state behind that and so on. Most people’s eyes would glaze over at that point, but it would be a more accurate representation.

  8. Bruce,
    Thank you for taking the time to respond to my inquiry. I always find it a bit annoying when people speak in certain terms about statistics, especially NAEP scores which are usually touted as the ” gold standard”. I have many issues with those tests, and even moreso the inferences drawn from them. Your perspective is helpful. I will continue to check your blog and learn along the way. Thanks Again,

Comments are closed.