How NOT to fix the New Jersey Achievement Gap

Late yesterday, the New Jersey Department of Education Released its long awaited report on the state school finance formula. For a little context, the formula was adopted in 2008 and upheld by the court as meeting the state constitutional standard for providing a thorough and efficient system of public schooling. But, court acceptance of the plan came with a requirement of a review of the formula after three years of implementation. After a change in administration, with additional legal battles over cuts in aid in the interim, we now have that report.  The idea was that the report would suggest any adjustments that may need to be made to the formula to make the distributions of aid across districts more appropriate/more adequate (more constitutional?). I laid out my series of proposed minor adjustments in a previous post.

Reduced to its simplest form, the current report argues that New Jersey’s biggest problem in public education is its achievement gap – the gap between poor and minority students and between non-poor and non-minority students.  And the obvious proposed fix? To reduce funding to high poverty, predominantly minority school districts and increase funding to less poor districts with fewer minorities.

Why? Because money and class size simply don’t matter. Instead, teacher quality and strategies like those  used in Harlem Childrens’ Zone do!

Here’s my quick, day-after, critique:

The Obvious Problem? New Jersey’s Huge & Unchanging Achievement Gap

The front end of the report provides lots of nifty graphs based on cohort proficiency rates on tests which change substantially in some years. The graphs are neatly laid out to validate the argument that New Jersey’s achievement gap is large and hasn’t changed much.  First, on the point of the largeness of the gap, in national context. I’ve explained here how the NJ poor-non-poor gap is actually relatively average nationally. That’s not to say that it’s acceptable, we ought to work on this, by whatever reasonable means we can.

Thankfully (so I don’t have to revisit all of the problems here), the remainder of the achievement gap analysis presented by NJDOE is thoroughly critiqued in a recent post by Matt Di   Carlo at Shanker Blog. DiCarlo summarizes some of the NJ achievement gap and trend data to point out:

The results for eighth grade math and fourth grade reading are more noteworthy – on both tests, eligible students in NJ scored 12 points higher in 2011 than in 2005, while the 2011 cohorts of non-eligible students were higher by roughly similar margins.

In other words, achievement gaps in NJ didn’t narrow during these years because both the eligible and non-eligible cohorts scored higher in 2011 versus 2005. Viewed in isolation, the persistence of the resulting gaps might seem like a policy failure. But, while nobody can be satisfied with these differences and addressing them must be a focus going forward, the stability of the gaps actually masks notable success among both groups of students (at least to the degree that these changes reflect “real” progress rather than compositional changes).

http://shankerblog.org/?p=5102

Revelation? Gaps are a function of the height of the highs as much as the depth of the lows. If both get better, gaps don’t close as much. Gaps are still a problem, and must be addressed even if the highs get higher, because opportunity for access to college and on the labor market is relative. But, the framing of the NJ achievement gap by NJDOE is unhelpful in this regard, and the proposed solutions harmful. How does it make sense then, to provide greater increases in state aid to those students in districts at the highs and less to the lows?

Supporting Claims for Solutions?

Of course, to support the eventual pre-determined (utterly absurd) conclusion that the way to close this achievement gap is to cut aid to the poor and give it to the less poor requires that the report validate that money really has nothing to do with it. That, arguably, all of that money and increased staffing actually made things worse. Further, that cutting money from poor districts is what will make them better. I guess it also then stands to reason that giving larger aid increases to less poor districts might also make them worse, and viola – the achievement gap shrinks!

• Claim 1: Money Has Nothing to do with It

The claims that money doesn’t matter are built on some graphs which could easily make my list of dumbest graphs (or at least most pointless, deceptive, meaningless ones). Here’s one which is intended to convince the reader that all of that money sent to Abbott districts was for naught:

The report uses the graph to conclude:

While the above analysis is not sufficient to say whether new spending has had a positive impact on student achievement, it makes clear that financial resources are not the only – and perhaps not even the most important – driver of achievement.

If the graph isn’t sufficient to make this point, then why use the graph to try to make this point? Clearly, looking only at two variables – percent change in revenue and percent change in proficiency rates – is not even sufficient to make the softened claim “perhaps not even the most important” factor in improving student achievement.  These assertions can’t be supported in any way by this graph.

But even more suspect is the assertion embedded in the policy recommendations  that therefore, cutting aid from high poverty districts will cause no harm.

Better research on whether and to what extent school finance reforms improve student outcomes &/or equity of outcomes shows that in fact, school finance reforms can and do improve both the level and distribution of student outcomes: http://www.tcrecord.org/content.asp?contentid=16106

Higher quality research, in contrast, shows that states that implemented significant reforms to the level and/or distribution of funding tend to have significant gains in student outcomes.

Further, research on the broader question (based on real analysis) of whether and how class size and money matter indicates that, in simple terms, money does matter, and that things that cost money, like class size reduction and improving teacher quality (which does cost money) matter:  http://www.shankerinstitute.org/images/doesmoneymatter_final.pdf

Perhaps most importantly, even the research that has cast doubt on the strength of the positive influence of money on student outcomes has never validated that cuts to funding are not harmful and may be helpful. This is an absurd and unfounded claim.

Richard Murnane of Harvard said it well enough back in the early 1990s:

“In my view, it is simply indefensible to use the results of quantitative studies of the relationship between school resources and student achievement as a basis for concluding that additional funds cannot help public school districts. Equally disturbing is the claim that the removal of funds… typically does no harm.” (p. 457)

Murnane, R. (1991) Interpreting the Evidence on Does Money Matter? Harvard Journal of Legislation. 28 p. 457-464

Though not directly stated in the NJDOE report, it is implicit in the recommendations.

• Claim 2: Teacher Quality & Harlem Childrens’ Zone-Style Strategies Can Close the Gap

Deeply embedded in the NJDOE report, making the transition from claims of dire achievement gaps toward how to fix them, is a discussion of how the obvious solutions based on current research must have to do with improving teacher quality and doing stuff like Harlem Childrens’ Zone does.  The NJDOE report includes two particularly bold statements that these two strategies alone – but certainly not money – can close the black-white achievement gap:

Having a highly effective teacher for three to five years can erase the deficits that the typical disadvantaged student brings to school.xxiii

Evidence from the Harlem Children’s Zone provides a similar demonstration of the power of schools to close the black-white achievement gap existing in New York.xxiv

Needless to say, these interpretations of the existing research are a massive unwarranted stretch. Matt Di    Carlo addresses the issue of  how many teachers does it take to close the achievement gap?

Even then, the implicit assertion of the report in general, that money has nothing to do with teacher quality or the distribution of teacher quality, is ridiculous. As I explain here:

A substantial body of literature has accumulated to validate the conclusion that both
teachers’ overall wages and relative wages affect the quality of those who choose to enter the teaching profession, and whether they stay once they get in. For example, Murnane and Olson (1989) found that salaries affect the decision to enter teaching and the duration of the teaching career, while Figlio (1997, 2002) and Ferguson (1991) concluded that higher salaries are associated with more qualified teachers.

http://www.shankerinstitute.org/images/doesmoneymatter_final.pdf

And further, on the flip side, cuts to funding and severe constraints on spending growth can reduce teacher quality:

Research on the flip side of this issue – evaluating spending constraints or reductions – reveals the potential harm to teaching quality that flows from leveling down or reducing spending. For example, David Figlio and Kim Rueben (2001) note that, “Using data from the National Center for Education Statistics we find that tax limits systematically reduce the average quality of education majors, as well as new public school teachers in states that have passed these limits.”

And, if we are interested in achievement gaps, and better distributing the quality of teachers across richer and poorer districts and children:

Salaries also play a potentially important role in improving the equity of student outcomes. While several studies show that higher salaries relative to labor market norms can draw higher quality candidates into teaching, the evidence also indicates that relative teacher salaries across schools and districts may influence the distribution of teaching quality. For example, Ondrich, Pas and Yinger (2008) “find that teachers in districts with higher salaries relative to non-teaching salaries in the same county are less likely to leave teaching and that a teacher is less likely to change districts when he or she teaches in a district near the top of the teacher salary distribution in that county.”

But even more strikingly, these interpretations ignore entirely that what Harlem Childrens Zone does, above and beyond anything else is to spend a ton of money (raising as much as $60,000 per pupil in private giving in some years, for additional information, see this post) and spend much of that money on providing smaller class sizes than surrounding NYC district schools.  So, in effect, what Harlem Childrens Zone shows us (in its best light) is that we can make modest progress toward closing achievement  gaps by leveraging substantial additional financial resources to provide comprehensive wrap-around community resources coupled with small class sizes.

The Proposal: Cut Aid to the Poor and Give More to the Non-Poor (& Less Poor)

After the rather predictable preamble about New Jersey’s achievement gap, coupled with classic claims that money clearly isn’t the answer, and things that actually cost money, but we’ll pretend don’t really cost money are the answer, the obvious recommendations for changes to the school finance formula are to reduce aid to the poor and give it to the less poor.

Here are the distributions of the percent change in state aid for 2012-13 across K-12 districts and the per pupil (preliminary estimates in need of updated enrollment figures) by districts arranged from lower to higher concentrations of low income children:

K-12 Unified Districts Only

K-12 Unified Districts Only

The report argues specifically that the adjustments in the aid formula for low income children should be reduced. That they should be reduced because they were increased without basis, over original recommendations provided to the state board back in 2003 (but hidden until 2006). In short, that those low income kids really don’t need that much and will be better off without it.

I critique those original recommendations in this report. Essentially, the argument is that there is simply no basis for providing as much as an additional 57% per low income child in high poverty concentration districts, therefore we should reduce it. The icing in the cake in this argument is a table in which the report points out that Texas, Vermont and Maine provide less than this. How in the heck they chose Texas, Vermont and Maine is beyond me. These states are at least a little different from NJ… and… from each other.

Beyond that, it should go without saying that the decisions of policymakers in three completely different states that aren’t New Jersey really have little or nothing to do with the cost of providing equal educational opportunity to low income kids in New Jersey.  Are we going to base all of our policies on Vermont… and Texas simultaneously? That would be a real trick? Consider the possibilities?

As my reported linked above points out, the weights in the original analysis were too low, and were thus adjusted upwards, though not necessarily far enough? On what basis? Well, the actual research on the costs of providing equal educational opportunities for low income children points to weights nearer to double, not 40% or 50% higher than average.  Here’s the most directly relevant article, from the Economics of Education Review, and here’s a link to a National Research Council report on the subject.

In a further effort to reduce aid to poorer districts (in a way that will have multiplicative effects throughout the formula) NJDOE proposes to base the allocation of aid on Average Daily Attendance. This is actually a classic, well understood Trick of the Trade for shifting aid away from poorer districts which for a variety of reasons outside their control have lower attendance rates. Way back when I started this blog, one of the topics I wrote about was these seemingly innocuous tricks (a subject of my research).  While other states do continue to use these policies, since their effects are well understood, to recommend such a change is shameless.

But even setting aside the empirical evidence on “costs,” how can it possibly make sense that achievement gaps between richer and poorer districts will be moderated by taking money from poorer districts and redistributing it back to less poor ones?

That’s the report in its essence.

We’ve got big achievement gaps.

Money doesn’t matter – in fact – it must be making things worse not better.

Therefore, to close the gaps, we need to give less of that harmful money to the poor, and more to the non-poor.

Go figure?