Friday Ratings Madness: Quality Counts, Students First & Funding Fairness

It’s been a fun week for grading the states. First we had the wacky ratings from Students First which graded states largely on the extent to which they had adopted the preferred policies of that organization. Then we had the old-standard Education Week Quality Counts. When it comes to their finance rating system, little has changed in recent years. These two reports, of course, produced substantially conflicting results.

One might argue that both reports and ranking systems, like our School Funding Fairness report, include several indicators intended to identify policy conditions for success. This has been the standard response of Students First when they have been criticized on the basis that the states that they have applauded most tend to have pretty low average outcomes.  But, the Students First report, Quality Counts and our Funding Fairness report differ quite substantially on what we consider to be policy conditions for success. 

Students First has put policy conditions into three categories – 1) elevating the teaching profession, 2) parent empowerment and 3) finance and governance.  Students first gives no consideration across any of these categories to whether teacher wages, for example are sufficient to recruit/retain high quality candidates into teaching or whether wages are specifically competitive in high need schools. Students First gives no consideration to whether funding, overall, is sufficient to provide either/both competitive wages or reasonable class sizes, generally, or specifically in high need schools.  It would appear to be their opinion (as was rather clearly expressed by Eric Lerum in a video conference) that overall level or distribution of funding isn’t the issue – but rather that their preferred policies are what matters, regardless of funding (since the only funding/resource equity considerations in their rankings pertained to whether charter schools received what they consider equal funding – no validation provided!)

Education Week goes old school especially on their school finance rankings. I don’t have time/space to address all of their rankings. As I will show below, some of their old-school measures seem to capture relatively useful information, but others do not. Let’s quickly summarize the measures they use.

• Fiscal Neutrality: Fiscal neutrality measures the relationship between district spending and district wealth. State school finance formulas are partly intended to disrupt this relationship – reduce the likelihood that wealthier districts spend systematically more. This measure is often still useful, but may be complicated by the fact that school finance formulas also try to address differences in student needs and costs. To the extend that higher need kids live in poorer districts (not always the case that taxable property wealth and student need are tightly associated), this indicator may work to partly capture both.
• McLoone Index: Named for school finance legend Gene McLoone! This index tells us how close, on average, the per pupil spending of districts in the lower half (serving the lower half of kids) are to the median. That is, to what extent does the state formula succeed in “leveling up” the bottom half to the middle. A McLoone of 100 would mean that the lower half is equal to the middle. But this index in particular can produce some screwy results. Say for example a state has one or a few very large districts with high need populations and those districts constitute both the lower half and the middle (they have nearly or all of the bottom half of kids). A state with one or a handful of high need large districts with spending lower than everyone else (the upper half) might still get a McLoone of 100. But it would be a really crappy school finance system! (with all due respect to Gene!)
• Coefficient of Variation: The coefficient of variation simply measures the extent of variation in per pupil spending as a percent of the mean per pupil spending. A CV of 10% indicates that 2/3 of children attend districts with per pupil spending within 10% of the mean. The problem with the CV is that, while it measures variation, it doesn’t capture the difference between GOOD variation and BAD variation. Modern state school finance formulas try to create variation in funding to accommodate differences in student needs. Education Week uses nominal weights to “adjust” for differences in student needs, but some state school finance systems actually adjust more aggressively for needs than do their weights. Those states are penalized in the CV.
• Spending Index & Percent at/Above National Mean: A few reports back Education Week wanted to construct a form of “spending adequacy” figure to compare spending levels across states and the shares of kids with access to what they considered more “adequate” spending. So they adopted this measure and index based on the percent of children in each state who attended districts that spent at least the same as the national average district (spending adjusted for regional wage variation). This figure does generally capture spending level differences across states – adjusted for wage variation – but doesn’t, for example capture spending level differences corrected for student population differences, or the shares of students who might be attending very small, remote rural districts.

Ed week includes a few additional indicators like the restricted range – or difference in spending between the 95th and 5th %ile district, but these are largely redundant with the CV & McLoone and suffer the same problems of not accounting for other cost factors – or state aid formulas that aggressively adjust for needs and costs.

We had set out to correct for many of the problems in the Ed Week approach when we started work  on our Funding Fairness report. Specifically, we wanted to make comparisons that better accounted for differences in needs and costs across districts and states and that could be used to characterize state school finance policies consistently, without suffering some of the problems of old-school indicators like the CV or McLoone Index. We also look at spending level – using a statistical model based on 3 years of data to project the per pupil state and local revenue of a district with a) average poverty rate, b) in an average wage labor market and c) with 2,000 or more students and average population density. That is, our projected state and local revenue figures are adjusted for poverty, competitive wages, size and population density. We use the same model to then evaluate whether, on average – and in a predictable pattern – state and local revenues are systematically higher (progressive) or lower (regressive) in higher poverty districts (relative to lower poverty districts). That is, does the system overall target resources to higher poverty districts – controlling for the other factors.

That prerequisite discussion aside, let’s take a look at how all of this stuff lines up – How the Ed Week Indicators line up with the Funding Fairness Indicators and how both line up with the Students First Indicators. Finally, I look at how all line up with various outcome measures.

Again… all of these funding related indicators are about policy conditions for success, rather than success itself.

First up – and here’s a relative no-brainer – both our funding fairness report and Ed Week Quality Counts include an indicator of state funding effort – or share of state capacity allocated to elementary and secondary education.  I can’t speak for Ed Week, but we include ours to acknowledge that some states spend more than others (do better on our spending level measure) because they can and that we should grade them at least partly on their effort.  Figure 1 shows that our effort measure and Ed Week’s effort measure are pretty highly correlated.

Figure 1

Figure 2, by contrast, shows that the Students First funding GPA isn’t related at all with the Ed Weeks effort indicator, and by extension with ours. Ed Week (and we) consider funding effort to be an underlying policy condition for success, apparently, Students First doesn’t .

Figure 2

Figure 3 compares our funding level indicator and Education Week’s spending index – or relative adequacy indicator. Clearly the two are highly related… but the Ed Week indicator caps out at 100% – or where 100% of the children attend districts above the national average spending. Personally, I prefer indicators that capture the full range of variation.  But again, our spending level measure and Ed Week’s spending index are picking up much of the same information – relative spending differences across states.

Figure 3

But, Figure 4 shows that Students First finance rating scheme really doesn’t relate at all to Education Week’s spending index, suggesting that overall availability of resources – like the effort to raise them – is inconsequential in the eyes of Students First.

Figure 4

Figure 5 shows the relationship between our funding progressiveness indicator and Ed Week’s fiscal neutrality indicator. For many states, the two are picking up similar things. In states like New Jersey or Utah, where higher poverty districts have more resources than lower poverty ones, the systems have also achieved fiscal neutrality (disrupted the relationship between wealth and spending). By contrast, in states like Illinois or North Carolina, the wealth-spending relationship remains strong and positive (higher wealth – higher spending) and higher poverty districts receive systematically fewer resources!

Figure 5

Recall that Illinois received one of the best grades on finance from Students First. Apparently, in addition to effort and spending level, fiscal neutrality and need based funding are also inconsequential to Students First when it comes to funding issues. Figure 6 shows the relationship between funding progressiveness and Students Firsts funding related GPA. Note that all of Students First’s funding superstars (Illinois, New York, Rhode Island and Michigan) are less than stellar on funding fairness.

Figure 6

Figure 7 relates the Ed Week CV to our funding fairness measure, showing that states with progressive funding distributions including New Jersey, Ohio and Massachusetts are actually penalized by this measure.  The CV does not differentiate between need based variation as occurs in these states and wealth-drive variation as occurs in New Hampshire.  We all seem to agree – Ed Week, Students First and us… that New Hampshire’s funding is…well… not so good!

Figure 7

Moving on, here’s the relationship between our funding fairness measure and the McLoone Index! Not much going on here… and but for a few specific examples… it’s actually hard to tell what the McLoone really captures these days in complex state school finance systems. At least it captures that New Hampshire school funding… well… sucks! But other than that, the McLoone really doesn’t capture much valuable additional information regarding equity.

Figure 8

So then, how do these various policy conditions for success relate to various outcome measures. In this table and the following graphs I explore that question, using the following outcomes:

1. Reduction in % below proficient (from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG12-03_CatchingUp.pdf)
2. Annual Standardized Gain (NAEP, from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG12-03_CatchingUp.pdf)
3. Adjusted (for initial level) Annual Standardized Gain
4. Reading and Math NAEP 8th Grade 2011
5. Reading and Math NAEP 8th Grade for Lowest Income Group (Free Lunch) 2011

Table 1 shows the correlations between each of the indicators addressed above and the outcome measures listed above.  Note that each of these correlations a) is relatively modest to non-existent and b) merely represents a relationship whereby when X is higher, so too is Y. Underlying causal relationships involve a complex web of factors including socio-economic and demographic conditions, etc.

Table 1

Figure 9 ranks the correlations between policy conditions and reduction in % below proficient at the 8th grade level. Interestingly, variation (inequity – bad and good) in spending is most positively associated with reduction in % below proficient. Beyond that, our funding level indicator and the two funding level indicators from Ed Week are next in line.  Students First’s teaching profession indicator is next… but their funding indicator further down. The figure seems to suggest that higher spending states, even where that spending is unequal, are doing okay on reducing % below proficient – but this is a pattern that can clearly be influenced by regional variation.

Figure 9

Figure 10 shows the correlations – ranked high to low – between each policy condition and adjusted standardized gain. In this case, adjusted standardized gains are most highly correlated with our spending level indicator, the Ed Week spending adequacy indicator, and our progressiveness indicator and Ed Week’s neutrality indicator. One might infer from this that more equitable and adequate funding is associated with greater long term average gains on NAEP… but again, regional differences may drive this to an extent. To get an idea of which states have better “adjusted annual gains” see the figure in Appendix A. Higher adjusted gains are states above the trendline and lower adjusted gains are those below the trendline.  Not all states are included (in the graph or correlations) for lack of baseline data year (I may work on updating this with multiple baseline years & tests. This is just a start).

Figure 10

Finally, we have Figure 11, which compares correlations with the NAEP scores of the lowest income children (which across states were not associated with the average income of the families of those children). These are children in families below the 130% income level for poverty.  As in Figure 9, states with the greatest spending variation seemed to have higher low income NAEP scores. Beyond that however, funding level, effort and wage competitiveness (Teaching Penalty data) seem to be positively correlated with low income NAEP scores. That is, states with higher funding levels, that put up more funding effort, and that have more competitive teacher wages (weekly, relative to non-teachers) have higher low income NAEP scores.

In Figure 11, all of the students first indicators (GPAs) are negatively associated with low income student NAEP scores. That is, low income children are doing much worse in states that got good grades from Students First.  That said, many of the conditions Students First included as setting the state for future success are policies only recently implemented in these states.

Figure 11

So that’s it… my run down of the relationship between this week’s state rankings data, how they relate (or not) to our School Funding Fairness Report and how they relate to various outcome measures. I’ll let the rest of you run with it from here! Cheers!

Data Sources:

Students First Report Card

School Funding Fairness

Ed Week Quality Counts (Finance)

Teaching Penalty

Relevant Additional Readings

Baker, B. D., & Corcoran, S. P. (2012). The Stealth Inequities of School Funding: How State and Local School Finance Systems Perpetuate Inequitable Student Spending. Center for American Progress.

Baker, B. D. (2012). Revisiting the Age-Old Question: Does Money Matter in Education?. Albert Shanker Institute.

Friday Finance 101 Series

Appendix A: NAEP Standardized Gains and 1990 Scores